Thursday, 2017-05-25

*** spsurya has joined #openstack-kolla00:05
*** sbezverk has joined #openstack-kolla00:06
*** hongbin has quit IRC00:10
*** kolla-slack has quit IRC00:11
*** kolla-slack1 has joined #openstack-kolla00:11
*** huzhengchuan has quit IRC00:23
*** huzhengchuan_ has quit IRC00:23
*** jtriley has joined #openstack-kolla00:23
*** chas has joined #openstack-kolla00:24
*** huzhengchuan has joined #openstack-kolla00:26
*** eanylin has joined #openstack-kolla00:26
*** huzhengchuan_ has joined #openstack-kolla00:27
*** chas has quit IRC00:29
*** sgrasley has joined #openstack-kolla00:30
*** manheim has joined #openstack-kolla00:31
*** huzhengchuan_ has quit IRC00:31
*** huzhengchuan has quit IRC00:31
*** manheim has quit IRC00:35
*** gfidente has quit IRC00:35
*** duonghq_ is now known as duonghq00:40
duonghqmorning guys00:41
Seyonggood morning00:42
*** huzhengchuan_ has joined #openstack-kolla00:44
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles)  https://review.openstack.org/39868400:45
*** huzhengchuan has joined #openstack-kolla00:48
*** ducttape_ has joined #openstack-kolla00:54
*** jtriley has quit IRC00:57
*** cuongnv has joined #openstack-kolla01:02
*** n0isyn0i1e has joined #openstack-kolla01:12
openstackgerritzhubingbing proposed openstack/kolla master: test kolla gate  https://review.openstack.org/46780701:13
*** qiliang27 has joined #openstack-kolla01:14
*** n0isyn0ise has quit IRC01:14
*** ducttape_ has quit IRC01:19
*** ducttape_ has joined #openstack-kolla01:20
*** ducttap__ has joined #openstack-kolla01:25
*** ducttap__ has quit IRC01:27
*** manheim has joined #openstack-kolla01:28
*** ducttap__ has joined #openstack-kolla01:28
*** ducttap__ has quit IRC01:28
*** ducttape_ has quit IRC01:28
openstackgerritAndrew Bruce proposed openstack/kolla-kubernetes master: openvswitch not required to be kernel module  https://review.openstack.org/46776901:32
*** hrw has quit IRC01:32
*** sgrasley has quit IRC01:33
*** manheim has quit IRC01:33
*** hrw has joined #openstack-kolla01:34
*** manheim has joined #openstack-kolla01:37
*** manheim has quit IRC01:41
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles)  https://review.openstack.org/39868401:42
*** sbezverk has quit IRC01:42
*** tovin07_ has joined #openstack-kolla01:51
*** qiliang27 has quit IRC01:54
*** unicell has quit IRC01:58
bmaltaisNOticed something. Was trying to add new compute node to kolla-ansible. Added it to inventory file and ran kolla-ansible bootstrap-servers02:00
*** stefan_bo has joined #openstack-kolla02:01
bmaltaisThe result was a non functional environment...02:02
bmaltaisI think the bootstrap process touched existing node instead on just applying changes to new one and broke it as a result of restarting docker02:03
*** zhurong has joined #openstack-kolla02:05
*** sbezverk has joined #openstack-kolla02:07
*** sbezverk has quit IRC02:15
bmaltaisI would suggest that the bootstrap process 1st check if /etc/kolla exist on the node and if it does it should just skip it.02:21
*** chas has joined #openstack-kolla02:24
*** huzhengchuan has quit IRC02:26
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only neccesary tasks (all other roles)  https://review.openstack.org/39868502:26
*** huzhengchuan_ has quit IRC02:26
*** chas has quit IRC02:30
*** lucasxu has joined #openstack-kolla02:33
*** xinliang has quit IRC02:34
*** itlinux has joined #openstack-kolla02:37
*** huzhengchuan_ has joined #openstack-kolla02:38
*** huzhengchuan has joined #openstack-kolla02:40
*** zhuzeyu has joined #openstack-kolla02:45
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix etc/hosts format with baremetal role  https://review.openstack.org/46782702:46
*** xinliang has joined #openstack-kolla02:47
*** xinliang has quit IRC02:47
*** xinliang has joined #openstack-kolla02:47
*** caowei has joined #openstack-kolla02:50
*** itlinux has quit IRC03:03
*** lucasxu has quit IRC03:06
*** lucasxu has joined #openstack-kolla03:08
*** srwilkers has quit IRC03:09
openstackgerritzhubingbing proposed openstack/kolla master: Fix rally create verifier error  https://review.openstack.org/46783703:14
*** ducttape_ has joined #openstack-kolla03:16
*** ducttap__ has joined #openstack-kolla03:17
*** ducttape_ has quit IRC03:17
*** lucasxu has quit IRC03:18
*** yuanying has joined #openstack-kolla03:24
*** zhubingbing has quit IRC03:25
*** manheim has joined #openstack-kolla03:26
*** spsurya has quit IRC03:32
*** stefan_bo has quit IRC03:37
*** zhubingbing_ has joined #openstack-kolla03:40
*** ducttap__ has quit IRC03:51
*** daidv_ has joined #openstack-kolla03:55
*** manheim has quit IRC04:00
*** manheim has joined #openstack-kolla04:02
*** manheim has quit IRC04:06
*** ducttape_ has joined #openstack-kolla04:08
*** zhurong has quit IRC04:08
*** ducttape_ has quit IRC04:12
*** zhurong has joined #openstack-kolla04:16
*** unicell has joined #openstack-kolla04:19
*** chas has joined #openstack-kolla04:25
*** skramaja has joined #openstack-kolla04:25
*** manheim has joined #openstack-kolla04:29
*** yingjun has joined #openstack-kolla04:30
*** chas has quit IRC04:30
*** janki has joined #openstack-kolla04:32
*** manheim has quit IRC04:34
*** manheim has joined #openstack-kolla04:36
*** manheim has quit IRC04:41
*** manheim has joined #openstack-kolla04:44
*** unicell1 has joined #openstack-kolla04:47
*** unicell has quit IRC04:47
*** manheim has quit IRC04:48
*** manheim has joined #openstack-kolla04:50
*** manheim has quit IRC04:55
*** manheim has joined #openstack-kolla05:03
*** manheim has quit IRC05:07
*** zhuzeyu has quit IRC05:13
*** zhuzeyu has joined #openstack-kolla05:13
*** zhuzeyu has quit IRC05:20
*** rwsu has joined #openstack-kolla05:27
*** stefan_bo has joined #openstack-kolla05:51
*** korzen has joined #openstack-kolla05:52
*** sayantan_ has quit IRC06:02
*** manheim has joined #openstack-kolla06:03
*** caowei_ has joined #openstack-kolla06:15
*** caowei has quit IRC06:15
*** caowei_ is now known as caowei06:15
*** aagate has quit IRC06:17
*** pcaruana has joined #openstack-kolla06:20
*** stefan_bo has quit IRC06:23
manheimhello06:24
*** chas has joined #openstack-kolla06:26
openstackgerritJuan Badia Payno proposed openstack/kolla master: WIP - Modified fluentd Image  https://review.openstack.org/46721906:30
*** chas has quit IRC06:30
*** sayantan_ has joined #openstack-kolla06:34
*** sayantan_ has quit IRC06:47
*** sayantan_ has joined #openstack-kolla06:47
*** sayantan_ has quit IRC06:47
*** sayantan_ has joined #openstack-kolla06:48
*** stefan_bo has joined #openstack-kolla06:53
*** mnasiadka has joined #openstack-kolla06:59
*** shardy has joined #openstack-kolla07:10
*** dixiaoli has joined #openstack-kolla07:13
*** sayantan_ has quit IRC07:13
*** manheim has quit IRC07:22
masberI just installed openstack + magnum using kolla-ansible. How should I use magnum cli?07:22
masberdo I need to run docker exec magnum-api magnum ...?07:22
*** Serlex has joined #openstack-kolla07:25
*** manheim has joined #openstack-kolla07:25
*** imcsk8 has quit IRC07:25
*** manheim has quit IRC07:27
*** manheim has joined #openstack-kolla07:27
*** zhurong has quit IRC07:30
*** manheim has quit IRC07:31
*** imcsk8 has joined #openstack-kolla07:34
*** zhubingbing_ has quit IRC07:36
*** manheim has joined #openstack-kolla07:38
*** manheim has quit IRC07:40
*** manheim has joined #openstack-kolla07:43
*** egonzalez has joined #openstack-kolla07:44
*** zhurong has joined #openstack-kolla07:45
*** manheim has quit IRC07:47
*** manheim has joined #openstack-kolla07:50
*** mgoddard has joined #openstack-kolla07:51
openstackgerritMerged openstack/kolla-ansible master: Fix etc/hosts format with baremetal role  https://review.openstack.org/46782707:51
*** skramaja has quit IRC07:52
*** tvignaud_pto has quit IRC07:58
*** daidv_ has quit IRC08:03
*** manheim has quit IRC08:06
*** stefan_bo has quit IRC08:06
*** stefan_bo has joined #openstack-kolla08:07
*** manheim has joined #openstack-kolla08:10
*** gfidente has joined #openstack-kolla08:14
*** manheim has quit IRC08:14
*** tvignaud_pto has joined #openstack-kolla08:15
*** zhubingbing_ has joined #openstack-kolla08:18
*** manheim has joined #openstack-kolla08:19
*** manheim has quit IRC08:23
*** chas has joined #openstack-kolla08:27
*** manheim has joined #openstack-kolla08:29
*** athomas has joined #openstack-kolla08:29
*** pbourke has quit IRC08:30
*** pbourke has joined #openstack-kolla08:30
*** chas has quit IRC08:31
*** manheim has quit IRC08:33
*** lpetrut has joined #openstack-kolla08:37
*** unicell1 has quit IRC08:46
*** unicell has joined #openstack-kolla08:46
*** manheim has joined #openstack-kolla08:47
*** mgoddard has quit IRC08:53
*** zhurong has quit IRC08:54
hawiis there something i have to do with external elasticsearch, did enable_central_logging: "no" and added elasticsearch_address, but nothing08:57
*** magicboiz has quit IRC09:00
egonzalezhawi, afik the only requirement is that elasticsearch_address != kolla_internal_vip_address09:01
*** jmccarthy has joined #openstack-kolla09:03
*** zhurong has joined #openstack-kolla09:04
openstackgerritNuman Siddique proposed openstack/kolla master: Support openvswitch-ovn docker images  https://review.openstack.org/45917909:04
*** manheim has quit IRC09:11
*** manheim has joined #openstack-kolla09:13
*** zhubingbing_ has quit IRC09:18
*** mewald has joined #openstack-kolla09:22
*** duonghq has quit IRC09:30
*** zhurong has quit IRC09:35
*** yingjun has quit IRC09:37
*** sambetts|afk is now known as sambetts09:46
*** manheim has quit IRC09:48
*** manheim has joined #openstack-kolla09:53
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod  https://review.openstack.org/46727209:54
*** manheim has quit IRC09:57
*** rmart04 has joined #openstack-kolla09:58
*** tovin07_ has quit IRC10:00
hrwmorning10:08
hrwupgraded to 64 cpu cores aarch64 machine. will see how kolla goes on it10:09
*** ducttape_ has joined #openstack-kolla10:13
*** cuongnv has quit IRC10:17
*** ducttape_ has quit IRC10:18
*** manheim has joined #openstack-kolla10:26
*** chas has joined #openstack-kolla10:28
*** caowei has quit IRC10:29
*** manheim has quit IRC10:30
*** chas has quit IRC10:33
*** dixiaoli has quit IRC10:39
egonzalezcan someone try a mariadb upgrade from ocata to master? newton-ocata works fine but to master containers keep restarting https://bugs.launchpad.net/kolla-ansible/+bug/169250710:40
openstackLaunchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New]10:40
*** manheim has joined #openstack-kolla10:44
*** manheim has quit IRC10:48
*** mnasiadka has quit IRC10:57
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster  https://review.openstack.org/46688111:02
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment. This should help new users get started quickly by having all of the needed configuration to accomplish the task in one place.  https://review.openstack.org/46769511:04
*** skramaja has joined #openstack-kolla11:04
*** stefan_bo has quit IRC11:09
*** rhallisey has joined #openstack-kolla11:12
sean-k-mooneyegonzalez: are you seeing that on ubuntu ?11:15
egonzalezsean-k-mooney, centos source11:15
sean-k-mooneyegonzalez: well unless we backport the pinning i did for master i think ocata is still pointing to 10.0 not 10.0.3011:16
*** manheim has joined #openstack-kolla11:17
egonzalezsean-k-mooney, used ocata images from dockerhub with mariadb 10.0.3011:17
sean-k-mooneyegonzalez: whats the workflow to upgrade? build both images, kolla-ansible deploy --tags mariadb, kolla-ansable upgrade --tags mariadb?11:17
*** manheim has quit IRC11:18
*** manheim has joined #openstack-kolla11:18
egonzalezkolla-ansible deploy --tags haproxy,mariadb, change version at globals.yml and kolla-ansible upgrade --tags haproxy,mariadb11:18
egonzalezwith multinode, probably allinone work11:19
sean-k-mooneyok cool ill try that and see what happens. never tried upgradeing yet11:19
sean-k-mooneyah mult not that will take me a little longer to set up11:19
egonzalezallinone works11:26
*** yingjun has joined #openstack-kolla11:27
*** n0isyn0i1e is now known as n0isyn0ise11:28
*** mgoddard has joined #openstack-kolla11:29
*** manheim has quit IRC11:30
*** magicboiz has joined #openstack-kolla11:32
*** magicboiz has quit IRC11:37
jmccarthyBy chance, anyone run into a complaint like this before with TASK [neutron : Running Neutron bootstrap container] ?12:00
jmccarthy  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/conf/common.py", line 101, in <module>     help=_("Hostname to be used by the Neutron server, "   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 1429, in __init__     **kwargs)   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 904, in __init__     self._check_default()   File "/var/lib/kolla/venv/lib/python2.7/site-pac12:00
*** sgrasley has joined #openstack-kolla12:02
*** bmaltais_ has joined #openstack-kolla12:03
bmaltais_Morning. I noticed something while adding a new node last night.12:04
bmaltais_I use the bootstrap feature of kolla-ansible.12:05
egonzalezjmccarthy, nope, that option is expecting a hostname https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L9912:05
bmaltais_I added the new blank node in the inventory file and ran bootstrap -i inventory12:05
egonzalezby default is its hostname, unless using nova_fake12:05
*** vhosakot has joined #openstack-kolla12:05
bmaltais_The result was a disaster. The environment was dead. Somehow bootstrap will re-run on already bootstrapped nodes and break them in the process.12:06
bmaltais_I think bootstrap should check for the presence of /etc/kolla and should not run on nodes it find it on to prevent breakage.12:06
jmccarthy@egonezalez you mean I can't have the hostname come up as a 'short' name ? 'hostname' of the node is control01_ru612:08
egonzalezjmccarthy, can you share the whole trace of the error? also some logs may be present at /var/lib/docker/volumes/kolla_logs/_data/neutron12:09
*** manheim has joined #openstack-kolla12:12
jmccarthy@egonzalez: This is the info I have: http://paste.openstack.org/show/610635/12:13
*** manheim has quit IRC12:14
egonzalezjmccarthy, underscores in hostname are not valid characters, thats why oslo is complaining about12:17
egonzalezjmccarthy, from the RFC http://www.rfc-editor.org/rfc/rfc1912.txt Allowable characters in a label for a host name are only ASCII12:22
egonzalez   letters, digits, and the `-' character12:22
jmccarthyHas it only started complaining about that lately ? These systems have had these names for ages ..12:22
egonzalezjmccarthy, what version are you using? there was a recent (~1 month) in neutron to validate hostname instead of a string12:23
*** manheim has joined #openstack-kolla12:23
vhosakotyes, I've seen oslo complain about underscores in hostname too.  Docker (now moby) also has issues if there are underscores in hostnames --> https://github.com/moby/moby/issues/22057.    some DNS servers fail to resolve/set domain names when underscores are seen in hostnames.12:24
vhosakotbmaltais_: haven't used bootstrap-servers at all :)  I'll try to see what going on before I leave for my vacation today, if not, I'll see after I return from vacation after June 6th.12:25
jmccarthyI'm using stuff that I'm building off master, hmm ok I'll try changing the hostnames12:25
egonzalezjmccarthy, related change https://github.com/openstack/neutron/commit/ac799dc016f6d5d30b46087c6f0a82ac01cb5be312:25
jmccarthyOk cool, I'll give that a shot - thanks :) !12:27
*** ducttape_ has joined #openstack-kolla12:27
egonzalezbmaltais, should be idempotent unless you updated packages, which maybe cause conflicts because some versions are pinned. please open a bug with logs and all information12:27
*** manheim has quit IRC12:28
*** Pavo has joined #openstack-kolla12:28
*** chas has joined #openstack-kolla12:29
vhosakotegonzalez: jmccarthy wonder if oslo runs this unit test in their gate ;)   ->   https://github.com/openstack/oslo.config/blob/master/oslo_config/tests/test_cfg.py#L2457-L246512:29
egonzaleznot sure, only checked the rfc the validate hehe12:30
vhosakot;)12:30
bmaltais_egonzalez: One of the task is restarting docker... and I think it is the one that caused the castle to crumble... but I might be wrong about the exact cause... but running bootstrap resulted in a bunch of containers in restart mode. fluentd was one of them.12:31
*** chas has quit IRC12:33
*** rwellum has joined #openstack-kolla12:34
egonzalezvhosakot, is tested https://github.com/openstack/oslo.config/blob/master/oslo_config/tests/test_types.py#L77212:35
bmaltais_egonzalez: I think it happen in the post-install.yml when docker is restarted... not sure, could be other steps. I don't quite understand the ansible checks and balances12:35
vhosakotegonzalez: ah, good find, so it a negative test that will fail.12:36
*** yingjun has quit IRC12:37
*** mnasiadka has joined #openstack-kolla12:38
egonzalezbmaltais, you updated docker before executing bootstrap? if so, can you check current docker version?12:38
*** manheim has joined #openstack-kolla12:39
*** manheim has quit IRC12:43
*** ducttape_ has quit IRC12:43
*** chas has joined #openstack-kolla12:45
bmaltais_egonzalez: I did not. I just ran bootstrap once with 3 nodes. Made sure all was working. Then added the new node to inventory, Ran bootstrap again and the environment was no longer responding.12:45
bmaltais_egonzalez: I am re-building and will test again capturing the bootstrap output to see what is returned.12:46
bmaltais_egonzalez: Right now all docker version are at 1.12.0, build 8eab29e12:47
*** chas has quit IRC12:49
*** manheim has joined #openstack-kolla12:50
*** manheim has quit IRC12:50
*** manheim has joined #openstack-kolla12:50
*** sbezverk has joined #openstack-kolla12:52
*** iniazi_ has quit IRC12:52
bmaltais_egonzalez: OK, Environment is crashed after running bootstrap. Docker versior is the same.12:53
bmaltais_egonzalez: I had a watch on all servers and saw docker restart at the same time on all of them... This is causing a mariadb issue for sure.12:54
bmaltais_egonzalez: Yep, mariadb is in a restart look. Will attempt a repair and see.12:55
*** fguillot has joined #openstack-kolla12:55
bmaltais_s/look/loop12:55
*** manheim has quit IRC12:56
bmaltais_egonzalez: If I knew ansible better I would propose a change prevent running boostrap on already bootstrapped nodes by checking for /etc/kolla directory on them.12:58
*** jchhatbar has joined #openstack-kolla12:58
*** janki has quit IRC12:58
egonzaleznever is late for learning new things ;)12:58
vhosakotbmaltais_: I thought https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/baremetal/tasks/bootstrap-servers.yml is run for baremetal/bifrost deploys, you can check with sean-k-mooney12:58
bmaltais_egonzalez: Now even a mariadb recovery will not complete.12:58
bmaltais_vhosakot: I will check with him and see what he think12:59
sean-k-mooneyvhosakot: no its inteneded to replace the quickstart guide basically12:59
vhosakotah, thanks for the info, cool13:00
*** manheim has joined #openstack-kolla13:00
bmaltais_sean-k-mooney: I ran in an issue with bootstrap running on already bootstrapped nodes13:00
sean-k-mooneyvhosakot: it can be used after bifrost but it should work with any ubuntu/centos base install13:00
vhosakotsean-k-mooney: ah ok, bmaltais_ used it to add a compute node :)13:00
bmaltais_It restart docker on all existing node causing a major crash of the environment13:00
bmaltais_sean-k-mooney: Shouldn't it skip already bootstrapped nodes all together to avoid issues?13:01
sean-k-mooneybmaltais_: for that to work you would need to add --limit <compute node name> so that it only bootstraps the compute node13:01
sean-k-mooneybmaltais_: well it was inteded to be used only  for fresh installs13:02
vhosakotthat's what I thought too, for new installs13:02
sean-k-mooneythat said restarting docker should not break things13:02
bmaltais_sean-k-mooney: OK. I never used that option... good to know. But for a good safety measure shouldn't it behave like that and perhaps a globals.yml flag could tell it to do it none the less when a user really feel like it?13:02
sean-k-mooneyit happens everytime you reboot13:02
bmaltais_sean-k-mooney: But it is nice to be able to bootstrap new nodes afterward...13:03
hrwdid someone checked in which moment of building images docker itself became a weak spot?13:03
bmaltais_Isean-k-mooney: If an operator make a typo then the whole environment would go down13:03
sean-k-mooneybmaltais_: if we did that we would have to introduce some kind of sentenal on each host to know it had been bootstrapped13:04
*** manheim has quit IRC13:04
sean-k-mooneybmaltais_: yep13:04
bmaltais_sean-k-mooney: Bu checking for /etc/kolla directory, yes13:04
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469013:04
bmaltais_sean-k-mooney: That directory get created when running bootstrap13:04
sean-k-mooneybm that may or may not work but we could add something like that13:06
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469013:06
*** srwilkers has joined #openstack-kolla13:06
bmaltais_sean-k-mooney: Sure. that was just a suggestion. I am sure there are better sentry to use.13:06
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode  https://review.openstack.org/45469013:06
sean-k-mooneybmaltais_: rather that checking for /etc/kolla i would be more comfrotable checking for /etc/kolla/bootstraped13:06
hawii'm getting FAILED! => {"failed": true, "msg": "The conditional check '{{ (ceph_files_json.stdout | from_json).changed }}' failed. The error was: No JSON object could be decoded"} when deploying 4.0.0 Fetching Ceph keyrings13:07
bmaltais_sean-k-mooney: Make sense. So the bootstrap process would create a file at then end that could be detected to prevent from running again.13:07
hawinever happened before13:07
*** powerd has joined #openstack-kolla13:08
sean-k-mooneybasically i dont want to get to a point where the bootstrap role partly completes and has created the directoy but not actully bootstraped properly13:08
sean-k-mooneyso ya and then we could add a --rebootstrap option or something if we wanted to run on all nodes ignoreing if they were previously bootstraped13:09
*** manheim has joined #openstack-kolla13:10
sean-k-mooneybmaltais_: others have extended it but i originally wrote the bootstrap role in 3 days to help with the OSIC performance testing so there are many improments that could be made13:10
openstackgerritJuan J. Martínez proposed openstack/kolla master: Fix in sed line (s command)  https://review.openstack.org/46802613:11
bmaltais_sean-k-mooney: If my knowledge of ansible was ok I would give it a try but I am literally starting from 0. I would probably do more harm than good at this time.13:13
sean-k-mooneybmaltais_: by the way in the future you can also contol what it runs on by specifying the host you want to bootstrap in the baremetal group in your inventory13:13
*** korzen has quit IRC13:13
bmaltais_sean-k-mooney: OK. Right now the default example file make it include all nodes... I guess this is another way to control the scope.13:14
*** manheim has quit IRC13:14
*** manheim has joined #openstack-kolla13:14
bmaltais_sean-k-mooney: For now I will adjust my "add new node" documentation to refer to --limit13:14
*** jchhatbar is now known as janki13:15
*** huzhengchuan_ has quit IRC13:15
*** huzhengchuan has quit IRC13:15
sean-k-mooneyon --limit it should work but i have only tested the group way13:16
*** lucasxu has joined #openstack-kolla13:16
bmaltais_sean-k-mooney: Will find out once I rebuild my test environment13:17
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Remove the unnecessary jinja2 templating delimiters  https://review.openstack.org/46661213:17
sean-k-mooneybmaltais_: so what exectly happened to you test enviornment. it should have started working again after docker restarted13:18
sean-k-mooneyit would have gone offline while it was restarted but it should not have broken it permently13:19
bmaltais_sean-k-mooney: mariadb was in restart loop. Tried stoping all mariadb containers and perform a kolla-ansible recover-mariadb13:19
bmaltais_sean-k-mooney: But that never completed. The DB never came back up13:19
vhosakotbrb, 1513:19
*** vhosakot has quit IRC13:19
sean-k-mooneyah ok13:20
sean-k-mooneyrestarting docker is effectly the same as what happens if you reboot the node so that is the correct way to fix it13:20
bmaltais_sean-k-mooney: When I manually do a docker stop mariadb on each node I can bring it back after... but did not work after the bootstrap13:20
bmaltais_sean-k-mooney: Perhaps because everything collapse at the same time... vs running sequentially when doing it manually.13:21
sean-k-mooneyi might take a look at the bootstrap playbook again and see if i can refactor it to only restart docker if we really need to and to skip already bootstrapped nodes13:23
sean-k-mooneybmaltais_: can you open a bug for this just so we dont forget13:23
bmaltais_sean-k-mooney: Sure. I just need to figure out how to open a bug ;-) Will be my 1st.13:24
sean-k-mooneydo you have a launchpad account https://bugs.launchpad.net/kolla-ansible13:25
*** dixiaoli has joined #openstack-kolla13:26
*** lucasxu has quit IRC13:27
*** lucasxu has joined #openstack-kolla13:27
*** dixiaoli has quit IRC13:28
*** jtriley has joined #openstack-kolla13:33
*** lucasxu has quit IRC13:33
bmaltais_sean-k-mooney: Figured it out. Here is the link to the bug: https://bugs.launchpad.net/kolla-ansible/+bug/169350913:34
openstackLaunchpad bug 1693509 in kolla-ansible "Running bootstrap on existing node cause issues" [Undecided,New]13:34
*** srwilkers has quit IRC13:34
*** lucasxu has joined #openstack-kolla13:34
sean-k-mooneycoool thanks13:35
*** eanylin has quit IRC13:37
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod  https://review.openstack.org/46727213:40
*** AzazeL_ has quit IRC13:41
*** manheim has quit IRC13:42
*** lucasxu has quit IRC13:43
egonzalezhawi, where that error occurs? ceph, nova, cinder, glance, gnocchi?13:43
*** lucasxu has joined #openstack-kolla13:43
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod  https://review.openstack.org/46727213:46
*** manheim has joined #openstack-kolla13:46
*** iniazi_ has joined #openstack-kolla13:47
*** manheim has quit IRC13:50
*** eanylin has joined #openstack-kolla13:51
*** manheim has joined #openstack-kolla13:56
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster  https://review.openstack.org/46688113:56
*** manheim has quit IRC13:57
*** powerd has quit IRC13:57
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment.  https://review.openstack.org/46769513:58
*** andybrucenet has joined #openstack-kolla14:00
*** vhosakot has joined #openstack-kolla14:01
*** janki has quit IRC14:02
*** andybrucenet has quit IRC14:03
*** powerd has joined #openstack-kolla14:03
*** mewald has quit IRC14:05
*** mewald has joined #openstack-kolla14:06
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment.  https://review.openstack.org/46769514:07
*** ducttape_ has joined #openstack-kolla14:07
*** jamesPR has joined #openstack-kolla14:10
*** huzhengchuan_ has joined #openstack-kolla14:10
*** mewald has quit IRC14:10
*** manheim has joined #openstack-kolla14:11
*** daidv has quit IRC14:12
*** mewald has joined #openstack-kolla14:12
*** huzhengchuan has joined #openstack-kolla14:14
*** zhubingbing_ has joined #openstack-kolla14:14
*** mgoddard has quit IRC14:14
*** andybrucenet has joined #openstack-kolla14:16
*** powerd_ has joined #openstack-kolla14:17
*** daidv has joined #openstack-kolla14:18
*** manheim has quit IRC14:19
*** powerd has quit IRC14:20
sbezverkandybrucenet: ping14:20
*** powerd_ has quit IRC14:22
*** daidv has quit IRC14:22
hawiegonzalez: i did rm -rf /etc/kolla on storage,compute and controllers, stopped docker, then rm -rf /var/lib/docker/* and then started docker14:25
hawiamazing, now started to deploy again14:25
hawiit stopped deployment after ceph : Fetching Ceph keyrings14:27
hawibut now works fine14:27
*** ducttape_ has quit IRC14:27
hawias i had no daata to loose i removed everything14:28
vhosakothawi: you should've rebooted too before restarting/re-deploying :)14:33
*** ducttape_ has joined #openstack-kolla14:34
*** ducttape_ has quit IRC14:35
*** jemcevoy has quit IRC14:35
*** ducttape_ has joined #openstack-kolla14:35
hawivhosakot: actually, before removing these folders i did touch /forcefsck and tried reboot already :)14:36
hawidunno if forcefsck works on modern linuxes any more14:37
*** srwilkers has joined #openstack-kolla14:37
*** daidv has joined #openstack-kolla14:39
*** lucasxu has quit IRC14:39
*** ducttape_ has quit IRC14:39
*** huzhengchuan_ has quit IRC14:41
*** andybrucenet has quit IRC14:41
*** huzhengchuan has quit IRC14:42
*** hongbin has joined #openstack-kolla14:45
*** chas has joined #openstack-kolla14:45
*** unicell has quit IRC14:47
vhosakothawi: cool, good you rebooted.. almost thought forcefsck is a typo there.. lol ;)14:48
*** mgoddard has joined #openstack-kolla14:49
*** chas has quit IRC14:50
hawinow i do have magic souce for extremely clean deploys ;)14:51
openstackgerritzhubingbing proposed openstack/kolla master: Fix rally create verifier error  https://review.openstack.org/46783714:51
hawifinished it, good14:51
*** korzen has joined #openstack-kolla14:52
*** klowse has joined #openstack-kolla14:53
*** huzhengchuan_ has joined #openstack-kolla14:54
klowsewhen would the 4.0.1 images be on docker hub? Getting: Unknown error message: Tag 4.0.1 not found in repository docker.io...14:54
*** huzhengchuan has joined #openstack-kolla14:55
*** lucasxu has joined #openstack-kolla14:56
jamesPRklowse, you can build your own images ;)14:57
*** zhubingbing_ has quit IRC14:58
klowse:(14:59
*** dave-mccowan has joined #openstack-kolla15:01
*** korzen has quit IRC15:02
*** trozet has quit IRC15:02
*** trozet has joined #openstack-kolla15:04
*** Serlex has quit IRC15:07
*** andybrucenet has joined #openstack-kolla15:07
*** shardy has quit IRC15:09
openstackgerritJuan J. Martínez proposed openstack/kolla master: Fix sed replacement command in Ubuntu's install_docker  https://review.openstack.org/46802615:16
*** andybrucenet has quit IRC15:17
*** yingjun has joined #openstack-kolla15:22
*** manheim has joined #openstack-kolla15:24
*** ducttape_ has joined #openstack-kolla15:24
*** yingjun has quit IRC15:29
*** manheim has quit IRC15:30
*** manheim has joined #openstack-kolla15:30
openstackgerritBernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster  https://review.openstack.org/46688115:32
*** pcaruana has quit IRC15:38
*** yingjun has joined #openstack-kolla15:38
*** shardy has joined #openstack-kolla15:39
*** pcaruana has joined #openstack-kolla15:39
*** skramaja has quit IRC15:43
*** yingjun has quit IRC15:43
*** daidv has quit IRC15:44
*** p6 has quit IRC15:46
*** p6 has joined #openstack-kolla15:47
sdake@klowse which distro are you running?15:52
sdakeklowse i can push centos source/binary - but can't push ubuntu15:52
klowseusing centos-binary...but ive found info on tarballs.openstack.org/kolla/images/ and going to try and use that. seem some more recent fixes in it (4.0.2)?15:53
*** phuongnh has joined #openstack-kolla15:56
*** AzazeL_ has joined #openstack-kolla15:59
*** jascott1 has joined #openstack-kolla15:59
*** jascott1_ has joined #openstack-kolla16:00
chandankumarsdake: hello16:01
chandankumarsdake: do we have some guide on how to run tempest using kolla?16:01
*** lucasxu has quit IRC16:03
*** lucasxu has joined #openstack-kolla16:03
*** jascott1 has quit IRC16:04
*** egonzalez has quit IRC16:05
jascott1_o/16:07
sdake@chandankumar no i don't sorry16:10
jascott1_what up sdake!16:14
sdakesup jascott1_16:14
jascott1_back from my vacation16:15
jascott1_spent 12 hours in O'Hare on the way back. fun.16:16
jascott1_whatd i miss? ;)16:16
openstackgerritTim Rozet proposed openstack/kolla master: Adds opendaylight specific neutron-server container  https://review.openstack.org/46771216:17
*** sayantan_ has joined #openstack-kolla16:18
*** mnasiadka has quit IRC16:24
*** retr0h has quit IRC16:26
*** retr0h has joined #openstack-kolla16:26
sdakejascott1_ i almost took over your review :)16:27
sdakejascott1_ but didn't get a chance16:28
sdakejascott1_ have a look at this:16:28
*** shardy has quit IRC16:28
sdakehttps://review.openstack.org/#/c/464031/16:28
jascott1_yeah saw there was a ton of iteration on  it16:29
*** pcaruana has quit IRC16:30
*** pcaruana has joined #openstack-kolla16:30
*** mnasiadka has joined #openstack-kolla16:31
*** mnasiadka has quit IRC16:36
*** mnasiadka has joined #openstack-kolla16:38
sdakejascott1_ ya - its close to merge16:39
sdakejascott1_ what we need next is a "deploy" playbook16:39
sdakethato nly deploys kolla-kubernetes16:39
sdakejascott1_ you have most of that already done16:40
jascott1_ya16:40
sdakejascott1_ the idea is to wrap these in a container16:40
jascott1_just need a var to control whether k8s is installed or not16:40
jascott1_or just make separatae playbooks16:40
*** mnasiadka has quit IRC16:43
sdakejascott1_ i'd prefer separate playbooks16:44
*** jheroux has joined #openstack-kolla16:44
hrwhttps://review.openstack.org/#/c/467521 anyone? it changes name of python-oslo-vmware to python-oslo.vmware as first one is transitional package while second is real one.16:44
jascott1_sdake i think we can have best of both worlds and be able to call deploy but also have a top level that does it all16:45
*** huzhengchuan_ has quit IRC16:45
sdakejascott1_ yup - include may work here16:45
jascott1_cool I can refactor16:46
*** huzhengchuan has quit IRC16:46
*** chas has joined #openstack-kolla16:46
*** huzhengchuan has joined #openstack-kolla16:48
*** mnasiadka has joined #openstack-kolla16:48
*** jascott1 has joined #openstack-kolla16:49
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403116:49
*** huzhengchuan_ has joined #openstack-kolla16:49
*** jascott1_ has quit IRC16:51
*** chas has quit IRC16:52
*** mnasiadka has quit IRC16:53
*** vhosakot has quit IRC17:00
*** jamesbenson has joined #openstack-kolla17:04
*** kbaegis has joined #openstack-kolla17:04
*** mnasiadka has joined #openstack-kolla17:04
*** jmccarthy has left #openstack-kolla17:06
*** stefan_bo has joined #openstack-kolla17:07
*** sayantan_ has quit IRC17:07
*** sayantan_ has joined #openstack-kolla17:08
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403117:08
*** mnasiadka has quit IRC17:09
*** mnasiadka has joined #openstack-kolla17:10
*** huzhengchuan has quit IRC17:11
*** stefan_bo has quit IRC17:11
*** huzhengchuan has joined #openstack-kolla17:11
*** sayantan_ has quit IRC17:14
*** sayantan_ has joined #openstack-kolla17:14
*** mnasiadka has quit IRC17:15
kbaegisHey guys, what's the standard/best practice for setting up kolla-ansible?17:15
kbaegisi.e. if I don't want to do "sudo pip install -U kolla-ansible"17:15
kbaegisHow do I do that in an isolated venv that won't break my global site-packages?17:16
*** rmart04 has quit IRC17:16
kbaegisideally as a non-root user17:16
*** sayantan_ has quit IRC17:17
*** sayantan_ has joined #openstack-kolla17:17
*** mnasiadka has joined #openstack-kolla17:18
*** sambetts is now known as sambetts|afk17:18
*** lucasxu has quit IRC17:20
*** phuongnh has quit IRC17:21
*** korzen has joined #openstack-kolla17:22
*** sayantan_ has quit IRC17:22
*** sayantan_ has joined #openstack-kolla17:22
*** mnasiadka has quit IRC17:23
jascott1kbaegis if you setup a venv and install does it break your site packages?17:27
kbaegisIt breaks portage17:28
kbaegisFigured it out17:28
*** vhosakot has joined #openstack-kolla17:31
kbaegisjascott1: I was using a venv, but noexec was set on the directory17:31
kbaegisSo it wasn't using pip locally; still using global pip17:32
jascott1oh ok. thats more venv foo than im experiened with (which isnt much) but cool you got it sorted17:32
jascott1foo/fu17:32
*** sayantan_ has quit IRC17:39
*** sayantan_ has joined #openstack-kolla17:40
*** aagate has joined #openstack-kolla17:40
*** vhosakot has quit IRC17:40
kbaegisSweet! :) CI on gentoo with post-merge hooks17:46
kbaegisNow for CD17:46
*** Pavo has quit IRC17:47
*** iniazi has joined #openstack-kolla17:51
*** huzhengchuan_ has quit IRC17:52
*** huzhengchuan_ has joined #openstack-kolla17:52
*** iniazi_ has quit IRC17:53
*** rmart04 has joined #openstack-kolla17:54
*** lpetrut has quit IRC17:54
*** egonzalez has joined #openstack-kolla17:55
*** unicell has joined #openstack-kolla17:58
bmaltais_I am looking at the documentation and I am not clear how to accomplish the following...17:59
bmaltais_I will have a mix of different hardware/VM as part of the environment. Interfaces names will vary amongst them and probably the virtualisation (qemu/kvm) capabilities.18:00
*** ducttape_ has quit IRC18:00
*** sayantan_ has quit IRC18:00
bmaltais_How would I go about setting per node interfaces and per node virtualisation type?18:00
bmaltais_Right now those are applied to all nodes from the central config on the deployment node.18:01
egonzalezbmaltais_ for  interface names and each  var at the inventory, ex: https://docs.openstack.org/developer/kolla-ansible/multinode.html#edit-the-inventory-file18:02
egonzalezbmaltais_ for different  virt types,  add one file for each node  at /etc/kolla/config/nova/{{ inventory_hostname }}/nova.conf18:04
egonzalezbmaltais_ if have more kvm than qemu, just add a file for each qemu18:04
*** manheim has quit IRC18:04
egonzalez[libvirt]18:04
egonzalezvirt_type = qemu18:04
egonzalezcpu_mode = none18:04
egonzalezEOF18:04
bmaltais_egonzalez: OK, so something like "controller1 network_interface="ens160" neutron_external_interface="ens192" someothervarfromglobals.yml=something"18:05
egonzalezbmaltais_ yep18:05
bmaltais_egonzalez: Cool!18:05
*** manheim has joined #openstack-kolla18:06
bmaltais_egonzalez: Nice, I am starting to get the hang of all this. Very nice18:06
*** korzen has quit IRC18:07
*** manheim has quit IRC18:09
*** david-lyle has joined #openstack-kolla18:12
*** eaguilar has joined #openstack-kolla18:13
*** jamesbenson has quit IRC18:15
*** eaguilar has quit IRC18:18
*** david-lyle has quit IRC18:19
emccormickI"m actually having a little trouble doing the overrides in inventory. This statement in all.yml seems to always evaluate to what is in globals.yml and not what is in the inventory18:20
emccormickhostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address']18:20
emccormickI thought it was maybe to do with the way that Ansible messed up the tree structure after 2.1.0.0, but tweaking the structure didn't seem to fix it either18:21
*** dgonzalez has quit IRC18:22
egonzalezemccormick how you  override vars, recently saw that inventory group vars are not overriden but var in inventory hosts do18:22
*** dgonzalez has joined #openstack-kolla18:22
*** korzen has joined #openstack-kolla18:23
emccormickcompute2.cc.osdev.cirrusseven.net network_interface=enp3s0f1 api_interface=enp3s0f1 tunnel_interface=enp3s0f018:23
emccormickis the override statement in my inventory file18:23
emccormickhowever, in globals.yml I'm doing api_interface: "ens33"18:24
emccormick which is what my controllers need18:24
emccormickthat's all that seems to be picking up. On deployment I"m getting this18:24
emccormickfatal: [compute2.cc.osdev.cirrusseven.net]: FAILED! => {"failed": true, "msg": "{{ hostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address'] if orchestration_engine == 'ANSIBLE' else '0.0.0.0' }}: 'dict object' has no attribute u'ansible_ens33'"}18:24
*** ducttape_ has joined #openstack-kolla18:25
emccormickwhich makes sense since it has no ens33, but the override isn't taking18:25
egonzalezwhat task?18:25
*** manheim has joined #openstack-kolla18:25
emccormickcopying nova configs18:25
*** lucasxu has joined #openstack-kolla18:25
emccormickTASK [nova : Copying over nova.conf]18:26
egonzalezemccormick can  you try  changing api_interface_address in group_vars  to hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hostname]['api_interface']['ipv4']['address']18:28
*** manheim has quit IRC18:29
sdakesbezverk what do you make of http://logs.openstack.org/31/464031/36/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/4fca4c1/console.html#_2017-05-25_17_41_25_83924718:29
sbezverksdake: looks like ansible error, why there is error, I have no clue, the kubectl command looks good to me18:30
emccormicksure18:31
sdakejascott1 does | work in command in ansible?18:31
jascott1le pipe?18:31
jascott1yea18:32
jascott1oh wait do you need to use shell?18:32
jascott1if it doesnt work try shell :)18:32
egonzalezyep, need  shell for regex18:32
jascott1i think it does tho18:32
jascott1checking18:32
*** manheim has joined #openstack-kolla18:33
*** mgoddard has quit IRC18:34
jascott1sdake the task with | im using does use `shell` instead `command`18:35
jascott1yup gotta use shell afaik18:36
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes  https://review.openstack.org/46403118:37
*** manheim has quit IRC18:37
sdakejascott1 thanks dude18:37
jascott1np. seems like the rule is it can only execute the first thing so no pipes or operators etc18:37
jascott1(for command)18:38
*** rmart04 has quit IRC18:42
sdakei am using a PV with NFS defined, yet the container running against the PVCs is complaining "read only filesystem"18:45
sdakehowever if i mount the filesystem in linux, I can write files to it18:45
sdakeany ideas why kubernetes doesn't work here?18:45
*** sayantan_ has joined #openstack-kolla18:47
*** chas has joined #openstack-kolla18:47
jascott1sdake you are specifying spec.accessModes for pv? just checking18:51
sdakejascott1 yes18:51
sdakeRWX18:51
sdakei more or less cpied this: https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs18:52
sdakethe only delta is i am using a netapp filer instead of a nfs linux client18:52
*** chas has quit IRC18:52
sdakei am getting an errono30 inside my containers (which indicates read only filesystem)18:53
sdakeits as if kubernetes is mounting it RO18:53
*** mnasiadka has joined #openstack-kolla18:56
emccormickegonzalez: That left me with an error in memcached: failed: [controller1.cc.osdev.cirrusseven.net] (item=memcached) => {"failed": true, "item": "memcached", "msg": "AnsibleUndefinedVariable: {{ hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hostname]['api_interface']['ipv4']['address']] if orchestration_engine == 'ANSIBLE' else '0.0.0.0' }}: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no18:57
emccormickattribute 'ipv4'"}18:57
emccormickwhich I'm guessing is the first place it referenced that ansible_interface_address var18:58
*** srwilkers has quit IRC18:58
egonzalezemccormick sorry, typo hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hosname]['api_interface']]['ipv4']['address']18:58
egonzalezmissing one ] after api_interface18:58
emccormickoops yeah I was fishing for that too hehe thanks18:59
sbezverksdake :  do you use k8s's storage class or pvc point to nfs mount on the host?18:59
Slowersbezverk: do you remember why ipc=host was needed for iscsid?18:59
sdakesbezverk i create 3 pvs, and 3 pvcs18:59
sdakesbezverk https://da.gd/aFFv -> https://paste.fedoraproject.org/paste/vEF5uw794bxSn3A2mZ2uQl5M1UNdIGYhyRLivL9gydE=/19:00
jascott1sdake you do anything with RBAC for nfs provisioner?19:00
sbezverkSlower : we needed to interact  with lvm, otherwise all  lvm related operations were hanging19:00
sdakei applied the aboe patch on top of kolla-kubernetes master19:00
*** mnasiadka has quit IRC19:01
sbezverksdake :  I have not tried with nfs but with iscsi I needed to define storage class, pv and pvc19:02
sbezverksdake : check k8s storage docs19:03
*** mnasiadka has joined #openstack-kolla19:03
*** p6 has quit IRC19:03
*** p6 has joined #openstack-kolla19:04
jascott1sdake seems related https://github.com/kubernetes-incubator/nfs-provisioner/blob/master/docs/usage.md19:04
*** manheim has joined #openstack-kolla19:05
Slowersbezverk: ok cool thanks19:05
rcritI'm working on a source build for the novajoin container and noticed odd permissions with bind mounted files19:07
rcritI'm a complete noob when it comes to containers so maybe it's something I'm doing wrong19:07
emccormickegonzalez: Back to the original error with it picking up the global interface19:07
rcritI have USER nova in the Docker file for now (eventually to be a unique user)19:07
rcritwhen I bind mount files in the uid of the file is nova on the underlying system which doesn't match the uid of the container so my service can't read the config19:08
*** mnasiadka has quit IRC19:08
rcritinvocation like: docker run --network host -v /etc/nova/join.conf:/etc/nova/join.conf:ro ... -ti <id> /bin/bash19:08
kbaegisHey guys, anyone seen "Unexpected Exception: unsupported operand type(s) for %: 'bytes' and 'bytes'" in a while?19:09
*** pcaruana has quit IRC19:09
*** manheim has quit IRC19:09
kbaegisseems to be an issue on kolla-ansible for both python 3.4 and 2.719:09
kbaegisAlmost certainly my fault19:09
egonzalezemccormick please, open a bug, I will investigate more on that19:09
emccormickegonzalez ok will do.19:10
kbaegistraceback: https://hastebin.com/risuwipusi.sql19:11
emccormickegonzalez: I"m actually doing this with the Newton (3.0.3) version, but it looks the same for Ocata and Master. Do you want me to just open it against Master?19:11
*** jamesbenson has joined #openstack-kolla19:12
*** mnasiadka has joined #openstack-kolla19:13
egonzalezemccormick sure19:16
jamesbensonyay 😁 Magnum and murano deployed easy  :-). Now onto AODH and Gnocchi. :-)19:16
sdakesbezverk do you have an example storageclass for nfs?19:17
*** mnasiadka has quit IRC19:18
openstackgerritMerged openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod  https://review.openstack.org/46727219:21
kbaegisDoes kolla-ansible require python 3.5?19:22
*** jamesPR has quit IRC19:22
*** mnasiadka has joined #openstack-kolla19:22
emccormickegonzalez: opened. Thanks19:26
*** manheim has joined #openstack-kolla19:27
*** mnasiadka has quit IRC19:28
*** manheim has quit IRC19:32
*** mnasiadka has joined #openstack-kolla19:33
hrw~curse kubernetes-entrypoint image for being x86-64 only19:35
*** mnasiadka has quit IRC19:38
*** mnasiadka has joined #openstack-kolla19:42
*** mnasiadka has quit IRC19:47
*** shaggydog97 has joined #openstack-kolla19:49
kfox1111sbezverk: https://github.com/kubernetes/kubernetes/pull/4644419:51
kfox1111something to watch.19:51
sdakegetting this error Error: error validating "": error validating data: found invalid field storageClassName for v1.PersistentVolumeSpec19:52
sdakewith this pv/pvc combo: https://da.gd/8tEtc -> https://paste.fedoraproject.org/paste/LuuVKFLW7ai5peuql-S6f15M1UNdIGYhyRLivL9gydE=/19:52
sdakeany ideas?19:52
kfox1111looking...19:53
sdakekubectl get storageclasses19:53
sdakeNAME      TYPE19:53
sdakeslow      kubernetes.io/nfs19:53
sdakethanks kfox111119:53
kfox1111hmm...19:53
*** mnasiadka has joined #openstack-kolla19:54
kfox1111oh.19:54
kfox1111theres a storageClassName in the pv. I'm not sure thats allowed.19:55
sdakehttps://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes19:55
*** bmaltais_ has quit IRC19:55
sdakekfox1111 if i run without a storageclass, the nfs mounted filesystem is mounted read only (errno30 via mariadb)19:57
kfox1111I was thinking the storage class was what provisioned the pv.19:58
jamesbensonSo an open question.  Has anyone made their own unit tests to verify things are working?  I created a small script to deploy the public network, private network, subnets, routers, upload cirros, deploy an vm, and ssh into it... but I'm thinking about aodh, marano, etc?19:58
jamesbensonI'm curious to see what others have done, if anything.19:58
*** mnasiadka has quit IRC19:59
sdakekfox1111 unclear - i've read these docs about 20 times, and i cna't find a clear answer19:59
kfox1111sdake: ah. I see...19:59
sdakekfox1111 what is odd is on different hardware,  I can mount the nfs volume correctly19:59
kfox1111what version of k8s you using?19:59
kfox1111I think storageClassName on the pv is a 1.6 feature.19:59
sdake1.5.219:59
kfox1111it may be an annotation of:19:59
kfox1111volume.beta.kubernetes.io/storage-class19:59
kfox1111before that.19:59
*** fguillot has quit IRC20:00
sdakecan you hand-feed me the syntax plz :)20:00
kfox1111https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes20:00
kfox1111uh....20:00
kfox1111metadata:20:00
kfox1111  annotations:20:00
kfox1111    "volume.beta.kubernetes.io/storage-class": slow20:00
sdakethanks20:00
kfox1111np20:01
jascott1jamesbenson I have wondered the same. I think we should be able to use existing openstack tools like rally20:01
jamesbensonbut on private clouds?20:01
jamesbensonah, sorry20:02
jamesbensonjascott1: just read about rally... I was thinking it was something else.20:02
*** mnasiadka has joined #openstack-kolla20:02
*** mgoddard has joined #openstack-kolla20:03
jascott1jamesbenson iirc rally is a nice executor for temptest20:03
jascott1and others?20:03
jamesbensonjascott1: Do you know if this is what Mirantis did for the health checks?  Sounds possible.20:04
jascott1health check to me implies on going "are you still there" pings or requests20:05
jascott1this sounds like smoke test20:05
jascott1but no i dont know what mirantis did20:05
kfox1111are you talknig about fuels ui for the checks it did?20:06
kbaegisFigured it out. Needed python3.5, pip3.520:06
kbaegis:/20:06
kfox1111if so, I'm pretty sure that was tempest.20:06
*** mnasiadka has quit IRC20:07
jamesbensonjascott1: yeah, post deployment, you can do a health check: requests image list, create instance flavor, keypair, volumes, launch instance with file injection... HA tests, ceilometer tests to check events and notifications, etc.20:08
jamesbensonjascott1: I'm looking at rally, and enjoying it.  I think I'll deploy that and try to run some smoke tests....20:08
jascott1hopefully we will add some docs about using these tools to test a kk8s deployment20:09
jascott1seems like we could also run in gate even if non-voting20:11
*** david-lyle has joined #openstack-kolla20:12
*** mnasiadka has joined #openstack-kolla20:12
*** korzen has quit IRC20:13
*** david-lyle has quit IRC20:17
*** mnasiadka has quit IRC20:17
jascott1sdake you are merging destroy under ansible/ ? i was thinking more tools/deploy or something as to not confuse it with the older ansible stuff20:19
sdakejascott1 i think we should have several top level plays20:19
sdakedestroy, config, upgrade, deploy, etc20:19
jascott1agree20:19
jascott1om only asking about the path20:19
*** sayantan_ has quit IRC20:20
sdakeyup i think it all belongs in ansible :)20:20
*** sayantan_ has joined #openstack-kolla20:20
jascott1sdake im confused about what in ansible/ is obsolete... arent those service roles no longer needed?20:20
*** lucasxu has quit IRC20:21
*** lucasxu has joined #openstack-kolla20:22
sdakejascott1 those are config roles20:22
jascott1sounds important ;)20:23
sdakekfox1111 i get this result when launching my cluster with that nfs provisioner20:23
sdakeNAME       STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE20:23
sdakeglance     Pending                                      2m20:23
jamesbensonjascott1: kk8s deployment20:24
jamesbenson?20:24
jascott1kolla-kubernetes (kk8s)20:24
jamesbensonthx20:24
*** mnasiadka has joined #openstack-kolla20:25
openstackgerritDavid Moreau Simard proposed openstack/kolla-ansible master: gzip the ARA report after generating it  https://review.openstack.org/46815520:26
*** mgoddard has quit IRC20:27
dmsimardcould I get some quick reviews on ^ ? to alleviate load on logs.o.,o20:28
*** mnasiadka has quit IRC20:30
*** mnasiadka has joined #openstack-kolla20:33
*** mewald has quit IRC20:37
kfox1111sdake: not sure. I have not used taht functionality at all yet. :/20:38
*** mnasiadka has quit IRC20:38
sdakekfox1111 sounds fair - mind i bounce some debug ideas off you as I have them - even though you have no experinece with it?20:38
kfox1111sure. :)20:38
*** kbyrne has quit IRC20:38
sdakekfox1111 specifically is a provisioner required to get nfs to work20:38
sdakekfox1111 i got nfs to work on a different cluster connected to NFS served up via linux20:39
sdakehowever NFS served up via netapp returns read only filesystem20:39
sdakei.e. the first case works -> linux serving nfs20:39
kfox1111thats a bit weird.20:39
sdakethe second case fails -> netapp serving up nfs20:39
sdakeagreed i  don't understand it at all20:39
kfox1111in btoh cases, you manually created all the pv's?20:39
sdakekfox1111 i deployed with service charts - however hacked them to run with my pvs20:40
*** manheim has joined #openstack-kolla20:41
sdakedescribe of pvs: https://da.gd/QftD8 -> https://paste.fedoraproject.org/paste/J0-7OBxbquv8VZ4cwPjMTl5M1UNdIGYhyRLivL9gydE=/20:42
kfox1111we should make the pv charts optional so users can just make them themselves.20:42
sdakenote ReadONly is false20:42
*** mnasiadka has joined #openstack-kolla20:42
kfox1111were they created in the netapp as rw?20:42
sdakehttps://da.gd/2LWv -> https://paste.fedoraproject.org/paste/zSvM-eT20xp5fsONKa9biF5M1UNdIGYhyRLivL9gydE=/20:43
sdakekfox1111 i can mount the netapp from linux just fine and write files20:43
sdakethat second paste is the pvcs20:44
sbezverkkfox1111 : thanks, let's hope it will go through20:44
kfox1111sbezverk: +120:45
sdakehttps://da.gd/ncuBB -> https://paste.fedoraproject.org/paste/eKLoilifvj3Mylc7nuxlj15M1UNdIGYhyRLivL9gydE=/20:45
sdakekfox1111 you will see that mariadb thinks the filesystem is readonly20:45
jascott1sdake what is the reason behind using `hosts: localhost` in destroy playbook20:45
kfox1111sdake: and the pv view?20:45
sdakehttps://paste.fedoraproject.org/paste/J0-7OBxbquv8VZ4cwPjMTl5M1UNdIGYhyRLivL9gydE=/20:45
*** manheim has quit IRC20:45
sbezverkkfox1111: could you ack openvswitch change? it seems rabbitmq-mgmt is dependent on it..20:46
kfox1111sbezverk: did a while ago.20:46
kfox1111sdake: very strange...20:46
kfox1111selinux disabled?20:46
sbezverkkfox1111 : it got lost I think20:46
sdakekfox1111 yup20:46
kfox1111sbezverk: k. got the link handy?20:46
*** athomas has quit IRC20:46
*** mnasiadka has quit IRC20:47
kfox1111sdake: oh.... um... idea....20:47
sdakeall ideas = winning :)20:47
kfox1111some remote file servers expect unix uid's to exist on them or to an ldap instance referenced by them.20:47
*** powerd has joined #openstack-kolla20:47
kfox1111or, it has root squash turned on.20:47
sdake[admin@h7-node0 kolla]$ ls -ld mariadb20:48
sdakedrwxrwxrwx. 2 42434 42434 4096 May 25 14:52 mariadb20:48
*** chas has joined #openstack-kolla20:48
sdakethat is the mariadb PID in use20:48
kfox1111sdake: so... lets verify...20:49
*** shaggydog97 has quit IRC20:49
kfox1111add that uid to your host, su to it, and see if it can access the nfs mount directly.20:49
sdakecool20:49
kfox1111the other thing to check is if rootsquash is enabled on the nfs mount.20:49
kfox1111if the init container is runnign root, it may map the user to 'nobody' or an equiv,20:49
kfox1111and it will look read only to th root user.20:50
sdake[admin@h7-node0 mariadb]$ touch a20:50
sdake[admin@h7-node0 mariadb]$ ls -l20:50
sdaketotal 020:50
sdake-rw-rw-r--. 1 admin admin 0 May 25 16:49 a20:50
sdake[admin@h7-node0 mariadb]$ pwd20:50
kolla-slack1<sbezverk> @kfox1111: https://kubernetes.slack.com/team/kfox111120:50
sbezverkkfox1111: sorry you were right20:51
kfox1111sbezverk: no worries.20:51
kfox1111sdake: admin on the applience != root using the export.20:51
kfox1111\usuauly its the latter that gets squashed.20:51
*** chas has quit IRC20:52
*** mnasiadka has joined #openstack-kolla20:53
sdakei can touch a file as uid=0 and that works20:54
sdakeand the resulting uid = 020:54
sdakeso think its not root squash20:54
kfox1111hmm.. ok. not root squash then.20:54
sbezverksdake : there is always a chance to be a valid k8s bug20:55
sbezverkI would suggest to check with sig-storage folks20:55
sdakesbezverk although it works with nfs from a linux host?20:55
sbezverkespecially because of that20:56
sbezverkit means kubelet mounts it incorrectl;y20:56
sbezverkor using wrong credentials20:56
kfox1111I know lustre moutns get grumpy if the uid's used to write to it don't exist in the pw file or getent of the mds.20:57
kfox1111so the other test is to make that uid a user on the box, then mount it locally, swithc to the user and test.20:57
*** mnasiadka has quit IRC20:58
sdakekfox1111 i did that as well20:58
sdakeone thing I noticed is the netapp only has nfsv320:58
sdakeis there a way to override the nfs version in the pv?20:58
kfox1111sdake: did it work?20:58
sdakeyup that worked20:58
kfox1111hmm...20:59
kfox1111I submitted a k8s issue for nfs mount options months ago.20:59
kfox1111I know it was beign worked on, but I'm not sure it made it yet.20:59
* sdake zugs20:59
kfox1111sdake. it was. but time to upgrade. :)21:00
kfox1111it was fixed in 1.621:00
kfox1111https://github.com/kubernetes/kubernetes.github.io/pull/274321:00
*** mgoddard has joined #openstack-kolla21:00
*** mnasiadka has joined #openstack-kolla21:03
*** itlinux has joined #openstack-kolla21:05
*** ducttape_ has quit IRC21:06
*** jtriley has quit IRC21:06
*** mnasiadka has quit IRC21:07
*** jascott1 has quit IRC21:08
*** jascott1 has joined #openstack-kolla21:08
*** chas has joined #openstack-kolla21:09
*** manheim has joined #openstack-kolla21:10
*** mgoddard has quit IRC21:11
*** mnasiadka has joined #openstack-kolla21:12
*** jascott1 has quit IRC21:13
*** chas has quit IRC21:14
*** manheim has quit IRC21:15
*** mnasiadka has quit IRC21:17
*** mnasiadka has joined #openstack-kolla21:22
*** m1dev has joined #openstack-kolla21:23
*** jheroux has quit IRC21:25
*** mnasiadka has quit IRC21:27
*** ddyer has quit IRC21:29
rwellumkfox1111: question about br-ex - typically do I have to create this bridge or is this done when running through the deployment?21:29
*** chas has joined #openstack-kolla21:30
kfox1111rwellum: its configurable in the openvswitch microcharts.21:30
kfox1111I tipically create it outside the deployment, but I also tend to run openvswitch on the host outside of k8s anyway.21:31
kfox1111I've only got one nic, so use openvswitch to do the vlan stuff for the host as well.21:31
rwellumOk because we use it in the cloud.yaml - for keepalived to bind VIP to it - but not sure if I have to create it and attach my neutron interface to it or not.21:32
*** mnasiadka has joined #openstack-kolla21:33
*** dave-mccowan has quit IRC21:33
*** mgoddard has joined #openstack-kolla21:34
*** chas has quit IRC21:34
*** mnasiadka has quit IRC21:37
*** manheim has joined #openstack-kolla21:38
*** ddyer has joined #openstack-kolla21:39
*** lucasxu has quit IRC21:39
*** chas has joined #openstack-kolla21:40
*** jamesPR has joined #openstack-kolla21:40
*** manheim has quit IRC21:42
*** itlinux has quit IRC21:42
*** mnasiadka has joined #openstack-kolla21:43
*** mnasiadka has quit IRC21:47
*** Pavo has joined #openstack-kolla21:50
sbezverkrwellum : do not forget then to run job to bring br-ex up, otherwise keepalived will fail21:51
*** mnasiadka has joined #openstack-kolla21:52
*** iniazi has quit IRC21:53
*** chas has quit IRC21:54
*** chas has joined #openstack-kolla21:54
*** mgoddard has quit IRC21:55
*** Pavo has quit IRC21:56
*** mnasiadka has quit IRC21:56
*** jamesbenson has quit IRC21:58
*** chas has quit IRC21:59
*** manheim has joined #openstack-kolla22:04
*** manheim has quit IRC22:04
*** manheim_ has joined #openstack-kolla22:06
*** jascott1 has joined #openstack-kolla22:09
emccormick22:14
*** jascott1 has quit IRC22:14
*** chas has joined #openstack-kolla22:15
*** mnasiadka has joined #openstack-kolla22:19
*** chas has quit IRC22:20
sdakekfox1111 i hae confirmed it is not a nfsv4 / nfsv3 issue by forcing nfsv3 on the environment without the netapp22:20
*** itlinux has joined #openstack-kolla22:22
*** mnasiadka has quit IRC22:24
*** Pavo has joined #openstack-kolla22:25
*** Pavo has quit IRC22:28
sbezverksdake : file a bug somebody from sig-storage will look at it22:39
sdakei found the root cause - there is a .snapshot directory on the netapp which is a read-only filesystem22:40
sdakechmod tries to chmod it during startup - which implodes22:40
sdakesbezverk http://paste.openstack.org/show/610688/22:42
*** powerd has quit IRC22:42
kfox1111sdake: fun.22:43
sbezverksdake: what creates .snapshot and why mariadb see it? could it be that nfs mount is not correctly defined?22:45
sdakehttps://library.netapp.com/ecmdocs/ECMP1368826/html/GUID-56684CAA-FA05-450C-93D4-C12D8A695F07.html22:46
sbezverksdake: you should probably disable snapshot completely22:47
sdakeif it were my choice..22:47
*** Pavo has joined #openstack-kolla22:47
*** jascott1 has joined #openstack-kolla22:47
*** ducttape_ has joined #openstack-kolla22:48
sbezverksdake: hm, it is odd if they force you to use it even if you do not need22:48
kfox1111sdake: can you do a subpath in the pv?22:48
*** unicell has quit IRC22:49
kfox1111ie, make netapp:/foo volume, then in the pv, reference netapp:/foo/data or something as the root.22:49
*** unicell1 has joined #openstack-kolla22:49
sdakekfox1111 / is what is exported from the netapp22:49
sdakekfox1111 i htink every mount to anything in that path creates a new subvolume22:49
kfox1111yeah. but I'm saying imoprt something under that in the pv.22:49
*** iniazi has joined #openstack-kolla22:49
sdakei am doing that22:49
*** huzhengchuan_ has quit IRC22:49
*** huzhengchuan has quit IRC22:49
kfox1111and each subdir has a .snapshot thingy?22:50
sdakeeach one a pv creates yes22:50
kfox1111the pv shoudln't be creating a volume though...22:50
*** Pavo has quit IRC22:50
sdakewhen it connects to the netapp, it creates a volume22:50
kfox1111oh. you have a netapp dynamic provisioner?22:50
sdakeno22:50
sdake / is exported from the netapp22:51
sdakeif you mount /vol/abc22:51
sdakeabc gets created with a .snapshot read only filesystem22:51
sbezverkvol options volname nosnap [on|off]22:51
kfox1111that seems broken.22:51
kfox1111yeah. that thingy. do that. :)22:51
sdakesbezverk you assume I have access to the netapp :)22:51
sbezverkoh I thought it is in your home lab22:52
*** Pavo has joined #openstack-kolla22:52
*** powerd has joined #openstack-kolla22:53
kbaegisIf anyone has time, new blocker on gentoo: https://hastebin.com/owomofepib.rb22:53
sdakeI suspect someone has already run into this: commit fe61742d1c6ba62ba98cc0f97f0d1faafe29139f22:54
sdakeAuthor: Jeffrey Zhang <zhang.lei.fly@gmail.com>22:54
sdakeDate:   Tue May 9 11:18:39 2017 +080022:54
sdake    Catch exception during calling os.chmod in set_configs.py22:54
sdake22:54
sdake    set_perms may fail in some case, but it is not critical. Just leave a22:54
sdake    warning rather than exit.22:54
kbaegisVery nearly have non-root build/deploy using gentoo22:54
kbaegisWhat does basestring refer to?22:55
sbezverksdake: I got mac :)22:55
*** chas has joined #openstack-kolla22:56
*** chas has quit IRC23:01
*** egonzalez has quit IRC23:06
*** itlinux has quit IRC23:07
*** Pavo has quit IRC23:08
*** manheim_ has quit IRC23:10
*** rhallisey has quit IRC23:15
*** ducttape_ has quit IRC23:15
*** ducttape_ has joined #openstack-kolla23:16
*** rhallisey has joined #openstack-kolla23:28
*** ducttape_ has quit IRC23:30
*** powerd has quit IRC23:33
*** rhallisey has quit IRC23:36
*** chas has joined #openstack-kolla23:37
*** rwellum has quit IRC23:41
*** sayantan_ has quit IRC23:42
*** chas has quit IRC23:42
*** sayantan_ has joined #openstack-kolla23:42
*** manheim has joined #openstack-kolla23:46
sean-k-mooneyhas anyone had issues pushing images to a registry http://paste.openstack.org/show/610691/23:49
*** yingjun has joined #openstack-kolla23:49
sdakeshould this work:     mariadb:23:50
sdake       all:23:50
sdake         image_full: "docker.io/kolla/centos-source-mariadb:4.0.2"23:50
sdake23:50
sdake(in cloud.yaml)23:50
*** manheim has quit IRC23:50
kfox1111sdake: I think so.23:51
kfox1111do a --dry-run --debug to see.23:51
*** kbaegis has quit IRC23:54
*** eaguilar has joined #openstack-kolla23:57
*** hongbin has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!