| *** spsurya has joined #openstack-kolla | 00:05 | |
| *** sbezverk has joined #openstack-kolla | 00:06 | |
| *** hongbin has quit IRC | 00:10 | |
| *** kolla-slack has quit IRC | 00:11 | |
| *** kolla-slack1 has joined #openstack-kolla | 00:11 | |
| *** huzhengchuan has quit IRC | 00:23 | |
| *** huzhengchuan_ has quit IRC | 00:23 | |
| *** jtriley has joined #openstack-kolla | 00:23 | |
| *** chas has joined #openstack-kolla | 00:24 | |
| *** huzhengchuan has joined #openstack-kolla | 00:26 | |
| *** eanylin has joined #openstack-kolla | 00:26 | |
| *** huzhengchuan_ has joined #openstack-kolla | 00:27 | |
| *** chas has quit IRC | 00:29 | |
| *** sgrasley has joined #openstack-kolla | 00:30 | |
| *** manheim has joined #openstack-kolla | 00:31 | |
| *** huzhengchuan_ has quit IRC | 00:31 | |
| *** huzhengchuan has quit IRC | 00:31 | |
| *** manheim has quit IRC | 00:35 | |
| *** gfidente has quit IRC | 00:35 | |
| *** duonghq_ is now known as duonghq | 00:40 | |
| duonghq | morning guys | 00:41 |
|---|---|---|
| Seyong | good morning | 00:42 |
| *** huzhengchuan_ has joined #openstack-kolla | 00:44 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles) https://review.openstack.org/398684 | 00:45 |
| *** huzhengchuan has joined #openstack-kolla | 00:48 | |
| *** ducttape_ has joined #openstack-kolla | 00:54 | |
| *** jtriley has quit IRC | 00:57 | |
| *** cuongnv has joined #openstack-kolla | 01:02 | |
| *** n0isyn0i1e has joined #openstack-kolla | 01:12 | |
| openstackgerrit | zhubingbing proposed openstack/kolla master: test kolla gate https://review.openstack.org/467807 | 01:13 |
| *** qiliang27 has joined #openstack-kolla | 01:14 | |
| *** n0isyn0ise has quit IRC | 01:14 | |
| *** ducttape_ has quit IRC | 01:19 | |
| *** ducttape_ has joined #openstack-kolla | 01:20 | |
| *** ducttap__ has joined #openstack-kolla | 01:25 | |
| *** ducttap__ has quit IRC | 01:27 | |
| *** manheim has joined #openstack-kolla | 01:28 | |
| *** ducttap__ has joined #openstack-kolla | 01:28 | |
| *** ducttap__ has quit IRC | 01:28 | |
| *** ducttape_ has quit IRC | 01:28 | |
| openstackgerrit | Andrew Bruce proposed openstack/kolla-kubernetes master: openvswitch not required to be kernel module https://review.openstack.org/467769 | 01:32 |
| *** hrw has quit IRC | 01:32 | |
| *** sgrasley has quit IRC | 01:33 | |
| *** manheim has quit IRC | 01:33 | |
| *** hrw has joined #openstack-kolla | 01:34 | |
| *** manheim has joined #openstack-kolla | 01:37 | |
| *** manheim has quit IRC | 01:41 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only necessary tasks (default roles) https://review.openstack.org/398684 | 01:42 |
| *** sbezverk has quit IRC | 01:42 | |
| *** tovin07_ has joined #openstack-kolla | 01:51 | |
| *** qiliang27 has quit IRC | 01:54 | |
| *** unicell has quit IRC | 01:58 | |
| bmaltais | NOticed something. Was trying to add new compute node to kolla-ansible. Added it to inventory file and ran kolla-ansible bootstrap-servers | 02:00 |
| *** stefan_bo has joined #openstack-kolla | 02:01 | |
| bmaltais | The result was a non functional environment... | 02:02 |
| bmaltais | I think the bootstrap process touched existing node instead on just applying changes to new one and broke it as a result of restarting docker | 02:03 |
| *** zhurong has joined #openstack-kolla | 02:05 | |
| *** sbezverk has joined #openstack-kolla | 02:07 | |
| *** sbezverk has quit IRC | 02:15 | |
| bmaltais | I would suggest that the bootstrap process 1st check if /etc/kolla exist on the node and if it does it should just skip it. | 02:21 |
| *** chas has joined #openstack-kolla | 02:24 | |
| *** huzhengchuan has quit IRC | 02:26 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/398685 | 02:26 |
| *** huzhengchuan_ has quit IRC | 02:26 | |
| *** chas has quit IRC | 02:30 | |
| *** lucasxu has joined #openstack-kolla | 02:33 | |
| *** xinliang has quit IRC | 02:34 | |
| *** itlinux has joined #openstack-kolla | 02:37 | |
| *** huzhengchuan_ has joined #openstack-kolla | 02:38 | |
| *** huzhengchuan has joined #openstack-kolla | 02:40 | |
| *** zhuzeyu has joined #openstack-kolla | 02:45 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix etc/hosts format with baremetal role https://review.openstack.org/467827 | 02:46 |
| *** xinliang has joined #openstack-kolla | 02:47 | |
| *** xinliang has quit IRC | 02:47 | |
| *** xinliang has joined #openstack-kolla | 02:47 | |
| *** caowei has joined #openstack-kolla | 02:50 | |
| *** itlinux has quit IRC | 03:03 | |
| *** lucasxu has quit IRC | 03:06 | |
| *** lucasxu has joined #openstack-kolla | 03:08 | |
| *** srwilkers has quit IRC | 03:09 | |
| openstackgerrit | zhubingbing proposed openstack/kolla master: Fix rally create verifier error https://review.openstack.org/467837 | 03:14 |
| *** ducttape_ has joined #openstack-kolla | 03:16 | |
| *** ducttap__ has joined #openstack-kolla | 03:17 | |
| *** ducttape_ has quit IRC | 03:17 | |
| *** lucasxu has quit IRC | 03:18 | |
| *** yuanying has joined #openstack-kolla | 03:24 | |
| *** zhubingbing has quit IRC | 03:25 | |
| *** manheim has joined #openstack-kolla | 03:26 | |
| *** spsurya has quit IRC | 03:32 | |
| *** stefan_bo has quit IRC | 03:37 | |
| *** zhubingbing_ has joined #openstack-kolla | 03:40 | |
| *** ducttap__ has quit IRC | 03:51 | |
| *** daidv_ has joined #openstack-kolla | 03:55 | |
| *** manheim has quit IRC | 04:00 | |
| *** manheim has joined #openstack-kolla | 04:02 | |
| *** manheim has quit IRC | 04:06 | |
| *** ducttape_ has joined #openstack-kolla | 04:08 | |
| *** zhurong has quit IRC | 04:08 | |
| *** ducttape_ has quit IRC | 04:12 | |
| *** zhurong has joined #openstack-kolla | 04:16 | |
| *** unicell has joined #openstack-kolla | 04:19 | |
| *** chas has joined #openstack-kolla | 04:25 | |
| *** skramaja has joined #openstack-kolla | 04:25 | |
| *** manheim has joined #openstack-kolla | 04:29 | |
| *** yingjun has joined #openstack-kolla | 04:30 | |
| *** chas has quit IRC | 04:30 | |
| *** janki has joined #openstack-kolla | 04:32 | |
| *** manheim has quit IRC | 04:34 | |
| *** manheim has joined #openstack-kolla | 04:36 | |
| *** manheim has quit IRC | 04:41 | |
| *** manheim has joined #openstack-kolla | 04:44 | |
| *** unicell1 has joined #openstack-kolla | 04:47 | |
| *** unicell has quit IRC | 04:47 | |
| *** manheim has quit IRC | 04:48 | |
| *** manheim has joined #openstack-kolla | 04:50 | |
| *** manheim has quit IRC | 04:55 | |
| *** manheim has joined #openstack-kolla | 05:03 | |
| *** manheim has quit IRC | 05:07 | |
| *** zhuzeyu has quit IRC | 05:13 | |
| *** zhuzeyu has joined #openstack-kolla | 05:13 | |
| *** zhuzeyu has quit IRC | 05:20 | |
| *** rwsu has joined #openstack-kolla | 05:27 | |
| *** stefan_bo has joined #openstack-kolla | 05:51 | |
| *** korzen has joined #openstack-kolla | 05:52 | |
| *** sayantan_ has quit IRC | 06:02 | |
| *** manheim has joined #openstack-kolla | 06:03 | |
| *** caowei_ has joined #openstack-kolla | 06:15 | |
| *** caowei has quit IRC | 06:15 | |
| *** caowei_ is now known as caowei | 06:15 | |
| *** aagate has quit IRC | 06:17 | |
| *** pcaruana has joined #openstack-kolla | 06:20 | |
| *** stefan_bo has quit IRC | 06:23 | |
| manheim | hello | 06:24 |
| *** chas has joined #openstack-kolla | 06:26 | |
| openstackgerrit | Juan Badia Payno proposed openstack/kolla master: WIP - Modified fluentd Image https://review.openstack.org/467219 | 06:30 |
| *** chas has quit IRC | 06:30 | |
| *** sayantan_ has joined #openstack-kolla | 06:34 | |
| *** sayantan_ has quit IRC | 06:47 | |
| *** sayantan_ has joined #openstack-kolla | 06:47 | |
| *** sayantan_ has quit IRC | 06:47 | |
| *** sayantan_ has joined #openstack-kolla | 06:48 | |
| *** stefan_bo has joined #openstack-kolla | 06:53 | |
| *** mnasiadka has joined #openstack-kolla | 06:59 | |
| *** shardy has joined #openstack-kolla | 07:10 | |
| *** dixiaoli has joined #openstack-kolla | 07:13 | |
| *** sayantan_ has quit IRC | 07:13 | |
| *** manheim has quit IRC | 07:22 | |
| masber | I just installed openstack + magnum using kolla-ansible. How should I use magnum cli? | 07:22 |
| masber | do I need to run docker exec magnum-api magnum ...? | 07:22 |
| *** Serlex has joined #openstack-kolla | 07:25 | |
| *** manheim has joined #openstack-kolla | 07:25 | |
| *** imcsk8 has quit IRC | 07:25 | |
| *** manheim has quit IRC | 07:27 | |
| *** manheim has joined #openstack-kolla | 07:27 | |
| *** zhurong has quit IRC | 07:30 | |
| *** manheim has quit IRC | 07:31 | |
| *** imcsk8 has joined #openstack-kolla | 07:34 | |
| *** zhubingbing_ has quit IRC | 07:36 | |
| *** manheim has joined #openstack-kolla | 07:38 | |
| *** manheim has quit IRC | 07:40 | |
| *** manheim has joined #openstack-kolla | 07:43 | |
| *** egonzalez has joined #openstack-kolla | 07:44 | |
| *** zhurong has joined #openstack-kolla | 07:45 | |
| *** manheim has quit IRC | 07:47 | |
| *** manheim has joined #openstack-kolla | 07:50 | |
| *** mgoddard has joined #openstack-kolla | 07:51 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix etc/hosts format with baremetal role https://review.openstack.org/467827 | 07:51 |
| *** skramaja has quit IRC | 07:52 | |
| *** tvignaud_pto has quit IRC | 07:58 | |
| *** daidv_ has quit IRC | 08:03 | |
| *** manheim has quit IRC | 08:06 | |
| *** stefan_bo has quit IRC | 08:06 | |
| *** stefan_bo has joined #openstack-kolla | 08:07 | |
| *** manheim has joined #openstack-kolla | 08:10 | |
| *** gfidente has joined #openstack-kolla | 08:14 | |
| *** manheim has quit IRC | 08:14 | |
| *** tvignaud_pto has joined #openstack-kolla | 08:15 | |
| *** zhubingbing_ has joined #openstack-kolla | 08:18 | |
| *** manheim has joined #openstack-kolla | 08:19 | |
| *** manheim has quit IRC | 08:23 | |
| *** chas has joined #openstack-kolla | 08:27 | |
| *** manheim has joined #openstack-kolla | 08:29 | |
| *** athomas has joined #openstack-kolla | 08:29 | |
| *** pbourke has quit IRC | 08:30 | |
| *** pbourke has joined #openstack-kolla | 08:30 | |
| *** chas has quit IRC | 08:31 | |
| *** manheim has quit IRC | 08:33 | |
| *** lpetrut has joined #openstack-kolla | 08:37 | |
| *** unicell1 has quit IRC | 08:46 | |
| *** unicell has joined #openstack-kolla | 08:46 | |
| *** manheim has joined #openstack-kolla | 08:47 | |
| *** mgoddard has quit IRC | 08:53 | |
| *** zhurong has quit IRC | 08:54 | |
| hawi | is there something i have to do with external elasticsearch, did enable_central_logging: "no" and added elasticsearch_address, but nothing | 08:57 |
| *** magicboiz has quit IRC | 09:00 | |
| egonzalez | hawi, afik the only requirement is that elasticsearch_address != kolla_internal_vip_address | 09:01 |
| *** jmccarthy has joined #openstack-kolla | 09:03 | |
| *** zhurong has joined #openstack-kolla | 09:04 | |
| openstackgerrit | Numan Siddique proposed openstack/kolla master: Support openvswitch-ovn docker images https://review.openstack.org/459179 | 09:04 |
| *** manheim has quit IRC | 09:11 | |
| *** manheim has joined #openstack-kolla | 09:13 | |
| *** zhubingbing_ has quit IRC | 09:18 | |
| *** mewald has joined #openstack-kolla | 09:22 | |
| *** duonghq has quit IRC | 09:30 | |
| *** zhurong has quit IRC | 09:35 | |
| *** yingjun has quit IRC | 09:37 | |
| *** sambetts|afk is now known as sambetts | 09:46 | |
| *** manheim has quit IRC | 09:48 | |
| *** manheim has joined #openstack-kolla | 09:53 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod https://review.openstack.org/467272 | 09:54 |
| *** manheim has quit IRC | 09:57 | |
| *** rmart04 has joined #openstack-kolla | 09:58 | |
| *** tovin07_ has quit IRC | 10:00 | |
| hrw | morning | 10:08 |
| hrw | upgraded to 64 cpu cores aarch64 machine. will see how kolla goes on it | 10:09 |
| *** ducttape_ has joined #openstack-kolla | 10:13 | |
| *** cuongnv has quit IRC | 10:17 | |
| *** ducttape_ has quit IRC | 10:18 | |
| *** manheim has joined #openstack-kolla | 10:26 | |
| *** chas has joined #openstack-kolla | 10:28 | |
| *** caowei has quit IRC | 10:29 | |
| *** manheim has quit IRC | 10:30 | |
| *** chas has quit IRC | 10:33 | |
| *** dixiaoli has quit IRC | 10:39 | |
| egonzalez | can someone try a mariadb upgrade from ocata to master? newton-ocata works fine but to master containers keep restarting https://bugs.launchpad.net/kolla-ansible/+bug/1692507 | 10:40 |
| openstack | Launchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New] | 10:40 |
| *** manheim has joined #openstack-kolla | 10:44 | |
| *** manheim has quit IRC | 10:48 | |
| *** mnasiadka has quit IRC | 10:57 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster https://review.openstack.org/466881 | 11:02 |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment. This should help new users get started quickly by having all of the needed configuration to accomplish the task in one place. https://review.openstack.org/467695 | 11:04 |
| *** skramaja has joined #openstack-kolla | 11:04 | |
| *** stefan_bo has quit IRC | 11:09 | |
| *** rhallisey has joined #openstack-kolla | 11:12 | |
| sean-k-mooney | egonzalez: are you seeing that on ubuntu ? | 11:15 |
| egonzalez | sean-k-mooney, centos source | 11:15 |
| sean-k-mooney | egonzalez: well unless we backport the pinning i did for master i think ocata is still pointing to 10.0 not 10.0.30 | 11:16 |
| *** manheim has joined #openstack-kolla | 11:17 | |
| egonzalez | sean-k-mooney, used ocata images from dockerhub with mariadb 10.0.30 | 11:17 |
| sean-k-mooney | egonzalez: whats the workflow to upgrade? build both images, kolla-ansible deploy --tags mariadb, kolla-ansable upgrade --tags mariadb? | 11:17 |
| *** manheim has quit IRC | 11:18 | |
| *** manheim has joined #openstack-kolla | 11:18 | |
| egonzalez | kolla-ansible deploy --tags haproxy,mariadb, change version at globals.yml and kolla-ansible upgrade --tags haproxy,mariadb | 11:18 |
| egonzalez | with multinode, probably allinone work | 11:19 |
| sean-k-mooney | ok cool ill try that and see what happens. never tried upgradeing yet | 11:19 |
| sean-k-mooney | ah mult not that will take me a little longer to set up | 11:19 |
| egonzalez | allinone works | 11:26 |
| *** yingjun has joined #openstack-kolla | 11:27 | |
| *** n0isyn0i1e is now known as n0isyn0ise | 11:28 | |
| *** mgoddard has joined #openstack-kolla | 11:29 | |
| *** manheim has quit IRC | 11:30 | |
| *** magicboiz has joined #openstack-kolla | 11:32 | |
| *** magicboiz has quit IRC | 11:37 | |
| jmccarthy | By chance, anyone run into a complaint like this before with TASK [neutron : Running Neutron bootstrap container] ? | 12:00 |
| jmccarthy | File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/conf/common.py", line 101, in <module> help=_("Hostname to be used by the Neutron server, " File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 1429, in __init__ **kwargs) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 904, in __init__ self._check_default() File "/var/lib/kolla/venv/lib/python2.7/site-pac | 12:00 |
| *** sgrasley has joined #openstack-kolla | 12:02 | |
| *** bmaltais_ has joined #openstack-kolla | 12:03 | |
| bmaltais_ | Morning. I noticed something while adding a new node last night. | 12:04 |
| bmaltais_ | I use the bootstrap feature of kolla-ansible. | 12:05 |
| egonzalez | jmccarthy, nope, that option is expecting a hostname https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L99 | 12:05 |
| bmaltais_ | I added the new blank node in the inventory file and ran bootstrap -i inventory | 12:05 |
| egonzalez | by default is its hostname, unless using nova_fake | 12:05 |
| *** vhosakot has joined #openstack-kolla | 12:05 | |
| bmaltais_ | The result was a disaster. The environment was dead. Somehow bootstrap will re-run on already bootstrapped nodes and break them in the process. | 12:06 |
| bmaltais_ | I think bootstrap should check for the presence of /etc/kolla and should not run on nodes it find it on to prevent breakage. | 12:06 |
| jmccarthy | @egonezalez you mean I can't have the hostname come up as a 'short' name ? 'hostname' of the node is control01_ru6 | 12:08 |
| egonzalez | jmccarthy, can you share the whole trace of the error? also some logs may be present at /var/lib/docker/volumes/kolla_logs/_data/neutron | 12:09 |
| *** manheim has joined #openstack-kolla | 12:12 | |
| jmccarthy | @egonzalez: This is the info I have: http://paste.openstack.org/show/610635/ | 12:13 |
| *** manheim has quit IRC | 12:14 | |
| egonzalez | jmccarthy, underscores in hostname are not valid characters, thats why oslo is complaining about | 12:17 |
| egonzalez | jmccarthy, from the RFC http://www.rfc-editor.org/rfc/rfc1912.txt Allowable characters in a label for a host name are only ASCII | 12:22 |
| egonzalez | letters, digits, and the `-' character | 12:22 |
| jmccarthy | Has it only started complaining about that lately ? These systems have had these names for ages .. | 12:22 |
| egonzalez | jmccarthy, what version are you using? there was a recent (~1 month) in neutron to validate hostname instead of a string | 12:23 |
| *** manheim has joined #openstack-kolla | 12:23 | |
| vhosakot | yes, I've seen oslo complain about underscores in hostname too. Docker (now moby) also has issues if there are underscores in hostnames --> https://github.com/moby/moby/issues/22057. some DNS servers fail to resolve/set domain names when underscores are seen in hostnames. | 12:24 |
| vhosakot | bmaltais_: haven't used bootstrap-servers at all :) I'll try to see what going on before I leave for my vacation today, if not, I'll see after I return from vacation after June 6th. | 12:25 |
| jmccarthy | I'm using stuff that I'm building off master, hmm ok I'll try changing the hostnames | 12:25 |
| egonzalez | jmccarthy, related change https://github.com/openstack/neutron/commit/ac799dc016f6d5d30b46087c6f0a82ac01cb5be3 | 12:25 |
| jmccarthy | Ok cool, I'll give that a shot - thanks :) ! | 12:27 |
| *** ducttape_ has joined #openstack-kolla | 12:27 | |
| egonzalez | bmaltais, should be idempotent unless you updated packages, which maybe cause conflicts because some versions are pinned. please open a bug with logs and all information | 12:27 |
| *** manheim has quit IRC | 12:28 | |
| *** Pavo has joined #openstack-kolla | 12:28 | |
| *** chas has joined #openstack-kolla | 12:29 | |
| vhosakot | egonzalez: jmccarthy wonder if oslo runs this unit test in their gate ;) -> https://github.com/openstack/oslo.config/blob/master/oslo_config/tests/test_cfg.py#L2457-L2465 | 12:29 |
| egonzalez | not sure, only checked the rfc the validate hehe | 12:30 |
| vhosakot | ;) | 12:30 |
| bmaltais_ | egonzalez: One of the task is restarting docker... and I think it is the one that caused the castle to crumble... but I might be wrong about the exact cause... but running bootstrap resulted in a bunch of containers in restart mode. fluentd was one of them. | 12:31 |
| *** chas has quit IRC | 12:33 | |
| *** rwellum has joined #openstack-kolla | 12:34 | |
| egonzalez | vhosakot, is tested https://github.com/openstack/oslo.config/blob/master/oslo_config/tests/test_types.py#L772 | 12:35 |
| bmaltais_ | egonzalez: I think it happen in the post-install.yml when docker is restarted... not sure, could be other steps. I don't quite understand the ansible checks and balances | 12:35 |
| vhosakot | egonzalez: ah, good find, so it a negative test that will fail. | 12:36 |
| *** yingjun has quit IRC | 12:37 | |
| *** mnasiadka has joined #openstack-kolla | 12:38 | |
| egonzalez | bmaltais, you updated docker before executing bootstrap? if so, can you check current docker version? | 12:38 |
| *** manheim has joined #openstack-kolla | 12:39 | |
| *** manheim has quit IRC | 12:43 | |
| *** ducttape_ has quit IRC | 12:43 | |
| *** chas has joined #openstack-kolla | 12:45 | |
| bmaltais_ | egonzalez: I did not. I just ran bootstrap once with 3 nodes. Made sure all was working. Then added the new node to inventory, Ran bootstrap again and the environment was no longer responding. | 12:45 |
| bmaltais_ | egonzalez: I am re-building and will test again capturing the bootstrap output to see what is returned. | 12:46 |
| bmaltais_ | egonzalez: Right now all docker version are at 1.12.0, build 8eab29e | 12:47 |
| *** chas has quit IRC | 12:49 | |
| *** manheim has joined #openstack-kolla | 12:50 | |
| *** manheim has quit IRC | 12:50 | |
| *** manheim has joined #openstack-kolla | 12:50 | |
| *** sbezverk has joined #openstack-kolla | 12:52 | |
| *** iniazi_ has quit IRC | 12:52 | |
| bmaltais_ | egonzalez: OK, Environment is crashed after running bootstrap. Docker versior is the same. | 12:53 |
| bmaltais_ | egonzalez: I had a watch on all servers and saw docker restart at the same time on all of them... This is causing a mariadb issue for sure. | 12:54 |
| bmaltais_ | egonzalez: Yep, mariadb is in a restart look. Will attempt a repair and see. | 12:55 |
| *** fguillot has joined #openstack-kolla | 12:55 | |
| bmaltais_ | s/look/loop | 12:55 |
| *** manheim has quit IRC | 12:56 | |
| bmaltais_ | egonzalez: If I knew ansible better I would propose a change prevent running boostrap on already bootstrapped nodes by checking for /etc/kolla directory on them. | 12:58 |
| *** jchhatbar has joined #openstack-kolla | 12:58 | |
| *** janki has quit IRC | 12:58 | |
| egonzalez | never is late for learning new things ;) | 12:58 |
| vhosakot | bmaltais_: I thought https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/baremetal/tasks/bootstrap-servers.yml is run for baremetal/bifrost deploys, you can check with sean-k-mooney | 12:58 |
| bmaltais_ | egonzalez: Now even a mariadb recovery will not complete. | 12:58 |
| bmaltais_ | vhosakot: I will check with him and see what he think | 12:59 |
| sean-k-mooney | vhosakot: no its inteneded to replace the quickstart guide basically | 12:59 |
| vhosakot | ah, thanks for the info, cool | 13:00 |
| *** manheim has joined #openstack-kolla | 13:00 | |
| bmaltais_ | sean-k-mooney: I ran in an issue with bootstrap running on already bootstrapped nodes | 13:00 |
| sean-k-mooney | vhosakot: it can be used after bifrost but it should work with any ubuntu/centos base install | 13:00 |
| vhosakot | sean-k-mooney: ah ok, bmaltais_ used it to add a compute node :) | 13:00 |
| bmaltais_ | It restart docker on all existing node causing a major crash of the environment | 13:00 |
| bmaltais_ | sean-k-mooney: Shouldn't it skip already bootstrapped nodes all together to avoid issues? | 13:01 |
| sean-k-mooney | bmaltais_: for that to work you would need to add --limit <compute node name> so that it only bootstraps the compute node | 13:01 |
| sean-k-mooney | bmaltais_: well it was inteded to be used only for fresh installs | 13:02 |
| vhosakot | that's what I thought too, for new installs | 13:02 |
| sean-k-mooney | that said restarting docker should not break things | 13:02 |
| bmaltais_ | sean-k-mooney: OK. I never used that option... good to know. But for a good safety measure shouldn't it behave like that and perhaps a globals.yml flag could tell it to do it none the less when a user really feel like it? | 13:02 |
| sean-k-mooney | it happens everytime you reboot | 13:02 |
| bmaltais_ | sean-k-mooney: But it is nice to be able to bootstrap new nodes afterward... | 13:03 |
| hrw | did someone checked in which moment of building images docker itself became a weak spot? | 13:03 |
| bmaltais_ | Isean-k-mooney: If an operator make a typo then the whole environment would go down | 13:03 |
| sean-k-mooney | bmaltais_: if we did that we would have to introduce some kind of sentenal on each host to know it had been bootstrapped | 13:04 |
| *** manheim has quit IRC | 13:04 | |
| sean-k-mooney | bmaltais_: yep | 13:04 |
| bmaltais_ | sean-k-mooney: Bu checking for /etc/kolla directory, yes | 13:04 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode https://review.openstack.org/454690 | 13:04 |
| bmaltais_ | sean-k-mooney: That directory get created when running bootstrap | 13:04 |
| sean-k-mooney | bm that may or may not work but we could add something like that | 13:06 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode https://review.openstack.org/454690 | 13:06 |
| *** srwilkers has joined #openstack-kolla | 13:06 | |
| bmaltais_ | sean-k-mooney: Sure. that was just a suggestion. I am sure there are better sentry to use. | 13:06 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode https://review.openstack.org/454690 | 13:06 |
| sean-k-mooney | bmaltais_: rather that checking for /etc/kolla i would be more comfrotable checking for /etc/kolla/bootstraped | 13:06 |
| hawi | i'm getting FAILED! => {"failed": true, "msg": "The conditional check '{{ (ceph_files_json.stdout | from_json).changed }}' failed. The error was: No JSON object could be decoded"} when deploying 4.0.0 Fetching Ceph keyrings | 13:07 |
| bmaltais_ | sean-k-mooney: Make sense. So the bootstrap process would create a file at then end that could be detected to prevent from running again. | 13:07 |
| hawi | never happened before | 13:07 |
| *** powerd has joined #openstack-kolla | 13:08 | |
| sean-k-mooney | basically i dont want to get to a point where the bootstrap role partly completes and has created the directoy but not actully bootstraped properly | 13:08 |
| sean-k-mooney | so ya and then we could add a --rebootstrap option or something if we wanted to run on all nodes ignoreing if they were previously bootstraped | 13:09 |
| *** manheim has joined #openstack-kolla | 13:10 | |
| sean-k-mooney | bmaltais_: others have extended it but i originally wrote the bootstrap role in 3 days to help with the OSIC performance testing so there are many improments that could be made | 13:10 |
| openstackgerrit | Juan J. Martínez proposed openstack/kolla master: Fix in sed line (s command) https://review.openstack.org/468026 | 13:11 |
| bmaltais_ | sean-k-mooney: If my knowledge of ansible was ok I would give it a try but I am literally starting from 0. I would probably do more harm than good at this time. | 13:13 |
| sean-k-mooney | bmaltais_: by the way in the future you can also contol what it runs on by specifying the host you want to bootstrap in the baremetal group in your inventory | 13:13 |
| *** korzen has quit IRC | 13:13 | |
| bmaltais_ | sean-k-mooney: OK. Right now the default example file make it include all nodes... I guess this is another way to control the scope. | 13:14 |
| *** manheim has quit IRC | 13:14 | |
| *** manheim has joined #openstack-kolla | 13:14 | |
| bmaltais_ | sean-k-mooney: For now I will adjust my "add new node" documentation to refer to --limit | 13:14 |
| *** jchhatbar is now known as janki | 13:15 | |
| *** huzhengchuan_ has quit IRC | 13:15 | |
| *** huzhengchuan has quit IRC | 13:15 | |
| sean-k-mooney | on --limit it should work but i have only tested the group way | 13:16 |
| *** lucasxu has joined #openstack-kolla | 13:16 | |
| bmaltais_ | sean-k-mooney: Will find out once I rebuild my test environment | 13:17 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove the unnecessary jinja2 templating delimiters https://review.openstack.org/466612 | 13:17 |
| sean-k-mooney | bmaltais_: so what exectly happened to you test enviornment. it should have started working again after docker restarted | 13:18 |
| sean-k-mooney | it would have gone offline while it was restarted but it should not have broken it permently | 13:19 |
| bmaltais_ | sean-k-mooney: mariadb was in restart loop. Tried stoping all mariadb containers and perform a kolla-ansible recover-mariadb | 13:19 |
| bmaltais_ | sean-k-mooney: But that never completed. The DB never came back up | 13:19 |
| vhosakot | brb, 15 | 13:19 |
| *** vhosakot has quit IRC | 13:19 | |
| sean-k-mooney | ah ok | 13:20 |
| sean-k-mooney | restarting docker is effectly the same as what happens if you reboot the node so that is the correct way to fix it | 13:20 |
| bmaltais_ | sean-k-mooney: When I manually do a docker stop mariadb on each node I can bring it back after... but did not work after the bootstrap | 13:20 |
| bmaltais_ | sean-k-mooney: Perhaps because everything collapse at the same time... vs running sequentially when doing it manually. | 13:21 |
| sean-k-mooney | i might take a look at the bootstrap playbook again and see if i can refactor it to only restart docker if we really need to and to skip already bootstrapped nodes | 13:23 |
| sean-k-mooney | bmaltais_: can you open a bug for this just so we dont forget | 13:23 |
| bmaltais_ | sean-k-mooney: Sure. I just need to figure out how to open a bug ;-) Will be my 1st. | 13:24 |
| sean-k-mooney | do you have a launchpad account https://bugs.launchpad.net/kolla-ansible | 13:25 |
| *** dixiaoli has joined #openstack-kolla | 13:26 | |
| *** lucasxu has quit IRC | 13:27 | |
| *** lucasxu has joined #openstack-kolla | 13:27 | |
| *** dixiaoli has quit IRC | 13:28 | |
| *** jtriley has joined #openstack-kolla | 13:33 | |
| *** lucasxu has quit IRC | 13:33 | |
| bmaltais_ | sean-k-mooney: Figured it out. Here is the link to the bug: https://bugs.launchpad.net/kolla-ansible/+bug/1693509 | 13:34 |
| openstack | Launchpad bug 1693509 in kolla-ansible "Running bootstrap on existing node cause issues" [Undecided,New] | 13:34 |
| *** srwilkers has quit IRC | 13:34 | |
| *** lucasxu has joined #openstack-kolla | 13:34 | |
| sean-k-mooney | coool thanks | 13:35 |
| *** eanylin has quit IRC | 13:37 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod https://review.openstack.org/467272 | 13:40 |
| *** AzazeL_ has quit IRC | 13:41 | |
| *** manheim has quit IRC | 13:42 | |
| *** lucasxu has quit IRC | 13:43 | |
| egonzalez | hawi, where that error occurs? ceph, nova, cinder, glance, gnocchi? | 13:43 |
| *** lucasxu has joined #openstack-kolla | 13:43 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod https://review.openstack.org/467272 | 13:46 |
| *** manheim has joined #openstack-kolla | 13:46 | |
| *** iniazi_ has joined #openstack-kolla | 13:47 | |
| *** manheim has quit IRC | 13:50 | |
| *** eanylin has joined #openstack-kolla | 13:51 | |
| *** manheim has joined #openstack-kolla | 13:56 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster https://review.openstack.org/466881 | 13:56 |
| *** manheim has quit IRC | 13:57 | |
| *** powerd has quit IRC | 13:57 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment. https://review.openstack.org/467695 | 13:58 |
| *** andybrucenet has joined #openstack-kolla | 14:00 | |
| *** vhosakot has joined #openstack-kolla | 14:01 | |
| *** janki has quit IRC | 14:02 | |
| *** andybrucenet has quit IRC | 14:03 | |
| *** powerd has joined #openstack-kolla | 14:03 | |
| *** mewald has quit IRC | 14:05 | |
| *** mewald has joined #openstack-kolla | 14:06 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add a complete example of configuring a basic 3 nodes Ceph environment. https://review.openstack.org/467695 | 14:07 |
| *** ducttape_ has joined #openstack-kolla | 14:07 | |
| *** jamesPR has joined #openstack-kolla | 14:10 | |
| *** huzhengchuan_ has joined #openstack-kolla | 14:10 | |
| *** mewald has quit IRC | 14:10 | |
| *** manheim has joined #openstack-kolla | 14:11 | |
| *** daidv has quit IRC | 14:12 | |
| *** mewald has joined #openstack-kolla | 14:12 | |
| *** huzhengchuan has joined #openstack-kolla | 14:14 | |
| *** zhubingbing_ has joined #openstack-kolla | 14:14 | |
| *** mgoddard has quit IRC | 14:14 | |
| *** andybrucenet has joined #openstack-kolla | 14:16 | |
| *** powerd_ has joined #openstack-kolla | 14:17 | |
| *** daidv has joined #openstack-kolla | 14:18 | |
| *** manheim has quit IRC | 14:19 | |
| *** powerd has quit IRC | 14:20 | |
| sbezverk | andybrucenet: ping | 14:20 |
| *** powerd_ has quit IRC | 14:22 | |
| *** daidv has quit IRC | 14:22 | |
| hawi | egonzalez: i did rm -rf /etc/kolla on storage,compute and controllers, stopped docker, then rm -rf /var/lib/docker/* and then started docker | 14:25 |
| hawi | amazing, now started to deploy again | 14:25 |
| hawi | it stopped deployment after ceph : Fetching Ceph keyrings | 14:27 |
| hawi | but now works fine | 14:27 |
| *** ducttape_ has quit IRC | 14:27 | |
| hawi | as i had no daata to loose i removed everything | 14:28 |
| vhosakot | hawi: you should've rebooted too before restarting/re-deploying :) | 14:33 |
| *** ducttape_ has joined #openstack-kolla | 14:34 | |
| *** ducttape_ has quit IRC | 14:35 | |
| *** jemcevoy has quit IRC | 14:35 | |
| *** ducttape_ has joined #openstack-kolla | 14:35 | |
| hawi | vhosakot: actually, before removing these folders i did touch /forcefsck and tried reboot already :) | 14:36 |
| hawi | dunno if forcefsck works on modern linuxes any more | 14:37 |
| *** srwilkers has joined #openstack-kolla | 14:37 | |
| *** daidv has joined #openstack-kolla | 14:39 | |
| *** lucasxu has quit IRC | 14:39 | |
| *** ducttape_ has quit IRC | 14:39 | |
| *** huzhengchuan_ has quit IRC | 14:41 | |
| *** andybrucenet has quit IRC | 14:41 | |
| *** huzhengchuan has quit IRC | 14:42 | |
| *** hongbin has joined #openstack-kolla | 14:45 | |
| *** chas has joined #openstack-kolla | 14:45 | |
| *** unicell has quit IRC | 14:47 | |
| vhosakot | hawi: cool, good you rebooted.. almost thought forcefsck is a typo there.. lol ;) | 14:48 |
| *** mgoddard has joined #openstack-kolla | 14:49 | |
| *** chas has quit IRC | 14:50 | |
| hawi | now i do have magic souce for extremely clean deploys ;) | 14:51 |
| openstackgerrit | zhubingbing proposed openstack/kolla master: Fix rally create verifier error https://review.openstack.org/467837 | 14:51 |
| hawi | finished it, good | 14:51 |
| *** korzen has joined #openstack-kolla | 14:52 | |
| *** klowse has joined #openstack-kolla | 14:53 | |
| *** huzhengchuan_ has joined #openstack-kolla | 14:54 | |
| klowse | when would the 4.0.1 images be on docker hub? Getting: Unknown error message: Tag 4.0.1 not found in repository docker.io... | 14:54 |
| *** huzhengchuan has joined #openstack-kolla | 14:55 | |
| *** lucasxu has joined #openstack-kolla | 14:56 | |
| jamesPR | klowse, you can build your own images ;) | 14:57 |
| *** zhubingbing_ has quit IRC | 14:58 | |
| klowse | :( | 14:59 |
| *** dave-mccowan has joined #openstack-kolla | 15:01 | |
| *** korzen has quit IRC | 15:02 | |
| *** trozet has quit IRC | 15:02 | |
| *** trozet has joined #openstack-kolla | 15:04 | |
| *** Serlex has quit IRC | 15:07 | |
| *** andybrucenet has joined #openstack-kolla | 15:07 | |
| *** shardy has quit IRC | 15:09 | |
| openstackgerrit | Juan J. Martínez proposed openstack/kolla master: Fix sed replacement command in Ubuntu's install_docker https://review.openstack.org/468026 | 15:16 |
| *** andybrucenet has quit IRC | 15:17 | |
| *** yingjun has joined #openstack-kolla | 15:22 | |
| *** manheim has joined #openstack-kolla | 15:24 | |
| *** ducttape_ has joined #openstack-kolla | 15:24 | |
| *** yingjun has quit IRC | 15:29 | |
| *** manheim has quit IRC | 15:30 | |
| *** manheim has joined #openstack-kolla | 15:30 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster https://review.openstack.org/466881 | 15:32 |
| *** pcaruana has quit IRC | 15:38 | |
| *** yingjun has joined #openstack-kolla | 15:38 | |
| *** shardy has joined #openstack-kolla | 15:39 | |
| *** pcaruana has joined #openstack-kolla | 15:39 | |
| *** skramaja has quit IRC | 15:43 | |
| *** yingjun has quit IRC | 15:43 | |
| *** daidv has quit IRC | 15:44 | |
| *** p6 has quit IRC | 15:46 | |
| *** p6 has joined #openstack-kolla | 15:47 | |
| sdake | @klowse which distro are you running? | 15:52 |
| sdake | klowse i can push centos source/binary - but can't push ubuntu | 15:52 |
| klowse | using centos-binary...but ive found info on tarballs.openstack.org/kolla/images/ and going to try and use that. seem some more recent fixes in it (4.0.2)? | 15:53 |
| *** phuongnh has joined #openstack-kolla | 15:56 | |
| *** AzazeL_ has joined #openstack-kolla | 15:59 | |
| *** jascott1 has joined #openstack-kolla | 15:59 | |
| *** jascott1_ has joined #openstack-kolla | 16:00 | |
| chandankumar | sdake: hello | 16:01 |
| chandankumar | sdake: do we have some guide on how to run tempest using kolla? | 16:01 |
| *** lucasxu has quit IRC | 16:03 | |
| *** lucasxu has joined #openstack-kolla | 16:03 | |
| *** jascott1 has quit IRC | 16:04 | |
| *** egonzalez has quit IRC | 16:05 | |
| jascott1_ | o/ | 16:07 |
| sdake | @chandankumar no i don't sorry | 16:10 |
| jascott1_ | what up sdake! | 16:14 |
| sdake | sup jascott1_ | 16:14 |
| jascott1_ | back from my vacation | 16:15 |
| jascott1_ | spent 12 hours in O'Hare on the way back. fun. | 16:16 |
| jascott1_ | whatd i miss? ;) | 16:16 |
| openstackgerrit | Tim Rozet proposed openstack/kolla master: Adds opendaylight specific neutron-server container https://review.openstack.org/467712 | 16:17 |
| *** sayantan_ has joined #openstack-kolla | 16:18 | |
| *** mnasiadka has quit IRC | 16:24 | |
| *** retr0h has quit IRC | 16:26 | |
| *** retr0h has joined #openstack-kolla | 16:26 | |
| sdake | jascott1_ i almost took over your review :) | 16:27 |
| sdake | jascott1_ but didn't get a chance | 16:28 |
| sdake | jascott1_ have a look at this: | 16:28 |
| *** shardy has quit IRC | 16:28 | |
| sdake | https://review.openstack.org/#/c/464031/ | 16:28 |
| jascott1_ | yeah saw there was a ton of iteration on it | 16:29 |
| *** pcaruana has quit IRC | 16:30 | |
| *** pcaruana has joined #openstack-kolla | 16:30 | |
| *** mnasiadka has joined #openstack-kolla | 16:31 | |
| *** mnasiadka has quit IRC | 16:36 | |
| *** mnasiadka has joined #openstack-kolla | 16:38 | |
| sdake | jascott1_ ya - its close to merge | 16:39 |
| sdake | jascott1_ what we need next is a "deploy" playbook | 16:39 |
| sdake | thato nly deploys kolla-kubernetes | 16:39 |
| sdake | jascott1_ you have most of that already done | 16:40 |
| jascott1_ | ya | 16:40 |
| sdake | jascott1_ the idea is to wrap these in a container | 16:40 |
| jascott1_ | just need a var to control whether k8s is installed or not | 16:40 |
| jascott1_ | or just make separatae playbooks | 16:40 |
| *** mnasiadka has quit IRC | 16:43 | |
| sdake | jascott1_ i'd prefer separate playbooks | 16:44 |
| *** jheroux has joined #openstack-kolla | 16:44 | |
| hrw | https://review.openstack.org/#/c/467521 anyone? it changes name of python-oslo-vmware to python-oslo.vmware as first one is transitional package while second is real one. | 16:44 |
| jascott1_ | sdake i think we can have best of both worlds and be able to call deploy but also have a top level that does it all | 16:45 |
| *** huzhengchuan_ has quit IRC | 16:45 | |
| sdake | jascott1_ yup - include may work here | 16:45 |
| jascott1_ | cool I can refactor | 16:46 |
| *** huzhengchuan has quit IRC | 16:46 | |
| *** chas has joined #openstack-kolla | 16:46 | |
| *** huzhengchuan has joined #openstack-kolla | 16:48 | |
| *** mnasiadka has joined #openstack-kolla | 16:48 | |
| *** jascott1 has joined #openstack-kolla | 16:49 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 16:49 |
| *** huzhengchuan_ has joined #openstack-kolla | 16:49 | |
| *** jascott1_ has quit IRC | 16:51 | |
| *** chas has quit IRC | 16:52 | |
| *** mnasiadka has quit IRC | 16:53 | |
| *** vhosakot has quit IRC | 17:00 | |
| *** jamesbenson has joined #openstack-kolla | 17:04 | |
| *** kbaegis has joined #openstack-kolla | 17:04 | |
| *** mnasiadka has joined #openstack-kolla | 17:04 | |
| *** jmccarthy has left #openstack-kolla | 17:06 | |
| *** stefan_bo has joined #openstack-kolla | 17:07 | |
| *** sayantan_ has quit IRC | 17:07 | |
| *** sayantan_ has joined #openstack-kolla | 17:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:08 |
| *** mnasiadka has quit IRC | 17:09 | |
| *** mnasiadka has joined #openstack-kolla | 17:10 | |
| *** huzhengchuan has quit IRC | 17:11 | |
| *** stefan_bo has quit IRC | 17:11 | |
| *** huzhengchuan has joined #openstack-kolla | 17:11 | |
| *** sayantan_ has quit IRC | 17:14 | |
| *** sayantan_ has joined #openstack-kolla | 17:14 | |
| *** mnasiadka has quit IRC | 17:15 | |
| kbaegis | Hey guys, what's the standard/best practice for setting up kolla-ansible? | 17:15 |
| kbaegis | i.e. if I don't want to do "sudo pip install -U kolla-ansible" | 17:15 |
| kbaegis | How do I do that in an isolated venv that won't break my global site-packages? | 17:16 |
| *** rmart04 has quit IRC | 17:16 | |
| kbaegis | ideally as a non-root user | 17:16 |
| *** sayantan_ has quit IRC | 17:17 | |
| *** sayantan_ has joined #openstack-kolla | 17:17 | |
| *** mnasiadka has joined #openstack-kolla | 17:18 | |
| *** sambetts is now known as sambetts|afk | 17:18 | |
| *** lucasxu has quit IRC | 17:20 | |
| *** phuongnh has quit IRC | 17:21 | |
| *** korzen has joined #openstack-kolla | 17:22 | |
| *** sayantan_ has quit IRC | 17:22 | |
| *** sayantan_ has joined #openstack-kolla | 17:22 | |
| *** mnasiadka has quit IRC | 17:23 | |
| jascott1 | kbaegis if you setup a venv and install does it break your site packages? | 17:27 |
| kbaegis | It breaks portage | 17:28 |
| kbaegis | Figured it out | 17:28 |
| *** vhosakot has joined #openstack-kolla | 17:31 | |
| kbaegis | jascott1: I was using a venv, but noexec was set on the directory | 17:31 |
| kbaegis | So it wasn't using pip locally; still using global pip | 17:32 |
| jascott1 | oh ok. thats more venv foo than im experiened with (which isnt much) but cool you got it sorted | 17:32 |
| jascott1 | foo/fu | 17:32 |
| *** sayantan_ has quit IRC | 17:39 | |
| *** sayantan_ has joined #openstack-kolla | 17:40 | |
| *** aagate has joined #openstack-kolla | 17:40 | |
| *** vhosakot has quit IRC | 17:40 | |
| kbaegis | Sweet! :) CI on gentoo with post-merge hooks | 17:46 |
| kbaegis | Now for CD | 17:46 |
| *** Pavo has quit IRC | 17:47 | |
| *** iniazi has joined #openstack-kolla | 17:51 | |
| *** huzhengchuan_ has quit IRC | 17:52 | |
| *** huzhengchuan_ has joined #openstack-kolla | 17:52 | |
| *** iniazi_ has quit IRC | 17:53 | |
| *** rmart04 has joined #openstack-kolla | 17:54 | |
| *** lpetrut has quit IRC | 17:54 | |
| *** egonzalez has joined #openstack-kolla | 17:55 | |
| *** unicell has joined #openstack-kolla | 17:58 | |
| bmaltais_ | I am looking at the documentation and I am not clear how to accomplish the following... | 17:59 |
| bmaltais_ | I will have a mix of different hardware/VM as part of the environment. Interfaces names will vary amongst them and probably the virtualisation (qemu/kvm) capabilities. | 18:00 |
| *** ducttape_ has quit IRC | 18:00 | |
| *** sayantan_ has quit IRC | 18:00 | |
| bmaltais_ | How would I go about setting per node interfaces and per node virtualisation type? | 18:00 |
| bmaltais_ | Right now those are applied to all nodes from the central config on the deployment node. | 18:01 |
| egonzalez | bmaltais_ for interface names and each var at the inventory, ex: https://docs.openstack.org/developer/kolla-ansible/multinode.html#edit-the-inventory-file | 18:02 |
| egonzalez | bmaltais_ for different virt types, add one file for each node at /etc/kolla/config/nova/{{ inventory_hostname }}/nova.conf | 18:04 |
| egonzalez | bmaltais_ if have more kvm than qemu, just add a file for each qemu | 18:04 |
| *** manheim has quit IRC | 18:04 | |
| egonzalez | [libvirt] | 18:04 |
| egonzalez | virt_type = qemu | 18:04 |
| egonzalez | cpu_mode = none | 18:04 |
| egonzalez | EOF | 18:04 |
| bmaltais_ | egonzalez: OK, so something like "controller1 network_interface="ens160" neutron_external_interface="ens192" someothervarfromglobals.yml=something" | 18:05 |
| egonzalez | bmaltais_ yep | 18:05 |
| bmaltais_ | egonzalez: Cool! | 18:05 |
| *** manheim has joined #openstack-kolla | 18:06 | |
| bmaltais_ | egonzalez: Nice, I am starting to get the hang of all this. Very nice | 18:06 |
| *** korzen has quit IRC | 18:07 | |
| *** manheim has quit IRC | 18:09 | |
| *** david-lyle has joined #openstack-kolla | 18:12 | |
| *** eaguilar has joined #openstack-kolla | 18:13 | |
| *** jamesbenson has quit IRC | 18:15 | |
| *** eaguilar has quit IRC | 18:18 | |
| *** david-lyle has quit IRC | 18:19 | |
| emccormick | I"m actually having a little trouble doing the overrides in inventory. This statement in all.yml seems to always evaluate to what is in globals.yml and not what is in the inventory | 18:20 |
| emccormick | hostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address'] | 18:20 |
| emccormick | I thought it was maybe to do with the way that Ansible messed up the tree structure after 2.1.0.0, but tweaking the structure didn't seem to fix it either | 18:21 |
| *** dgonzalez has quit IRC | 18:22 | |
| egonzalez | emccormick how you override vars, recently saw that inventory group vars are not overriden but var in inventory hosts do | 18:22 |
| *** dgonzalez has joined #openstack-kolla | 18:22 | |
| *** korzen has joined #openstack-kolla | 18:23 | |
| emccormick | compute2.cc.osdev.cirrusseven.net network_interface=enp3s0f1 api_interface=enp3s0f1 tunnel_interface=enp3s0f0 | 18:23 |
| emccormick | is the override statement in my inventory file | 18:23 |
| emccormick | however, in globals.yml I'm doing api_interface: "ens33" | 18:24 |
| emccormick | which is what my controllers need | 18:24 |
| emccormick | that's all that seems to be picking up. On deployment I"m getting this | 18:24 |
| emccormick | fatal: [compute2.cc.osdev.cirrusseven.net]: FAILED! => {"failed": true, "msg": "{{ hostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address'] if orchestration_engine == 'ANSIBLE' else '0.0.0.0' }}: 'dict object' has no attribute u'ansible_ens33'"} | 18:24 |
| *** ducttape_ has joined #openstack-kolla | 18:25 | |
| emccormick | which makes sense since it has no ens33, but the override isn't taking | 18:25 |
| egonzalez | what task? | 18:25 |
| *** manheim has joined #openstack-kolla | 18:25 | |
| emccormick | copying nova configs | 18:25 |
| *** lucasxu has joined #openstack-kolla | 18:25 | |
| emccormick | TASK [nova : Copying over nova.conf] | 18:26 |
| egonzalez | emccormick can you try changing api_interface_address in group_vars to hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hostname]['api_interface']['ipv4']['address'] | 18:28 |
| *** manheim has quit IRC | 18:29 | |
| sdake | sbezverk what do you make of http://logs.openstack.org/31/464031/36/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/4fca4c1/console.html#_2017-05-25_17_41_25_839247 | 18:29 |
| sbezverk | sdake: looks like ansible error, why there is error, I have no clue, the kubectl command looks good to me | 18:30 |
| emccormick | sure | 18:31 |
| sdake | jascott1 does | work in command in ansible? | 18:31 |
| jascott1 | le pipe? | 18:31 |
| jascott1 | yea | 18:32 |
| jascott1 | oh wait do you need to use shell? | 18:32 |
| jascott1 | if it doesnt work try shell :) | 18:32 |
| egonzalez | yep, need shell for regex | 18:32 |
| jascott1 | i think it does tho | 18:32 |
| jascott1 | checking | 18:32 |
| *** manheim has joined #openstack-kolla | 18:33 | |
| *** mgoddard has quit IRC | 18:34 | |
| jascott1 | sdake the task with | im using does use `shell` instead `command` | 18:35 |
| jascott1 | yup gotta use shell afaik | 18:36 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 18:37 |
| *** manheim has quit IRC | 18:37 | |
| sdake | jascott1 thanks dude | 18:37 |
| jascott1 | np. seems like the rule is it can only execute the first thing so no pipes or operators etc | 18:37 |
| jascott1 | (for command) | 18:38 |
| *** rmart04 has quit IRC | 18:42 | |
| sdake | i am using a PV with NFS defined, yet the container running against the PVCs is complaining "read only filesystem" | 18:45 |
| sdake | however if i mount the filesystem in linux, I can write files to it | 18:45 |
| sdake | any ideas why kubernetes doesn't work here? | 18:45 |
| *** sayantan_ has joined #openstack-kolla | 18:47 | |
| *** chas has joined #openstack-kolla | 18:47 | |
| jascott1 | sdake you are specifying spec.accessModes for pv? just checking | 18:51 |
| sdake | jascott1 yes | 18:51 |
| sdake | RWX | 18:51 |
| sdake | i more or less cpied this: https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs | 18:52 |
| sdake | the only delta is i am using a netapp filer instead of a nfs linux client | 18:52 |
| *** chas has quit IRC | 18:52 | |
| sdake | i am getting an errono30 inside my containers (which indicates read only filesystem) | 18:53 |
| sdake | its as if kubernetes is mounting it RO | 18:53 |
| *** mnasiadka has joined #openstack-kolla | 18:56 | |
| emccormick | egonzalez: That left me with an error in memcached: failed: [controller1.cc.osdev.cirrusseven.net] (item=memcached) => {"failed": true, "item": "memcached", "msg": "AnsibleUndefinedVariable: {{ hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hostname]['api_interface']['ipv4']['address']] if orchestration_engine == 'ANSIBLE' else '0.0.0.0' }}: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no | 18:57 |
| emccormick | attribute 'ipv4'"} | 18:57 |
| emccormick | which I'm guessing is the first place it referenced that ansible_interface_address var | 18:58 |
| *** srwilkers has quit IRC | 18:58 | |
| egonzalez | emccormick sorry, typo hostvars[inventory_hostname]['ansible_' + hostvars[inventory_hosname]['api_interface']]['ipv4']['address'] | 18:58 |
| egonzalez | missing one ] after api_interface | 18:58 |
| emccormick | oops yeah I was fishing for that too hehe thanks | 18:59 |
| sbezverk | sdake : do you use k8s's storage class or pvc point to nfs mount on the host? | 18:59 |
| Slower | sbezverk: do you remember why ipc=host was needed for iscsid? | 18:59 |
| sdake | sbezverk i create 3 pvs, and 3 pvcs | 18:59 |
| sdake | sbezverk https://da.gd/aFFv -> https://paste.fedoraproject.org/paste/vEF5uw794bxSn3A2mZ2uQl5M1UNdIGYhyRLivL9gydE=/ | 19:00 |
| jascott1 | sdake you do anything with RBAC for nfs provisioner? | 19:00 |
| sbezverk | Slower : we needed to interact with lvm, otherwise all lvm related operations were hanging | 19:00 |
| sdake | i applied the aboe patch on top of kolla-kubernetes master | 19:00 |
| *** mnasiadka has quit IRC | 19:01 | |
| sbezverk | sdake : I have not tried with nfs but with iscsi I needed to define storage class, pv and pvc | 19:02 |
| sbezverk | sdake : check k8s storage docs | 19:03 |
| *** mnasiadka has joined #openstack-kolla | 19:03 | |
| *** p6 has quit IRC | 19:03 | |
| *** p6 has joined #openstack-kolla | 19:04 | |
| jascott1 | sdake seems related https://github.com/kubernetes-incubator/nfs-provisioner/blob/master/docs/usage.md | 19:04 |
| *** manheim has joined #openstack-kolla | 19:05 | |
| Slower | sbezverk: ok cool thanks | 19:05 |
| rcrit | I'm working on a source build for the novajoin container and noticed odd permissions with bind mounted files | 19:07 |
| rcrit | I'm a complete noob when it comes to containers so maybe it's something I'm doing wrong | 19:07 |
| emccormick | egonzalez: Back to the original error with it picking up the global interface | 19:07 |
| rcrit | I have USER nova in the Docker file for now (eventually to be a unique user) | 19:07 |
| rcrit | when I bind mount files in the uid of the file is nova on the underlying system which doesn't match the uid of the container so my service can't read the config | 19:08 |
| *** mnasiadka has quit IRC | 19:08 | |
| rcrit | invocation like: docker run --network host -v /etc/nova/join.conf:/etc/nova/join.conf:ro ... -ti <id> /bin/bash | 19:08 |
| kbaegis | Hey guys, anyone seen "Unexpected Exception: unsupported operand type(s) for %: 'bytes' and 'bytes'" in a while? | 19:09 |
| *** pcaruana has quit IRC | 19:09 | |
| *** manheim has quit IRC | 19:09 | |
| kbaegis | seems to be an issue on kolla-ansible for both python 3.4 and 2.7 | 19:09 |
| kbaegis | Almost certainly my fault | 19:09 |
| egonzalez | emccormick please, open a bug, I will investigate more on that | 19:09 |
| emccormick | egonzalez ok will do. | 19:10 |
| kbaegis | traceback: https://hastebin.com/risuwipusi.sql | 19:11 |
| emccormick | egonzalez: I"m actually doing this with the Newton (3.0.3) version, but it looks the same for Ocata and Master. Do you want me to just open it against Master? | 19:11 |
| *** jamesbenson has joined #openstack-kolla | 19:12 | |
| *** mnasiadka has joined #openstack-kolla | 19:13 | |
| egonzalez | emccormick sure | 19:16 |
| jamesbenson | yay 😁 Magnum and murano deployed easy :-). Now onto AODH and Gnocchi. :-) | 19:16 |
| sdake | sbezverk do you have an example storageclass for nfs? | 19:17 |
| *** mnasiadka has quit IRC | 19:18 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Combine ironic-inspector deployments into one pod https://review.openstack.org/467272 | 19:21 |
| kbaegis | Does kolla-ansible require python 3.5? | 19:22 |
| *** jamesPR has quit IRC | 19:22 | |
| *** mnasiadka has joined #openstack-kolla | 19:22 | |
| emccormick | egonzalez: opened. Thanks | 19:26 |
| *** manheim has joined #openstack-kolla | 19:27 | |
| *** mnasiadka has quit IRC | 19:28 | |
| *** manheim has quit IRC | 19:32 | |
| *** mnasiadka has joined #openstack-kolla | 19:33 | |
| hrw | ~curse kubernetes-entrypoint image for being x86-64 only | 19:35 |
| *** mnasiadka has quit IRC | 19:38 | |
| *** mnasiadka has joined #openstack-kolla | 19:42 | |
| *** mnasiadka has quit IRC | 19:47 | |
| *** shaggydog97 has joined #openstack-kolla | 19:49 | |
| kfox1111 | sbezverk: https://github.com/kubernetes/kubernetes/pull/46444 | 19:51 |
| kfox1111 | something to watch. | 19:51 |
| sdake | getting this error Error: error validating "": error validating data: found invalid field storageClassName for v1.PersistentVolumeSpec | 19:52 |
| sdake | with this pv/pvc combo: https://da.gd/8tEtc -> https://paste.fedoraproject.org/paste/LuuVKFLW7ai5peuql-S6f15M1UNdIGYhyRLivL9gydE=/ | 19:52 |
| sdake | any ideas? | 19:52 |
| kfox1111 | looking... | 19:53 |
| sdake | kubectl get storageclasses | 19:53 |
| sdake | NAME TYPE | 19:53 |
| sdake | slow kubernetes.io/nfs | 19:53 |
| sdake | thanks kfox1111 | 19:53 |
| kfox1111 | hmm... | 19:53 |
| *** mnasiadka has joined #openstack-kolla | 19:54 | |
| kfox1111 | oh. | 19:54 |
| kfox1111 | theres a storageClassName in the pv. I'm not sure thats allowed. | 19:55 |
| sdake | https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes | 19:55 |
| *** bmaltais_ has quit IRC | 19:55 | |
| sdake | kfox1111 if i run without a storageclass, the nfs mounted filesystem is mounted read only (errno30 via mariadb) | 19:57 |
| kfox1111 | I was thinking the storage class was what provisioned the pv. | 19:58 |
| jamesbenson | So an open question. Has anyone made their own unit tests to verify things are working? I created a small script to deploy the public network, private network, subnets, routers, upload cirros, deploy an vm, and ssh into it... but I'm thinking about aodh, marano, etc? | 19:58 |
| jamesbenson | I'm curious to see what others have done, if anything. | 19:58 |
| *** mnasiadka has quit IRC | 19:59 | |
| sdake | kfox1111 unclear - i've read these docs about 20 times, and i cna't find a clear answer | 19:59 |
| kfox1111 | sdake: ah. I see... | 19:59 |
| sdake | kfox1111 what is odd is on different hardware, I can mount the nfs volume correctly | 19:59 |
| kfox1111 | what version of k8s you using? | 19:59 |
| kfox1111 | I think storageClassName on the pv is a 1.6 feature. | 19:59 |
| sdake | 1.5.2 | 19:59 |
| kfox1111 | it may be an annotation of: | 19:59 |
| kfox1111 | volume.beta.kubernetes.io/storage-class | 19:59 |
| kfox1111 | before that. | 19:59 |
| *** fguillot has quit IRC | 20:00 | |
| sdake | can you hand-feed me the syntax plz :) | 20:00 |
| kfox1111 | https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes | 20:00 |
| kfox1111 | uh.... | 20:00 |
| kfox1111 | metadata: | 20:00 |
| kfox1111 | annotations: | 20:00 |
| kfox1111 | "volume.beta.kubernetes.io/storage-class": slow | 20:00 |
| sdake | thanks | 20:00 |
| kfox1111 | np | 20:01 |
| jascott1 | jamesbenson I have wondered the same. I think we should be able to use existing openstack tools like rally | 20:01 |
| jamesbenson | but on private clouds? | 20:01 |
| jamesbenson | ah, sorry | 20:02 |
| jamesbenson | jascott1: just read about rally... I was thinking it was something else. | 20:02 |
| *** mnasiadka has joined #openstack-kolla | 20:02 | |
| *** mgoddard has joined #openstack-kolla | 20:03 | |
| jascott1 | jamesbenson iirc rally is a nice executor for temptest | 20:03 |
| jascott1 | and others? | 20:03 |
| jamesbenson | jascott1: Do you know if this is what Mirantis did for the health checks? Sounds possible. | 20:04 |
| jascott1 | health check to me implies on going "are you still there" pings or requests | 20:05 |
| jascott1 | this sounds like smoke test | 20:05 |
| jascott1 | but no i dont know what mirantis did | 20:05 |
| kfox1111 | are you talknig about fuels ui for the checks it did? | 20:06 |
| kbaegis | Figured it out. Needed python3.5, pip3.5 | 20:06 |
| kbaegis | :/ | 20:06 |
| kfox1111 | if so, I'm pretty sure that was tempest. | 20:06 |
| *** mnasiadka has quit IRC | 20:07 | |
| jamesbenson | jascott1: yeah, post deployment, you can do a health check: requests image list, create instance flavor, keypair, volumes, launch instance with file injection... HA tests, ceilometer tests to check events and notifications, etc. | 20:08 |
| jamesbenson | jascott1: I'm looking at rally, and enjoying it. I think I'll deploy that and try to run some smoke tests.... | 20:08 |
| jascott1 | hopefully we will add some docs about using these tools to test a kk8s deployment | 20:09 |
| jascott1 | seems like we could also run in gate even if non-voting | 20:11 |
| *** david-lyle has joined #openstack-kolla | 20:12 | |
| *** mnasiadka has joined #openstack-kolla | 20:12 | |
| *** korzen has quit IRC | 20:13 | |
| *** david-lyle has quit IRC | 20:17 | |
| *** mnasiadka has quit IRC | 20:17 | |
| jascott1 | sdake you are merging destroy under ansible/ ? i was thinking more tools/deploy or something as to not confuse it with the older ansible stuff | 20:19 |
| sdake | jascott1 i think we should have several top level plays | 20:19 |
| sdake | destroy, config, upgrade, deploy, etc | 20:19 |
| jascott1 | agree | 20:19 |
| jascott1 | om only asking about the path | 20:19 |
| *** sayantan_ has quit IRC | 20:20 | |
| sdake | yup i think it all belongs in ansible :) | 20:20 |
| *** sayantan_ has joined #openstack-kolla | 20:20 | |
| jascott1 | sdake im confused about what in ansible/ is obsolete... arent those service roles no longer needed? | 20:20 |
| *** lucasxu has quit IRC | 20:21 | |
| *** lucasxu has joined #openstack-kolla | 20:22 | |
| sdake | jascott1 those are config roles | 20:22 |
| jascott1 | sounds important ;) | 20:23 |
| sdake | kfox1111 i get this result when launching my cluster with that nfs provisioner | 20:23 |
| sdake | NAME STATUS VOLUME CAPACITY ACCESSMODES AGE | 20:23 |
| sdake | glance Pending 2m | 20:23 |
| jamesbenson | jascott1: kk8s deployment | 20:24 |
| jamesbenson | ? | 20:24 |
| jascott1 | kolla-kubernetes (kk8s) | 20:24 |
| jamesbenson | thx | 20:24 |
| *** mnasiadka has joined #openstack-kolla | 20:25 | |
| openstackgerrit | David Moreau Simard proposed openstack/kolla-ansible master: gzip the ARA report after generating it https://review.openstack.org/468155 | 20:26 |
| *** mgoddard has quit IRC | 20:27 | |
| dmsimard | could I get some quick reviews on ^ ? to alleviate load on logs.o.,o | 20:28 |
| *** mnasiadka has quit IRC | 20:30 | |
| *** mnasiadka has joined #openstack-kolla | 20:33 | |
| *** mewald has quit IRC | 20:37 | |
| kfox1111 | sdake: not sure. I have not used taht functionality at all yet. :/ | 20:38 |
| *** mnasiadka has quit IRC | 20:38 | |
| sdake | kfox1111 sounds fair - mind i bounce some debug ideas off you as I have them - even though you have no experinece with it? | 20:38 |
| kfox1111 | sure. :) | 20:38 |
| *** kbyrne has quit IRC | 20:38 | |
| sdake | kfox1111 specifically is a provisioner required to get nfs to work | 20:38 |
| sdake | kfox1111 i got nfs to work on a different cluster connected to NFS served up via linux | 20:39 |
| sdake | however NFS served up via netapp returns read only filesystem | 20:39 |
| sdake | i.e. the first case works -> linux serving nfs | 20:39 |
| kfox1111 | thats a bit weird. | 20:39 |
| sdake | the second case fails -> netapp serving up nfs | 20:39 |
| sdake | agreed i don't understand it at all | 20:39 |
| kfox1111 | in btoh cases, you manually created all the pv's? | 20:39 |
| sdake | kfox1111 i deployed with service charts - however hacked them to run with my pvs | 20:40 |
| *** manheim has joined #openstack-kolla | 20:41 | |
| sdake | describe of pvs: https://da.gd/QftD8 -> https://paste.fedoraproject.org/paste/J0-7OBxbquv8VZ4cwPjMTl5M1UNdIGYhyRLivL9gydE=/ | 20:42 |
| kfox1111 | we should make the pv charts optional so users can just make them themselves. | 20:42 |
| sdake | note ReadONly is false | 20:42 |
| *** mnasiadka has joined #openstack-kolla | 20:42 | |
| kfox1111 | were they created in the netapp as rw? | 20:42 |
| sdake | https://da.gd/2LWv -> https://paste.fedoraproject.org/paste/zSvM-eT20xp5fsONKa9biF5M1UNdIGYhyRLivL9gydE=/ | 20:43 |
| sdake | kfox1111 i can mount the netapp from linux just fine and write files | 20:43 |
| sdake | that second paste is the pvcs | 20:44 |
| sbezverk | kfox1111 : thanks, let's hope it will go through | 20:44 |
| kfox1111 | sbezverk: +1 | 20:45 |
| sdake | https://da.gd/ncuBB -> https://paste.fedoraproject.org/paste/eKLoilifvj3Mylc7nuxlj15M1UNdIGYhyRLivL9gydE=/ | 20:45 |
| sdake | kfox1111 you will see that mariadb thinks the filesystem is readonly | 20:45 |
| jascott1 | sdake what is the reason behind using `hosts: localhost` in destroy playbook | 20:45 |
| kfox1111 | sdake: and the pv view? | 20:45 |
| sdake | https://paste.fedoraproject.org/paste/J0-7OBxbquv8VZ4cwPjMTl5M1UNdIGYhyRLivL9gydE=/ | 20:45 |
| *** manheim has quit IRC | 20:45 | |
| sbezverk | kfox1111: could you ack openvswitch change? it seems rabbitmq-mgmt is dependent on it.. | 20:46 |
| kfox1111 | sbezverk: did a while ago. | 20:46 |
| kfox1111 | sdake: very strange... | 20:46 |
| kfox1111 | selinux disabled? | 20:46 |
| sbezverk | kfox1111 : it got lost I think | 20:46 |
| sdake | kfox1111 yup | 20:46 |
| kfox1111 | sbezverk: k. got the link handy? | 20:46 |
| *** athomas has quit IRC | 20:46 | |
| *** mnasiadka has quit IRC | 20:47 | |
| kfox1111 | sdake: oh.... um... idea.... | 20:47 |
| sdake | all ideas = winning :) | 20:47 |
| kfox1111 | some remote file servers expect unix uid's to exist on them or to an ldap instance referenced by them. | 20:47 |
| *** powerd has joined #openstack-kolla | 20:47 | |
| kfox1111 | or, it has root squash turned on. | 20:47 |
| sdake | [admin@h7-node0 kolla]$ ls -ld mariadb | 20:48 |
| sdake | drwxrwxrwx. 2 42434 42434 4096 May 25 14:52 mariadb | 20:48 |
| *** chas has joined #openstack-kolla | 20:48 | |
| sdake | that is the mariadb PID in use | 20:48 |
| kfox1111 | sdake: so... lets verify... | 20:49 |
| *** shaggydog97 has quit IRC | 20:49 | |
| kfox1111 | add that uid to your host, su to it, and see if it can access the nfs mount directly. | 20:49 |
| sdake | cool | 20:49 |
| kfox1111 | the other thing to check is if rootsquash is enabled on the nfs mount. | 20:49 |
| kfox1111 | if the init container is runnign root, it may map the user to 'nobody' or an equiv, | 20:49 |
| kfox1111 | and it will look read only to th root user. | 20:50 |
| sdake | [admin@h7-node0 mariadb]$ touch a | 20:50 |
| sdake | [admin@h7-node0 mariadb]$ ls -l | 20:50 |
| sdake | total 0 | 20:50 |
| sdake | -rw-rw-r--. 1 admin admin 0 May 25 16:49 a | 20:50 |
| sdake | [admin@h7-node0 mariadb]$ pwd | 20:50 |
| kolla-slack1 | <sbezverk> @kfox1111: https://kubernetes.slack.com/team/kfox1111 | 20:50 |
| sbezverk | kfox1111: sorry you were right | 20:51 |
| kfox1111 | sbezverk: no worries. | 20:51 |
| kfox1111 | sdake: admin on the applience != root using the export. | 20:51 |
| kfox1111 | \usuauly its the latter that gets squashed. | 20:51 |
| *** chas has quit IRC | 20:52 | |
| *** mnasiadka has joined #openstack-kolla | 20:53 | |
| sdake | i can touch a file as uid=0 and that works | 20:54 |
| sdake | and the resulting uid = 0 | 20:54 |
| sdake | so think its not root squash | 20:54 |
| kfox1111 | hmm.. ok. not root squash then. | 20:54 |
| sbezverk | sdake : there is always a chance to be a valid k8s bug | 20:55 |
| sbezverk | I would suggest to check with sig-storage folks | 20:55 |
| sdake | sbezverk although it works with nfs from a linux host? | 20:55 |
| sbezverk | especially because of that | 20:56 |
| sbezverk | it means kubelet mounts it incorrectl;y | 20:56 |
| sbezverk | or using wrong credentials | 20:56 |
| kfox1111 | I know lustre moutns get grumpy if the uid's used to write to it don't exist in the pw file or getent of the mds. | 20:57 |
| kfox1111 | so the other test is to make that uid a user on the box, then mount it locally, swithc to the user and test. | 20:57 |
| *** mnasiadka has quit IRC | 20:58 | |
| sdake | kfox1111 i did that as well | 20:58 |
| sdake | one thing I noticed is the netapp only has nfsv3 | 20:58 |
| sdake | is there a way to override the nfs version in the pv? | 20:58 |
| kfox1111 | sdake: did it work? | 20:58 |
| sdake | yup that worked | 20:58 |
| kfox1111 | hmm... | 20:59 |
| kfox1111 | I submitted a k8s issue for nfs mount options months ago. | 20:59 |
| kfox1111 | I know it was beign worked on, but I'm not sure it made it yet. | 20:59 |
| * sdake zugs | 20:59 | |
| kfox1111 | sdake. it was. but time to upgrade. :) | 21:00 |
| kfox1111 | it was fixed in 1.6 | 21:00 |
| kfox1111 | https://github.com/kubernetes/kubernetes.github.io/pull/2743 | 21:00 |
| *** mgoddard has joined #openstack-kolla | 21:00 | |
| *** mnasiadka has joined #openstack-kolla | 21:03 | |
| *** itlinux has joined #openstack-kolla | 21:05 | |
| *** ducttape_ has quit IRC | 21:06 | |
| *** jtriley has quit IRC | 21:06 | |
| *** mnasiadka has quit IRC | 21:07 | |
| *** jascott1 has quit IRC | 21:08 | |
| *** jascott1 has joined #openstack-kolla | 21:08 | |
| *** chas has joined #openstack-kolla | 21:09 | |
| *** manheim has joined #openstack-kolla | 21:10 | |
| *** mgoddard has quit IRC | 21:11 | |
| *** mnasiadka has joined #openstack-kolla | 21:12 | |
| *** jascott1 has quit IRC | 21:13 | |
| *** chas has quit IRC | 21:14 | |
| *** manheim has quit IRC | 21:15 | |
| *** mnasiadka has quit IRC | 21:17 | |
| *** mnasiadka has joined #openstack-kolla | 21:22 | |
| *** m1dev has joined #openstack-kolla | 21:23 | |
| *** jheroux has quit IRC | 21:25 | |
| *** mnasiadka has quit IRC | 21:27 | |
| *** ddyer has quit IRC | 21:29 | |
| rwellum | kfox1111: question about br-ex - typically do I have to create this bridge or is this done when running through the deployment? | 21:29 |
| *** chas has joined #openstack-kolla | 21:30 | |
| kfox1111 | rwellum: its configurable in the openvswitch microcharts. | 21:30 |
| kfox1111 | I tipically create it outside the deployment, but I also tend to run openvswitch on the host outside of k8s anyway. | 21:31 |
| kfox1111 | I've only got one nic, so use openvswitch to do the vlan stuff for the host as well. | 21:31 |
| rwellum | Ok because we use it in the cloud.yaml - for keepalived to bind VIP to it - but not sure if I have to create it and attach my neutron interface to it or not. | 21:32 |
| *** mnasiadka has joined #openstack-kolla | 21:33 | |
| *** dave-mccowan has quit IRC | 21:33 | |
| *** mgoddard has joined #openstack-kolla | 21:34 | |
| *** chas has quit IRC | 21:34 | |
| *** mnasiadka has quit IRC | 21:37 | |
| *** manheim has joined #openstack-kolla | 21:38 | |
| *** ddyer has joined #openstack-kolla | 21:39 | |
| *** lucasxu has quit IRC | 21:39 | |
| *** chas has joined #openstack-kolla | 21:40 | |
| *** jamesPR has joined #openstack-kolla | 21:40 | |
| *** manheim has quit IRC | 21:42 | |
| *** itlinux has quit IRC | 21:42 | |
| *** mnasiadka has joined #openstack-kolla | 21:43 | |
| *** mnasiadka has quit IRC | 21:47 | |
| *** Pavo has joined #openstack-kolla | 21:50 | |
| sbezverk | rwellum : do not forget then to run job to bring br-ex up, otherwise keepalived will fail | 21:51 |
| *** mnasiadka has joined #openstack-kolla | 21:52 | |
| *** iniazi has quit IRC | 21:53 | |
| *** chas has quit IRC | 21:54 | |
| *** chas has joined #openstack-kolla | 21:54 | |
| *** mgoddard has quit IRC | 21:55 | |
| *** Pavo has quit IRC | 21:56 | |
| *** mnasiadka has quit IRC | 21:56 | |
| *** jamesbenson has quit IRC | 21:58 | |
| *** chas has quit IRC | 21:59 | |
| *** manheim has joined #openstack-kolla | 22:04 | |
| *** manheim has quit IRC | 22:04 | |
| *** manheim_ has joined #openstack-kolla | 22:06 | |
| *** jascott1 has joined #openstack-kolla | 22:09 | |
| emccormick | 22:14 | |
| *** jascott1 has quit IRC | 22:14 | |
| *** chas has joined #openstack-kolla | 22:15 | |
| *** mnasiadka has joined #openstack-kolla | 22:19 | |
| *** chas has quit IRC | 22:20 | |
| sdake | kfox1111 i hae confirmed it is not a nfsv4 / nfsv3 issue by forcing nfsv3 on the environment without the netapp | 22:20 |
| *** itlinux has joined #openstack-kolla | 22:22 | |
| *** mnasiadka has quit IRC | 22:24 | |
| *** Pavo has joined #openstack-kolla | 22:25 | |
| *** Pavo has quit IRC | 22:28 | |
| sbezverk | sdake : file a bug somebody from sig-storage will look at it | 22:39 |
| sdake | i found the root cause - there is a .snapshot directory on the netapp which is a read-only filesystem | 22:40 |
| sdake | chmod tries to chmod it during startup - which implodes | 22:40 |
| sdake | sbezverk http://paste.openstack.org/show/610688/ | 22:42 |
| *** powerd has quit IRC | 22:42 | |
| kfox1111 | sdake: fun. | 22:43 |
| sbezverk | sdake: what creates .snapshot and why mariadb see it? could it be that nfs mount is not correctly defined? | 22:45 |
| sdake | https://library.netapp.com/ecmdocs/ECMP1368826/html/GUID-56684CAA-FA05-450C-93D4-C12D8A695F07.html | 22:46 |
| sbezverk | sdake: you should probably disable snapshot completely | 22:47 |
| sdake | if it were my choice.. | 22:47 |
| *** Pavo has joined #openstack-kolla | 22:47 | |
| *** jascott1 has joined #openstack-kolla | 22:47 | |
| *** ducttape_ has joined #openstack-kolla | 22:48 | |
| sbezverk | sdake: hm, it is odd if they force you to use it even if you do not need | 22:48 |
| kfox1111 | sdake: can you do a subpath in the pv? | 22:48 |
| *** unicell has quit IRC | 22:49 | |
| kfox1111 | ie, make netapp:/foo volume, then in the pv, reference netapp:/foo/data or something as the root. | 22:49 |
| *** unicell1 has joined #openstack-kolla | 22:49 | |
| sdake | kfox1111 / is what is exported from the netapp | 22:49 |
| sdake | kfox1111 i htink every mount to anything in that path creates a new subvolume | 22:49 |
| kfox1111 | yeah. but I'm saying imoprt something under that in the pv. | 22:49 |
| *** iniazi has joined #openstack-kolla | 22:49 | |
| sdake | i am doing that | 22:49 |
| *** huzhengchuan_ has quit IRC | 22:49 | |
| *** huzhengchuan has quit IRC | 22:49 | |
| kfox1111 | and each subdir has a .snapshot thingy? | 22:50 |
| sdake | each one a pv creates yes | 22:50 |
| kfox1111 | the pv shoudln't be creating a volume though... | 22:50 |
| *** Pavo has quit IRC | 22:50 | |
| sdake | when it connects to the netapp, it creates a volume | 22:50 |
| kfox1111 | oh. you have a netapp dynamic provisioner? | 22:50 |
| sdake | no | 22:50 |
| sdake | / is exported from the netapp | 22:51 |
| sdake | if you mount /vol/abc | 22:51 |
| sdake | abc gets created with a .snapshot read only filesystem | 22:51 |
| sbezverk | vol options volname nosnap [on|off] | 22:51 |
| kfox1111 | that seems broken. | 22:51 |
| kfox1111 | yeah. that thingy. do that. :) | 22:51 |
| sdake | sbezverk you assume I have access to the netapp :) | 22:51 |
| sbezverk | oh I thought it is in your home lab | 22:52 |
| *** Pavo has joined #openstack-kolla | 22:52 | |
| *** powerd has joined #openstack-kolla | 22:53 | |
| kbaegis | If anyone has time, new blocker on gentoo: https://hastebin.com/owomofepib.rb | 22:53 |
| sdake | I suspect someone has already run into this: commit fe61742d1c6ba62ba98cc0f97f0d1faafe29139f | 22:54 |
| sdake | Author: Jeffrey Zhang <zhang.lei.fly@gmail.com> | 22:54 |
| sdake | Date: Tue May 9 11:18:39 2017 +0800 | 22:54 |
| sdake | Catch exception during calling os.chmod in set_configs.py | 22:54 |
| sdake | 22:54 | |
| sdake | set_perms may fail in some case, but it is not critical. Just leave a | 22:54 |
| sdake | warning rather than exit. | 22:54 |
| kbaegis | Very nearly have non-root build/deploy using gentoo | 22:54 |
| kbaegis | What does basestring refer to? | 22:55 |
| sbezverk | sdake: I got mac :) | 22:55 |
| *** chas has joined #openstack-kolla | 22:56 | |
| *** chas has quit IRC | 23:01 | |
| *** egonzalez has quit IRC | 23:06 | |
| *** itlinux has quit IRC | 23:07 | |
| *** Pavo has quit IRC | 23:08 | |
| *** manheim_ has quit IRC | 23:10 | |
| *** rhallisey has quit IRC | 23:15 | |
| *** ducttape_ has quit IRC | 23:15 | |
| *** ducttape_ has joined #openstack-kolla | 23:16 | |
| *** rhallisey has joined #openstack-kolla | 23:28 | |
| *** ducttape_ has quit IRC | 23:30 | |
| *** powerd has quit IRC | 23:33 | |
| *** rhallisey has quit IRC | 23:36 | |
| *** chas has joined #openstack-kolla | 23:37 | |
| *** rwellum has quit IRC | 23:41 | |
| *** sayantan_ has quit IRC | 23:42 | |
| *** chas has quit IRC | 23:42 | |
| *** sayantan_ has joined #openstack-kolla | 23:42 | |
| *** manheim has joined #openstack-kolla | 23:46 | |
| sean-k-mooney | has anyone had issues pushing images to a registry http://paste.openstack.org/show/610691/ | 23:49 |
| *** yingjun has joined #openstack-kolla | 23:49 | |
| sdake | should this work: mariadb: | 23:50 |
| sdake | all: | 23:50 |
| sdake | image_full: "docker.io/kolla/centos-source-mariadb:4.0.2" | 23:50 |
| sdake | 23:50 | |
| sdake | (in cloud.yaml) | 23:50 |
| *** manheim has quit IRC | 23:50 | |
| kfox1111 | sdake: I think so. | 23:51 |
| kfox1111 | do a --dry-run --debug to see. | 23:51 |
| *** kbaegis has quit IRC | 23:54 | |
| *** eaguilar has joined #openstack-kolla | 23:57 | |
| *** hongbin has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!