| *** Pavo has joined #openstack-kolla | 00:02 | |
| spsurya | morning guys | 00:03 |
|---|---|---|
| *** Pavo has quit IRC | 00:06 | |
| kbaegis | Morning | 00:10 |
| *** Seyong has joined #openstack-kolla | 00:11 | |
| *** yingjun has joined #openstack-kolla | 00:13 | |
| *** kbaegis has quit IRC | 00:17 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 00:18 |
| *** srwilkers has joined #openstack-kolla | 00:23 | |
| masuberu | good morning all | 00:24 |
| masuberu | according to the kolla documentation - •api_interface - This interface is used for the management network. The management network is the network OpenStack services uses to communicate to each other and the databases. There are known security risks here, so it’s recommended to make this network internal, not accessible from outside. Defaults to network_interface | 00:24 |
| masuberu | however the op for horizon is within the api interface, so that means I will have to setup a NAT between the api interface and the normal users network? | 00:25 |
| *** stefan_bo has joined #openstack-kolla | 00:38 | |
| *** stefan_bo has quit IRC | 00:43 | |
| *** vhosakot has joined #openstack-kolla | 00:44 | |
| *** ssurana has quit IRC | 00:45 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 00:46 |
| *** vhosakot has quit IRC | 00:47 | |
| *** duonghq has joined #openstack-kolla | 00:47 | |
| duonghq | morning guys | 00:49 |
| *** cuongnv has joined #openstack-kolla | 00:56 | |
| *** Pavo has joined #openstack-kolla | 00:57 | |
| *** tovin07_ has joined #openstack-kolla | 01:04 | |
| *** inc0 has quit IRC | 01:07 | |
| *** inc0 has joined #openstack-kolla | 01:07 | |
| *** stefan_bo has joined #openstack-kolla | 01:17 | |
| *** Pavo has quit IRC | 01:23 | |
| *** huzhengchuan has joined #openstack-kolla | 01:30 | |
| *** hrw has quit IRC | 01:31 | |
| spsurya | sup duonghq | 01:32 |
| *** hrw has joined #openstack-kolla | 01:33 | |
| duonghq | hi spsurya | 01:34 |
| *** zul has quit IRC | 01:44 | |
| openstackgerrit | pangliye proposed openstack/kolla master: Install docker_client and groupadd https://review.openstack.org/465880 | 02:17 |
| *** salv-orlando has joined #openstack-kolla | 02:54 | |
| *** srwilkers has quit IRC | 03:06 | |
| *** kbaegis has joined #openstack-kolla | 03:09 | |
| spsurya | duonghq: https://review.openstack.org/#/c/465474/ | 03:16 |
| spsurya | hanging fruit | 03:16 |
| spsurya | ;) | 03:16 |
| *** caowei has joined #openstack-kolla | 03:23 | |
| *** goldyfruit has joined #openstack-kolla | 03:24 | |
| *** stefan_bo has quit IRC | 03:33 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role https://review.openstack.org/465862 | 03:36 |
| *** klindgren has joined #openstack-kolla | 03:43 | |
| *** klindgren_ has quit IRC | 03:44 | |
| *** goldyfruit has quit IRC | 03:44 | |
| *** kbaegis has quit IRC | 03:51 | |
| *** xinliang has quit IRC | 03:54 | |
| *** xinliang has joined #openstack-kolla | 03:55 | |
| *** xinliang has quit IRC | 03:55 | |
| *** xinliang has joined #openstack-kolla | 03:55 | |
| *** salv-orlando has quit IRC | 03:58 | |
| *** caowei has quit IRC | 04:06 | |
| *** rwsu has joined #openstack-kolla | 04:18 | |
| *** stefan_bo has joined #openstack-kolla | 04:24 | |
| duonghq | spsurya, done | 04:36 |
| *** caowei has joined #openstack-kolla | 04:37 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Updated the quick start guide https://review.openstack.org/465474 | 04:37 |
| *** caowei has quit IRC | 04:41 | |
| *** stefan_bo has quit IRC | 04:41 | |
| *** caowei has joined #openstack-kolla | 04:45 | |
| *** chandankumar has quit IRC | 04:46 | |
| *** jaosorior has joined #openstack-kolla | 04:52 | |
| spsurya | duonghq: +1 | 04:53 |
| *** janki has joined #openstack-kolla | 04:59 | |
| *** stefan_bo has joined #openstack-kolla | 05:06 | |
| *** caowei has quit IRC | 05:08 | |
| *** skramaja has joined #openstack-kolla | 05:11 | |
| *** salv-orlando has joined #openstack-kolla | 05:17 | |
| *** cuongnv has quit IRC | 05:25 | |
| *** caowei has joined #openstack-kolla | 05:34 | |
| *** cuongnv has joined #openstack-kolla | 05:38 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services https://review.openstack.org/466268 | 05:38 |
| *** yingjun has quit IRC | 05:54 | |
| sdake | sbezverk i heard from sam that insepctor is working | 06:01 |
| *** jascott1 has joined #openstack-kolla | 06:02 | |
| *** salv-orlando has quit IRC | 06:04 | |
| *** unicell has joined #openstack-kolla | 06:05 | |
| *** salv-orlando has joined #openstack-kolla | 06:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services https://review.openstack.org/466268 | 06:13 |
| *** chas has joined #openstack-kolla | 06:19 | |
| *** athomas has joined #openstack-kolla | 06:29 | |
| *** chas has quit IRC | 06:37 | |
| *** chas has joined #openstack-kolla | 06:37 | |
| *** chas has quit IRC | 06:41 | |
| *** chas has joined #openstack-kolla | 06:41 | |
| *** zhubingbing has joined #openstack-kolla | 06:49 | |
| *** chas has quit IRC | 06:55 | |
| *** shardy has joined #openstack-kolla | 06:55 | |
| *** chas has joined #openstack-kolla | 06:55 | |
| *** chas has quit IRC | 06:57 | |
| *** chas has joined #openstack-kolla | 06:57 | |
| *** matrohon has joined #openstack-kolla | 06:57 | |
| *** salv-orlando has quit IRC | 07:01 | |
| *** salv-orlando has joined #openstack-kolla | 07:03 | |
| *** jascott1 has quit IRC | 07:03 | |
| *** chas has quit IRC | 07:06 | |
| *** chas has joined #openstack-kolla | 07:06 | |
| *** salv-orlando has quit IRC | 07:08 | |
| *** chas has quit IRC | 07:11 | |
| *** pcaruana has joined #openstack-kolla | 07:11 | |
| *** pcaruana has quit IRC | 07:12 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove the unnecessary jinja2 templating delimiters https://review.openstack.org/466612 | 07:13 |
| *** pcaruana has joined #openstack-kolla | 07:13 | |
| *** korzen has joined #openstack-kolla | 07:15 | |
| *** chenn has joined #openstack-kolla | 07:27 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Use the different nova_cell0 database password from nova https://review.openstack.org/466615 | 07:35 |
| masuberu | good afternoon all, I am running into an issue when trying to use kolla-ansible to deploy openstack https://pastebin.com/raw/wZp6SXgV | 07:36 |
| masuberu | basically it fails during the ceph setup | 07:36 |
| *** david-lyle has quit IRC | 07:36 | |
| masuberu | cph-mon container does not start | 07:36 |
| masuberu | apparently a file is missing | 07:36 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Use the different nova_cell0 database password from nova https://review.openstack.org/466615 | 07:36 |
| *** Serlex has joined #openstack-kolla | 07:38 | |
| masuberu | I also tried destroying and redeploying the container but still didn't fix the issue | 07:38 |
| hrw | morning | 07:41 |
| *** david-lyle has joined #openstack-kolla | 07:42 | |
| *** david-lyle has quit IRC | 07:46 | |
| *** egonzalez has joined #openstack-kolla | 07:46 | |
| *** chas has joined #openstack-kolla | 07:47 | |
| *** zhuzeyu has joined #openstack-kolla | 07:51 | |
| *** blallau has joined #openstack-kolla | 07:54 | |
| *** manheim_ has joined #openstack-kolla | 07:58 | |
| *** jmccarthy has joined #openstack-kolla | 08:02 | |
| manheim_ | hello | 08:03 |
| *** david-lyle has joined #openstack-kolla | 08:03 | |
| egonzalez | sum manheim | 08:04 |
| *** jmccarthy has quit IRC | 08:07 | |
| *** jmccarthy has joined #openstack-kolla | 08:08 | |
| *** huzhengchuan_ has joined #openstack-kolla | 08:09 | |
| jaosorior | mandre: hey dude, can you help me out figuring out what went wrong here https://review.openstack.org/#/c/466268/ ? | 08:17 |
| egonzalez | jaosorior, No package python-keystone-middleware available. | 08:20 |
| egonzalez | http://logs.openstack.org/68/466268/3/check/gate-kolla-dsvm-build-centos-source-centos-7/946d410/logs/build/novajoin-base.txt.gz | 08:20 |
| *** caowei has quit IRC | 08:21 | |
| jaosorior | egonzalez: hey! thanks! sorry about the poking, still figuring out where logs are and such. | 08:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services https://review.openstack.org/466268 | 08:22 |
| egonzalez | jaosorior, open the console.html log and scroll down, there should be something like 2017-05-22 06:59:51.688101 | CRITICAL:tests.test_build:>>> Expected image 'novajoin-base' to succeed! | 08:22 |
| jaosorior | egonzalez: so I saw that. But didn't know novajoin-base had it's own log file. | 08:23 |
| egonzalez | once you know the image, open logs/build/<failed_image_logs> | 08:23 |
| *** caowei has joined #openstack-kolla | 08:24 | |
| jaosorior | egonzalez: that's good info. Thanks | 08:25 |
| *** n0isyn0i1e is now known as n0isyn0ise | 08:25 | |
| *** lpetrut has joined #openstack-kolla | 08:31 | |
| masuberu | good afternoon all, kolla-ansible deploy fails and ceph-mon does't start. https://pastebin.com/raw/wZp6SXgV | 08:32 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Fix log rotation issue on services https://review.openstack.org/466537 | 08:33 |
| *** janki has quit IRC | 08:35 | |
| *** janki has joined #openstack-kolla | 08:36 | |
| *** jchhatbar has joined #openstack-kolla | 08:37 | |
| *** dasTor has quit IRC | 08:37 | |
| *** dasTor has joined #openstack-kolla | 08:39 | |
| *** janki has quit IRC | 08:40 | |
| *** pbourke has joined #openstack-kolla | 08:43 | |
| egonzalez | masuberu, can you share previous ceph tasks, docker ps -a, docker logs ceph_mon and cat /etc/kolla/ceph-mon/config.json? made any customization in the files? | 08:44 |
| *** jchhatbar is now known as janki | 08:44 | |
| *** feisky has joined #openstack-kolla | 08:47 | |
| manheim_ | caowei you on? | 08:51 |
| jaosorior | is opendaylight having issues? | 08:52 |
| egonzalez | jaosorior, what kind of issues? currently only images are built but not deployed by ansible or k8s | 08:52 |
| jaosorior | egonzalez: oh, nevermind, it was a red herring on the log | 08:53 |
| egonzalez | jaosorior, build errors for images at https://github.com/openstack/kolla/blob/master/tests/test_build.py#L75 are ignored by the tests | 08:55 |
| jaosorior | egonzalez: it seems that the novajoin build failed http://logs.openstack.org/68/466268/4/check/gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial/9d6047e/console.html but I added it to the ignored_images | 08:55 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance https://review.openstack.org/452176 | 08:55 |
| jaosorior | I mean | 08:55 |
| jaosorior | exluded_images | 08:55 |
| jaosorior | egonzalez: right, I added them https://review.openstack.org/#/c/466268/4/tests/test_build.py | 08:56 |
| egonzalez | actually is failing aodh-base | 08:56 |
| jaosorior | oh | 08:56 |
| egonzalez | aodh removed etc/aodh folder and now uses config generator... | 08:57 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Fix aodh-base image https://review.openstack.org/466639 | 09:03 |
| hrw | restarted kolla builds on aarch64 (moonshot in mlab), ppc64le (power8 at Red Hat), x86_64 (my work laptop) | 09:05 |
| masuberu | egonzalez, ceph-mon container doesn't start, can't check logs | 09:07 |
| egonzalez | masuberu, before fetching keyrings mons are started, any other task failed before? `docker logs ceph_mon` work on failed containers | 09:10 |
| matrohon | egonzalez, hi | 09:12 |
| egonzalez | matrohon, sup | 09:12 |
| matrohon | egonzalez, inc0 : I fanally wrokaround my issues concerning the nova upgrade from newton to ocata | 09:12 |
| matrohon | https://bugs.launchpad.net/kolla-ansible/+bug/1692464 | 09:12 |
| openstack | Launchpad bug 1692464 in kolla-ansible "nova upgrade failures from newton to ocata" [Undecided,New] | 09:12 |
| matrohon | egonzalez, did you succeed at reproducing my bug? | 09:13 |
| egonzalez | matrohon, my upgrade succedd, although the workaround fix in the bug report should not fix the issue | 09:13 |
| matrohon | egonzalez, why do you say it doesn't fix the issue | 09:14 |
| matrohon | ? | 09:14 |
| egonzalez | matrohon, tested 3.0.3 to 4.0.0 not 4.0.3(do you have any public 4.0.3 images in some kind of repo to test with that version?) | 09:15 |
| matrohon | egonzalez, it's in a private repo, sorry | 09:15 |
| matrohon | egonzalez, but there is no custom config on the built images | 09:16 |
| egonzalez | matrohon, because cell0 dont have a transport url setting, you hide the issue and "mounted" None cell (legacy) into cell0 | 09:16 |
| matrohon | egonzalez, so my upgraded cloud is not operational because of the lack of the transport-url in cell0 config? | 09:18 |
| matrohon | I didn't try it yet :( | 09:18 |
| egonzalez | cell0 just points to a nova_cell0 db for failed and unscheduled instances, real instances are mapped into a cell called None (this have transport_url and points to nova db) | 09:18 |
| egonzalez | matrohon, if you use nova-manage cell_v2 list_cells --verbose will see connections for both cells | 09:19 |
| matrohon | egonzalez, I do, but I'm not sure to perfectly understand the cell magic behing... I need to read more docs! | 09:22 |
| egonzalez | matrohon, https://docs.openstack.org/developer/nova/cells.html#cells-v2 | 09:22 |
| egonzalez | yep, is quite hard to understand at the beginning | 09:22 |
| openstackgerrit | Chen proposed openstack/kolla-ansible master: Fix some wording issues on a comment https://review.openstack.org/466646 | 09:22 |
| egonzalez | matrohon, i'm going to upgrade again to latest ocata images | 09:26 |
| egonzalez | matrohon, 3.0.3 from dockerhub to 4.0.3(latest ocata) | 09:27 |
| zhubingbing | hi gus | 09:27 |
| zhubingbing | hi guys | 09:27 |
| matrohon | egonzalez, great | 09:27 |
| egonzalez | sup zhubingbing | 09:27 |
| zhubingbing | hi egonzalez | 09:27 |
| matrohon | egonzalez, the cell setup issue might be a subsequent issue of the first failure described in my bug report | 09:28 |
| egonzalez | matrohon, yep, is that what i want to re-verify, but may be an issue with other service | 09:29 |
| egonzalez | is fluent deployed and running in your nodes, from newton to ocata heka is replaced with fluentd | 09:30 |
| matrohon | I've run an upgrade to 4.0.2, it seems I'v to run a new upgrade to 4.0.3 :) | 09:30 |
| egonzalez | to 4.0.2 worked? | 09:31 |
| matrohon | egonzalez, with the workaround in teh bug report, the upgrade worked | 09:32 |
| masuberu | egonzalez, this is the only thing I can see form the logs --> https://pastebin.com/raw/wZp6SXgV | 09:38 |
| egonzalez | masuberu, resumed the cell creation and upgrade process http://paste.openstack.org/show/610138/ | 09:41 |
| egonzalez | matrohon, ^^ wrong name | 09:42 |
| egonzalez | masuberu, not use docker run into a failed container, use docker logs to see why the container failed to start | 09:42 |
| matrohon | egonzalez, ack, that what I saw for the upgrade, but I had to run another command for the upgrade to work : nova-manage --verbose cell_v2 map_cell_and_hosts | 09:45 |
| *** cuongnv has quit IRC | 09:49 | |
| openstackgerrit | Martin Mágr proposed openstack/kolla master: Sensu images https://review.openstack.org/463593 | 09:51 |
| egonzalez | matrohon, map_hosts is done by simple_cell-setup https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L1112 | 09:51 |
| *** sambetts_ is now known as sambetts | 09:51 | |
| *** zhuzeyu has quit IRC | 09:52 | |
| *** tovin07_ has quit IRC | 09:55 | |
| egonzalez | matrohon, can you share all nova upgrade tasks output with -vvv? | 10:01 |
| egonzalez | btw, map_cell_and_hosts is needed to finish upgrade or to allow create new instances after upgrade? | 10:03 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (2) https://review.openstack.org/466661 | 10:07 |
| *** zhuzeyu has joined #openstack-kolla | 10:08 | |
| *** chas has quit IRC | 10:08 | |
| *** chas has joined #openstack-kolla | 10:09 | |
| matrohon | egonzalez, my upgrade ha worked, I wo't be able to downgrade and reupgrade to get logs :) | 10:11 |
| matrohon | map_cell_and_hosts is needed to finish the upgrade | 10:12 |
| egonzalez | matrohon, can you share cell_v2 list_cells --verbose (hide your passwords ;) | 10:12 |
| egonzalez | matrohon, to ensure cells mapping are properly created | 10:12 |
| matrohon | otherwise, the "Running nova bootstrap" taskfails | 10:13 |
| matrohon | sure | 10:13 |
| *** chas has quit IRC | 10:13 | |
| matrohon | egonzalez, http://paste.openstack.org/show/610146/ | 10:15 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Clean up unused sanity check modules https://review.openstack.org/465515 | 10:15 |
| egonzalez | matrohon, my upgrade worked, going to share logs to check the outputs | 10:15 |
| egonzalez | matrohon, looks good that | 10:15 |
| matrohon | egonzalez, ok, the issue might come from my config then... | 10:16 |
| matrohon | egonzalez, I did previous trial that failed, before having a correct inventory, those upgrade trial might have moved my cloud in a messy state | 10:18 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (2) https://review.openstack.org/466661 | 10:19 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 10:22 |
| *** stefan_bo has quit IRC | 10:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Mount cinder volume for cinder-volume and cinder-backup all the time https://review.openstack.org/466666 | 10:25 |
| *** gfidente has joined #openstack-kolla | 10:26 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (3) https://review.openstack.org/466667 | 10:26 |
| egonzalez | matrohon, those are the steps I took for the upgrade http://paste.openstack.org/show/610147/ | 10:26 |
| egonzalez | and the outputs from the upgrade tasks and list cells | 10:27 |
| *** caowei has quit IRC | 10:28 | |
| egonzalez | matrohon, maybe if you played with various upgrade tries over the same environment broke something, many nova tasks first checks if some resource exists | 10:30 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (3) https://review.openstack.org/466669 | 10:32 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a Kolla 'devstack' mode https://review.openstack.org/454690 | 10:32 |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (4) https://review.openstack.org/466669 | 10:33 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Use internal interface to connect with keystone_ssh https://review.openstack.org/466674 | 10:35 |
| *** chas has joined #openstack-kolla | 10:43 | |
| *** chas has quit IRC | 10:47 | |
| *** chas has joined #openstack-kolla | 10:47 | |
| *** kbaegis has joined #openstack-kolla | 10:48 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note https://review.openstack.org/466676 | 10:53 |
| openstackgerrit | Pavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional https://review.openstack.org/464620 | 10:54 |
| *** numans has joined #openstack-kolla | 10:59 | |
| *** emccormick has joined #openstack-kolla | 11:01 | |
| *** bjolo has joined #openstack-kolla | 11:03 | |
| *** rwellum has joined #openstack-kolla | 11:04 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Mount cinder volume for cinder-volume and cinder-backup all the time https://review.openstack.org/466683 | 11:16 |
| openstackgerrit | Merged openstack/kolla-ansible master: Mount cinder volume for cinder-volume and cinder-backup all the time https://review.openstack.org/466666 | 11:17 |
| *** bigkol has joined #openstack-kolla | 11:24 | |
| bigkol | HELP | 11:24 |
| *** shardy has quit IRC | 11:27 | |
| *** dave-mccowan has joined #openstack-kolla | 11:30 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note https://review.openstack.org/466676 | 11:31 |
| *** emccormick has quit IRC | 11:31 | |
| *** zul has joined #openstack-kolla | 11:36 | |
| *** athomas has quit IRC | 11:38 | |
| bjolo | howdy | 11:45 |
| bjolo | Jeffrey4l, around? | 11:47 |
| *** jchhatbar has joined #openstack-kolla | 11:47 | |
| *** janki has quit IRC | 11:49 | |
| *** bigkol has quit IRC | 11:53 | |
| *** srwilkers has joined #openstack-kolla | 11:57 | |
| *** rhallisey has joined #openstack-kolla | 11:58 | |
| kbaegis | Hi all. I'm failing prechecks on the MountFlags=shared. I'm also no longer using systemd | 11:59 |
| kbaegis | Can anyone help me configure sysvinit to enable the shared flag? I can only seem to find instructions for the systemd setup | 12:00 |
| egonzalez | kbaegis, https://github.com/openstack/kolla-ansible/blob/stable/ocata/doc/quickstart.rst#additional-steps-for-upstart-and-other-non-systemd-distros | 12:02 |
| *** bjolo has quit IRC | 12:05 | |
| kbaegis | egonzalez: Thanks! I've actually done all that already. Still complains about MountFlags=shared not being set | 12:07 |
| *** bjolo has joined #openstack-kolla | 12:07 | |
| kbaegis | specifically neutron | 12:07 |
| kbaegis | I'll post the -vvv dump in a minute. It's running | 12:08 |
| egonzalez | kbaegis, hrm, support for sysv was removed and now only checks for systemd(master) https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L29 | 12:09 |
| egonzalez | in ocata still checking for sysv | 12:09 |
| kbaegis | I'm on stable/ocata | 12:13 |
| kbaegis | Also systemd is an awful demonspawn of an init process | 12:13 |
| kbaegis | Can we add support for sysv again? | 12:13 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix nova cell release note https://review.openstack.org/466676 | 12:14 |
| kbaegis | Here's the -vvv for stable/ocata https://hastebin.com/ekosabowuw.sql | 12:16 |
| egonzalez | kbaegis, this task assume that ubuntu > 14.04 uses systemd ansible_distribution == 'Ubuntu' and ansible_distribution_version > '14.04' | 12:17 |
| *** chenn has quit IRC | 12:19 | |
| *** chenn has joined #openstack-kolla | 12:20 | |
| kbaegis | egonzalez: Interesting choice. :) | 12:20 |
| egonzalez | anyone with galera knowledge have a hint on what can be the issue while upgrading from ocata->master? https://bugs.launchpad.net/kolla-ansible/+bug/1692507 | 12:23 |
| openstack | Launchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New] | 12:23 |
| egonzalez | images and templates didnt change since ocata | 12:24 |
| *** blallau has quit IRC | 12:25 | |
| *** bjolo has quit IRC | 12:37 | |
| kbaegis | So when switching to sysvinit deploy now hangs on "Waiting for nova-compute service up" | 12:40 |
| kbaegis | nova_compute container is up and functional, but it can't be reached | 12:41 |
| egonzalez | kbaegis, if executes nova service-list is there a compute service? | 12:42 |
| kbaegis | Sec. Have to write an admin-rc by hand | 12:43 |
| egonzalez | kbaegis, nope you can use post-deploy | 12:43 |
| egonzalez | that task will only copy a template with adminrc info | 12:44 |
| kbaegis | egonzalez: even if deploy fails? | 12:44 |
| egonzalez | yep, is just a copy file | 12:44 |
| kbaegis | egonzalez: no services show up | 12:44 |
| egonzalez | kbaegis, is rabbitmq running? what branch are you deploying? | 12:45 |
| kbaegis | stable/ocata | 12:45 |
| kbaegis | rabbitmq is certainly working | 12:45 |
| egonzalez | can you get logs from nova-compute? | 12:46 |
| kbaegis | Sure. https://hastebin.com/iqegizawaz.sql | 12:46 |
| egonzalez | kbaegis, logs from /var/lib/docker/volumes/kolla_logs/_data/kolla_logs/nova/nova-compute.log | 12:47 |
| kbaegis | Sure, sec | 12:47 |
| egonzalez | second kolla_logs not needed | 12:47 |
| kbaegis | https://hastebin.com/ebucapenuf.xml | 12:48 |
| *** Pavo has joined #openstack-kolla | 12:48 | |
| kbaegis | cinder service-list works fine btw | 12:50 |
| egonzalez | kbaegis, can you check this docker exec -ti kolla_toolbox mysql -unova -p<NOVA_DB_PASSWORD> -h <VIP> nova -e "SELECT * from services;" | 12:53 |
| kbaegis | Sure | 12:54 |
| kbaegis | sec | 12:54 |
| kbaegis | Why don't I just run that straight in the db container? | 12:55 |
| *** ArminderSingh has joined #openstack-kolla | 12:55 | |
| *** ArminderSingh- has quit IRC | 12:55 | |
| *** chenn has quit IRC | 12:56 | |
| kbaegis | Either way. Ran successfully | 12:56 |
| egonzalez | kbaegis, you can, what is easier for you, i dont use to enter into container or remote hosts, all executed from ansible | 12:56 |
| egonzalez | whats the output? | 12:56 |
| *** chenn has joined #openstack-kolla | 12:57 | |
| kbaegis | https://hastebin.com/ururixanof.rb | 12:57 |
| egonzalez | kbaegis, you are using master images with ocata code | 12:57 |
| kbaegis | No. I kolla-build myself | 12:57 |
| egonzalez | kbaegis, run docker exec nova_api pip freeze and share output | 12:58 |
| egonzalez | but this issue was due a change in nova master for multi cells | 12:59 |
| kbaegis | https://hastebin.com/rawehoticu.makefile | 12:59 |
| *** ArminderSingh has quit IRC | 13:00 | |
| egonzalez | kbaegis, nova==16.0.0.0b2.dev339 is master | 13:00 |
| egonzalez | ocata is 15* | 13:00 |
| kbaegis | egonzalez: Should I kolla-build after pulling 4.0.1? | 13:00 |
| *** manheim_ has quit IRC | 13:01 | |
| egonzalez | kbaegis, nope, pull just download images from repository | 13:01 |
| *** jheroux has joined #openstack-kolla | 13:01 | |
| *** salv-orlando has joined #openstack-kolla | 13:01 | |
| kbaegis | egonzalez: You mean from docker.io? | 13:01 |
| *** manheim_ has joined #openstack-kolla | 13:02 | |
| kbaegis | I'm rebuilding from 14.0.1 rather than master. Feeling kind of silly about that, but I didn't realize it would cause issues like this | 13:03 |
| kbaegis | Thanks egonzalez | 13:03 |
| egonzalez | kbaegis, is an issue of master, openstack projects changes things that can break our deployment | 13:04 |
| egonzalez | in this case ocata ansible dont work to deploy master/pike | 13:04 |
| *** zul has quit IRC | 13:05 | |
| *** manheim_ has quit IRC | 13:06 | |
| *** salv-orlando has quit IRC | 13:07 | |
| *** salv-orlando has joined #openstack-kolla | 13:07 | |
| *** huzhengchuan_ has quit IRC | 13:11 | |
| *** huzhengchuan has quit IRC | 13:12 | |
| *** zul has joined #openstack-kolla | 13:12 | |
| *** chenn has quit IRC | 13:14 | |
| *** chenn has joined #openstack-kolla | 13:15 | |
| *** salv-orlando has quit IRC | 13:21 | |
| *** eanylin has quit IRC | 13:22 | |
| *** manheim_ has joined #openstack-kolla | 13:32 | |
| *** jchhatbar has quit IRC | 13:33 | |
| *** ducttape_ has joined #openstack-kolla | 13:35 | |
| *** myoung|pto is now known as myoung_ | 13:48 | |
| Jeffrey4l | bjolo__, yep | 13:50 |
| *** schwicht has joined #openstack-kolla | 13:52 | |
| *** korzen has quit IRC | 13:53 | |
| openstackgerrit | Kurt Taylor proposed openstack/kolla master: Fix inconsistent build task attempt logging https://review.openstack.org/466738 | 13:59 |
| *** salv-orlando has joined #openstack-kolla | 14:03 | |
| *** goldyfruit has joined #openstack-kolla | 14:07 | |
| *** salv-orlando has quit IRC | 14:08 | |
| *** goldyfruit has quit IRC | 14:09 | |
| *** emccormick has joined #openstack-kolla | 14:10 | |
| *** skramaja has quit IRC | 14:18 | |
| *** eanylin has joined #openstack-kolla | 14:18 | |
| *** stefan_bo has joined #openstack-kolla | 14:20 | |
| *** stefan_bo has quit IRC | 14:25 | |
| sdake | Jeffrey4l what do you make of this http://logs.openstack.org/31/464031/34/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/d9d8a58/console.html#_2017-05-21_04_52_44_915497 | 14:28 |
| Jeffrey4l | sdake, seems like a ansible bug https://github.com/ansible/ansible/issues/12282 | 14:32 |
| *** david-lyle has quit IRC | 14:34 | |
| *** david-lyle has joined #openstack-kolla | 14:35 | |
| rhallisey | any cores mind looking at: https://review.openstack.org/#/c/465145/ | 14:35 |
| *** ddyer has quit IRC | 14:36 | |
| sdake | sbezverk is there a way to print all secrets in kubernetes? | 14:37 |
| sdake | sbezverk (the contents of the secrets) | 14:37 |
| sbezverk | sdake: but then they would not be secrets any more ;) | 14:38 |
| sdake | sbezverk i could swear i found a website on google that explained how to do so | 14:38 |
| sbezverk | you just need to do reverse of base64 | 14:39 |
| sdake | rhallisey enjoy | 14:39 |
| sdake | sbezverk in fact this code had base64 in it | 14:39 |
| sdake | sbezverk thanks for jogging my memory | 14:39 |
| rhallisey | sdake, cheers | 14:39 |
| sbezverk | sdake: again 1 liner in bash ;) | 14:40 |
| sdake | sbezverk i am more interested in password rotation | 14:40 |
| sdake | which is not a one liner in any language :) | 14:40 |
| *** ddyer has joined #openstack-kolla | 14:40 | |
| sbezverk | sdake: kfox1111 has programmed token rotation for fernet, check his code.. | 14:41 |
| sdake | sbezverk yup i know fernet rotates and i have had a brief look at his code | 14:41 |
| sdake | sbezverk in the case of full password rotation, all the configmaps must be modified | 14:41 |
| sdake | sbezverk any way work for the future :) | 14:42 |
| sdake | one of my new servers is flakey | 14:42 |
| * sdake growls | 14:43 | |
| *** korzen has joined #openstack-kolla | 14:43 | |
| sdake | sbezverk do you use uefi in your servers? | 14:44 |
| sbezverk | sdake: nope, do not see any reason | 14:44 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs https://review.openstack.org/463476 | 14:44 |
| sbezverk | just because it is there is not good enough reason for me.. | 14:45 |
| *** schwicht has quit IRC | 14:48 | |
| sdake | sbezverk fair - just curious :) thanks | 14:48 |
| sdake | uefi is pretty nice, especially secure boot (when it works) | 14:49 |
| sdake | i find in general secure boot gets in the way of good dev, although if I was going to prod, I'd use secure boot | 14:49 |
| *** david-lyle has quit IRC | 14:52 | |
| *** iniazi_ has joined #openstack-kolla | 14:52 | |
| *** salv-orlando has joined #openstack-kolla | 14:52 | |
| *** powerd has joined #openstack-kolla | 14:54 | |
| *** iniazi has quit IRC | 14:55 | |
| *** lucasxu has joined #openstack-kolla | 14:56 | |
| *** eaguilar has joined #openstack-kolla | 14:56 | |
| *** salv-orlando has quit IRC | 14:57 | |
| inc0 | good morning | 15:02 |
| inc0 | Jeffrey4l: arround? | 15:02 |
| kbaegis | inc0: Morning! | 15:03 |
| Jeffrey4l | inc0, morning | 15:03 |
| inc0 | hey, I won't be able to make it for Wed meeting probably | 15:03 |
| inc0 | would you be kind enough to run it? | 15:03 |
| Jeffrey4l | ok. | 15:03 |
| inc0 | thank you | 15:03 |
| Jeffrey4l | my pleasure ;) | 15:03 |
| inc0 | ok, off to gates I go, I'm furious because I'm really close to get it working, but still small issues appear here and there | 15:06 |
| *** powerd has quit IRC | 15:09 | |
| *** bmaltais has joined #openstack-kolla | 15:13 | |
| bmaltais | Good morning. I have a question regarding kolla-ansible | 15:13 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 15:14 |
| bmaltais | I built a 2 node test cluster and I am trying to figure out how to do a "shutdown/restart" | 15:14 |
| inc0 | bmaltais: of everything? | 15:14 |
| bmaltais | We have yearly power shutdown and I try to simulate what would happen when we do that | 15:14 |
| bmaltais | I found kolla-ansible stop -i multinode | 15:15 |
| inc0 | we were working on it, not sure what's state od it | 15:15 |
| bmaltais | But I can't find how to issue a "start" equivalent to cleanly start things up | 15:15 |
| inc0 | deploy will start things up | 15:15 |
| inc0 | try rerunning depliy | 15:15 |
| inc0 | deploy | 15:15 |
| bmaltais | OK. So just do a deploy on the whole multinode cluster to bring it back up. | 15:16 |
| inc0 | that's the plan | 15:16 |
| bmaltais | I will give that a try. Thanks inc0: | 15:16 |
| inc0 | one thing tho | 15:16 |
| inc0 | mariadb is notorious in terms of full shutdown | 15:16 |
| inc0 | so you might need to get this one up manually | 15:16 |
| bmaltais | inc0: This is what I see if I manually docker start processes on the nodes | 15:17 |
| bmaltais | mariadb get in a restart loop | 15:17 |
| inc0 | http://galeracluster.com/documentation-webpages/restartingcluster.html | 15:17 |
| bmaltais | I hope deploy will fix that | 15:17 |
| inc0 | it won't | 15:17 |
| inc0 | it's safety built into mariadb itself | 15:17 |
| inc0 | or rather, galera | 15:17 |
| inc0 | since galera is multi master | 15:18 |
| inc0 | one of nodes will have latest state | 15:18 |
| inc0 | issue is, we don't know which one | 15:18 |
| inc0 | and if you start wrong one, that may end with data corruption | 15:18 |
| inc0 | so you need to follow procedure I linked to bring up mariadb manually | 15:18 |
| bmaltais | I see... so this is a fairly significant issue | 15:19 |
| inc0 | in reality it's just chaning safe_to_bootstrap from 0 to 1 on single node | 15:19 |
| bmaltais | I will try that | 15:19 |
| inc0 | finding correct node is more problematic | 15:19 |
| inc0 | bmaltais: if you can gracefully shutdown cluster it will make your life easier | 15:19 |
| bmaltais | inc0: By gracefully you mean using the kolla-ansible stop command or something else? | 15:20 |
| inc0 | on top of that one issue, just starting up docker containers should work | 15:20 |
| inc0 | no, manually put down one mariadb container at the time | 15:20 |
| inc0 | I don't think our stop command is that clever | 15:20 |
| inc0 | would be nice addition tho;) | 15:21 |
| matrohon | egonzalez, you told me that you were testing upgrades to 4.0.3, but the current tag for ocata is 4.0.2, am I wrong? | 15:21 |
| inc0 | bmaltais: you can find files used in this guide I linked in /var/lib/docker/volumes/mariadb/_data | 15:21 |
| bmaltais | inc0: OK... will try that to see how much work it is | 15:21 |
| SamYaple | inc0: you have to change safe_to_bootstrap AND update the gcomm:// list to empty | 15:21 |
| *** eaguilar has quit IRC | 15:22 | |
| SamYaple | safe_to_bootstrap is a safety feature added on top of the old way | 15:22 |
| inc0 | yeah, what he said:) | 15:22 |
| egonzalez | matrohon, yep, images built for gates are 4.0.3, are the ones I used | 15:22 |
| inc0 | SamYaple: what's up with galera doing this automatically? | 15:22 |
| inc0 | you mentioned it one day | 15:22 |
| SamYaple | inc0: it can do it automatically in a number of situations, but you have to know those situations | 15:23 |
| SamYaple | the logic can be baked into startup scripts if those startup scripts can query the other nodes | 15:23 |
| SamYaple | if they are stand alone the logic can be backed into a 100% idempotent ansible playbook | 15:23 |
| SamYaple | there are only 2 sceanrios where galera doesnt know how to recover | 15:23 |
| SamYaple | and those involve real data loss | 15:24 |
| *** vhosakot has joined #openstack-kolla | 15:24 | |
| matrohon | egonzalez, I'm suprised, since the 4.0.2 has not even been released | 15:25 |
| *** jmccarthy has left #openstack-kolla | 15:25 | |
| inc0 | matrohon: it hasn't been pushed to dockerhub | 15:26 |
| inc0 | dockerhub is not really our release;) | 15:26 |
| inc0 | we're working on resolving this now | 15:26 |
| matrohon | inc0, but there is no 4.0.2 tag on git | 15:26 |
| egonzalez | matrohon, sorry was 4.0.2, 4.0.1 is released but not pushed to DH | 15:26 |
| matrohon | egonzalez, ack | 15:26 |
| inc0 | ahh, that's true, well if you build today it will be 4.0.2 | 15:26 |
| egonzalez | 4.0.2 is tagged internally by pbr | 15:26 |
| inc0 | until we tag 4.0.2 in git then it's 4.0.3 | 15:27 |
| egonzalez | release + 1 | 15:27 |
| matrohon | ok thanks | 15:27 |
| egonzalez | inc0, may be the db stuff related with https://bugs.launchpad.net/kolla-ansible/+bug/1692507 | 15:27 |
| openstack | Launchpad bug 1692507 in kolla-ansible "mariadb fail to upgrade ocata->master" [Undecided,New] | 15:27 |
| *** eaguilar has joined #openstack-kolla | 15:27 | |
| bmaltais | inc0: Given containers... how do you gracefully shutdown a container running mariadb? I guess docker stop <mariadb id> is not it... | 15:28 |
| egonzalez | just tested upgrade from ocata -=> master and mariadb keeps restarting | 15:28 |
| inc0 | bmaltais: it does...to a point | 15:28 |
| egonzalez | newton->ocata works fine | 15:28 |
| inc0 | it will timeout after 10s (I think that's the default) | 15:28 |
| inc0 | and then force kill | 15:28 |
| *** ducttape_ has quit IRC | 15:29 | |
| *** ducttape_ has joined #openstack-kolla | 15:30 | |
| inc0 | docker stop -t 30 mariadb will work better | 15:30 |
| inc0 | or whatever t you need | 15:30 |
| SamYaple | inc0: i dont think kolla has mariadb setup to cleanly exist, like ever | 15:30 |
| SamYaple | you have to issue the shutdown a different way than just TERM | 15:30 |
| SamYaple | 10 seconds, 30 seconds isnt going to matter much | 15:31 |
| SamYaple | it shutsdown cleanly in about 2 seconds on my active cluster when issued with mysqladmin | 15:31 |
| inc0 | well I think dump init will relay sigkill to mariadb | 15:31 |
| SamYaple | sigkill will just kill it | 15:31 |
| *** myoung_ is now known as myoung | 15:31 | |
| inc0 | ahh, yay | 15:31 |
| SamYaple | it wont respect term though | 15:31 |
| SamYaple | well i think it *does*, but the proper way to do it is with mysqladmin or the like | 15:32 |
| inc0 | so docker exec -it mariadb mysqladmin stop? | 15:32 |
| SamYaple | and have the process kill itself | 15:32 |
| SamYaple | shutdown, but thats the jist yea | 15:32 |
| inc0 | thanks SamYaple | 15:32 |
| inc0 | that's good knowledge | 15:32 |
| SamYaple | i enable socket based auth so it doesnt need a password either | 15:32 |
| inc0 | brb | 15:33 |
| vhosakot | bmaltais: is there a mariadb/mysql command to graceflly shutdown mariedb if mariadb wasn't running in a container? | 15:33 |
| *** powerd has joined #openstack-kolla | 15:34 | |
| *** lpetrut has quit IRC | 15:35 | |
| *** salv-orlando has joined #openstack-kolla | 15:36 | |
| vhosakot | masuberu: here are the steps to remove the ceph_mon_config container if ceph's volume or config.json is corrupted --> https://github.com/openstack/kolla-ansible/blob/master/doc/ceph-guide.rst#deploy-fails-with-fetching-ceph-keyrings--no-json-object-could-be-decoded | 15:38 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs https://review.openstack.org/463476 | 15:40 |
| vhosakot | masuberu: I'd check if this task passed and if ceph-mon.config.json exists on your ceph-mon node --> https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/ceph/tasks/config.yml#L12-L19 | 15:41 |
| *** srwilkers has quit IRC | 15:44 | |
| vhosakot | sdake: I just use deprecation_warnings=False in my ansible.cfg is deprecation warning cause blockers :) | 15:46 |
| bmaltais | inc0: Running "docker exec -it mariadb mysqladmin stop" produce error: 'Access denied for user 'mysql'@'localhost' (using password: NO)' | 15:47 |
| vhosakot | sdake: ^^^ about the warning in http://logs.openstack.org/31/464031/34/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/d9d8a58/console.html#_2017-05-21_04_52_44_915497. | 15:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/465258 | 15:48 |
| sdake | vhosakot thanks | 15:48 |
| vhosakot | sdake: np | 15:48 |
| bmaltais | inc0: Is there a default password for mariadb? | 15:48 |
| vhosakot | bmaltais: is it database_password in etc/kolla/passwords.yml on yout deploy node --> https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/tasks/register.yml#L9 | 15:51 |
| *** chenn has quit IRC | 15:52 | |
| *** manheim_ has quit IRC | 15:57 | |
| *** egonzalez has quit IRC | 15:58 | |
| *** pcaruana has quit IRC | 15:59 | |
| sdake | bmaltais the mariadb password is in the passwords.yml file | 15:59 |
| *** goldyfruit has joined #openstack-kolla | 16:01 | |
| *** AzazeL_ has joined #openstack-kolla | 16:01 | |
| vhosakot | kbaegis: the task https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L29-L30 assumes the target node has systemd/systemctl. If not, it is expected to fail if the command "systemctl" does not exist :) | 16:01 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Fixed all Warning and Errors in existing docs https://review.openstack.org/463476 | 16:02 |
| *** david-lyle has joined #openstack-kolla | 16:03 | |
| bmaltais | sdake: That worked... but running "mysqladmin -u root -psomepassword shutdown" work but the alwaysrestart on the container start it back up ;-) | 16:03 |
| *** lucasxu has quit IRC | 16:04 | |
| vhosakot | bmaltais: did you get somepassword from database_password in etc/kolla/passwords.yml on your deploy node? | 16:04 |
| bmaltais | So perhaps running "docker exec -it mariadb mysqladmin -u root -psomepass shutdown ; docker stop mariadb" would work... and it look like it does. | 16:05 |
| *** matrohon has quit IRC | 16:06 | |
| bmaltais | vhosakot: Yes, that was the one | 16:06 |
| vhosakot | bmaltais: ah cool | 16:06 |
| bmaltais | Now that both mariadb container are "cleanly" showdown I will attempt a restart... starting with the last node that ran mariadb and see if things will come back clean | 16:08 |
| *** chas has quit IRC | 16:09 | |
| vhosakot | bmaltais: are you using sysvinit instead of systemd in Ubuntu? | 16:11 |
| bmaltais | No, running on centos7 | 16:11 |
| vhosakot | bmaltais: ah ok | 16:11 |
| bmaltais | Unfortunatly evendoing a shutdown mariadb is still restarting ;-( | 16:12 |
| bmaltais | So clearly need more work than this ;-) | 16:12 |
| kbaegis | vhosakot: For those of us that are supremely unhappy with systemd, it would be nice if it could at least fallback to a behavior with sysvinit that doesn't completely break precheck | 16:14 |
| kbaegis | vhosakot: The documentation purports to support non-systemd setups (upstart/sysv) | 16:14 |
| vhosakot | kbaegis: I see this is the only task that needs systemctl which fails when systemctl is not available. It is a precheck task, if systemd does not exist, that task's failure could be ignored and the opertor could check docker's hralth manually using the eqvuivalent sysvinit command :) --> https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/precheck.yml#L29-L30 | 16:16 |
| *** lucasxu has joined #openstack-kolla | 16:18 | |
| vhosakot | kbaegis: agreed, doc has stuff about sysvinit but the roles/tasks assume systemctl. | 16:19 |
| *** tvignaud is now known as tvignaud_afk | 16:19 | |
| vhosakot | I remember seeing a systemd vs sysvinit cheatsheet to convert one to another for health checks and restarts. | 16:20 |
| *** korzen has quit IRC | 16:20 | |
| *** ntpttr__ is now known as ntpttr | 16:21 | |
| *** srwilkers has joined #openstack-kolla | 16:21 | |
| *** goldyfruit has quit IRC | 16:23 | |
| *** powerd has quit IRC | 16:23 | |
| kbaegis | vhosakot: Question: if the behavior can be gotten from a lower level function (ala docker), wouldn't you want it to check that rather than doing it through systemd? | 16:27 |
| kbaegis | I actually don't know if docker-api or docker-cli expose the necessary information, but if it does it would be nice | 16:29 |
| *** korzen has joined #openstack-kolla | 16:29 | |
| kbaegis | At a more basic level, I don't understand what MountFlags=shared does, nor does anyone on #docker apparently | 16:31 |
| kbaegis | Or maybe my ignorance is simply contagious and I'm asking the wrong questions :) | 16:33 |
| kfox1111 | morning. | 16:34 |
| vhosakot | kbaegis: right, don't know what gives the required info about MountFlags other than "systemctl show docker". I'll check if there is a docker CLI to provide the info about MountFlags. It is a task to check if MountFlags is set to shared on certain neutron containers. | 16:34 |
| vhosakot | morning kfox1111 | 16:34 |
| sbezverk | kfox1111: good morning | 16:34 |
| kfox1111 | sbezverk: yeah. working on it. | 16:35 |
| *** eaguilar has quit IRC | 16:35 | |
| vhosakot | kbaegis: will keep you posted. | 16:35 |
| kfox1111 | am dealing with a failing laptop today. :/ | 16:35 |
| kbaegis | vhosakot: Thank you! | 16:35 |
| sbezverk | kfox1111: got bored on sunday and clean up a bit more v4 gate jobs, hopefully soon we will see more green than red.. | 16:35 |
| vhosakot | np | 16:35 |
| kfox1111 | sbezverk: cool. :) | 16:35 |
| kfox1111 | shoot... another fire. :/ | 16:36 |
| *** korzen has quit IRC | 16:36 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: v4 gate. https://review.openstack.org/454841 | 16:39 |
| *** salv-orlando has quit IRC | 16:40 | |
| *** jaosorior is now known as jaosorior_away | 16:45 | |
| *** pbourke has quit IRC | 16:46 | |
| rwellum | sbezverk: starting today I start to see this when deploying - this is on a happily running cluster with canal installed https://www.irccloud.com/pastebin/qeapLkXr/ | 16:47 |
| *** unicell has quit IRC | 16:52 | |
| *** Serlex has quit IRC | 16:53 | |
| inc0 | back, sorry | 16:56 |
| inc0 | look at number of gates here:) | 16:57 |
| inc0 | https://review.openstack.org/#/c/466007/55 | 16:58 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 16:59 |
| *** salv-orlando has joined #openstack-kolla | 17:01 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 17:04 |
| *** salv-orlando has quit IRC | 17:05 | |
| bmaltais | OK, I think I figured out how to "properly?" shutdown a kolla-ansible cluster and then start it back up properly | 17:10 |
| bmaltais | Not automated right now and involve multiple manual steps but it worked | 17:11 |
| inc0 | nice bmaltais | 17:11 |
| vhosakot | bmaltais: cool, can you submit a ops_best_practices.rst please ;) | 17:11 |
| inc0 | anything besides mariadb was manual? | 17:11 |
| vhosakot | ^^^ | 17:12 |
| inc0 | yeah, vhosakot is right, would you mind writing short doc about it? | 17:12 |
| bmaltais | Essentially onew need to do "docker exec -it mariadb mysqladmin -u root -pckZoS17Pkg4LfMZJQIhKoJvML4P6KnOhm4lk3znW shutdown ; docker stop mariadb" on each controller nodes | 17:12 |
| inc0 | I'll be happy to guide you through process | 17:12 |
| bmaltais | Then do a "kolla-ansible stop -i multinode" | 17:12 |
| bmaltais | vhosakot: Sure. Not sure if it is "best" practice but it did work ;-) | 17:13 |
| inc0 | bmaltais: so what we wanted to add after summit | 17:13 |
| bmaltais | NExt you start back the nodes and do a "docker stop mariadb" on each controller nodes | 17:13 |
| inc0 | is set of quick howtos for different things | 17:13 |
| inc0 | stopping cluster would be great one to kick this off | 17:14 |
| vhosakot | bmaltais: ah, you used "kolla-ansible stop" as well.. we have https://github.com/openstack/kolla-ansible/blob/master/tools/kolla-ansible#L50 as well and it worked when I lasted tested it is Mitaka. | 17:14 |
| bmaltais | Last you do a "kolla-ansible mariadb_recovery -i multinode" | 17:14 |
| bmaltais | And done | 17:14 |
| *** lucasxu has quit IRC | 17:14 | |
| inc0 | vhosakot: mariadb needs to be manually gracefully shitdown | 17:14 |
| inc0 | if you want to avoid problems later | 17:14 |
| vhosakot | inc0: shitdown lol.. ah ok | 17:14 |
| inc0 | there you go | 17:15 |
| inc0 | unless you shit this down | 17:15 |
| inc0 | it's going to shit you up | 17:15 |
| bmaltais | inc0: That's if you forget the gracefull shutdown step ;-) | 17:15 |
| vhosakot | bmaltais: glad it is working... would be awesome if you please doc it with steps how to stop/shutdown and turn back on a kolla cluster. | 17:15 |
| bmaltais | vhosakot: WHere is the ops_best_practices.rst file? Don't see it in doc | 17:17 |
| inc0 | bmaltais: let's create it | 17:17 |
| inc0 | you'll going to be first one:) | 17:18 |
| *** unicell has joined #openstack-kolla | 17:18 | |
| bmaltais | OK. I will try to add it. I am new to gerrit/openstack documentation. | 17:18 |
| *** jrich523 has quit IRC | 17:19 | |
| inc0 | so, just create this file and add link to it here: https://github.com/openstack/kolla-ansible/blob/master/doc/index.rst#overview | 17:19 |
| inc0 | github doesn't render rst too well | 17:19 |
| inc0 | bmaltais: do you need help with gerrit itself? | 17:20 |
| vhosakot | bmaltais: it doesn't exist yet :) | 17:20 |
| vhosakot | bmaltais: I can help you add one :) I'm planning to add one to for both single-node and multi-node. | 17:20 |
| inc0 | vhosakot: forger single node imho | 17:21 |
| inc0 | let's focus on all things multinode | 17:21 |
| inc0 | as it's really not different and much more useful | 17:21 |
| vhosakot | right, ops don't use singlenode anyway, good point | 17:21 |
| inc0 | also graceful shutdown of cluster while talking about single node is...well... | 17:21 |
| inc0 | they call it cluster for a reason;) | 17:22 |
| vhosakot | right ;) | 17:22 |
| vhosakot | brb 2 mins, rebooting. | 17:22 |
| bmaltais | I used https://review.openstack.org/gitweb?p=openstack/kolla-ansible.git;a=tree;f=doc;h=ad2f262c1b240b43d743e3080d01af8c120c41b8;hb=HEAD in the past | 17:23 |
| bmaltais | But can't see how to add fiolesvia this interface | 17:23 |
| inc0 | bmaltais: let me link you guide | 17:24 |
| inc0 | I suggest reading this one https://docs.openstack.org/infra/manual/developers.html | 17:24 |
| *** vhosakot_ has joined #openstack-kolla | 17:25 | |
| bmaltais | OK | 17:25 |
| inc0 | don't hesitate to ask if you have any questions | 17:25 |
| bmaltais | It is sort of frustrating when the process is in the way of creativity ;-) | 17:25 |
| inc0 | it's not that bad | 17:25 |
| bmaltais | But I understand the need for proper change control and tracking | 17:26 |
| inc0 | and some process is required if you have north of 200 contributors;) | 17:26 |
| bmaltais | THe hard part is the 1st setup. | 17:26 |
| inc0 | yeah gerrit has its quirks | 17:26 |
| *** vhosakot has quit IRC | 17:27 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 17:31 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 17:31 |
| vhosakot_ | back | 17:33 |
| *** rpittau_ has joined #openstack-kolla | 17:42 | |
| *** kbaegis has quit IRC | 17:46 | |
| *** tonanhngo has joined #openstack-kolla | 17:48 | |
| *** rpittau_ has quit IRC | 17:49 | |
| *** kbaegis has joined #openstack-kolla | 17:49 | |
| *** sambetts is now known as sambetts|afk | 17:59 | |
| inc0 | guys we have issue at hand, kolla gates seems to be broken | 18:02 |
| kbaegis | Have you tried wiggling the latch? | 18:03 |
| kbaegis | :) | 18:03 |
| inc0 | yeah, I also tried surgical kick | 18:03 |
| inc0 | it seems like aodh became uninstallable | 18:04 |
| kbaegis | Well, where there's smoke there's Aodh | 18:06 |
| *** jemcevoy has joined #openstack-kolla | 18:08 | |
| *** chas has joined #openstack-kolla | 18:09 | |
| vhosakot_ | inc0: is it always the first Dockerfile? have you changed it to zaodh> ;) | 18:10 |
| jemcevoy | Are there any tools for pruning the elasticsearch indices Ours have grown to 36GB and are on the root partition. | 18:10 |
| *** lucasxu has joined #openstack-kolla | 18:12 | |
| *** chas has quit IRC | 18:14 | |
| *** itlinux has joined #openstack-kolla | 18:21 | |
| bmaltais | inc0: vhosakot: Thank's, figured out gerrit and now working on my 1st commit ;-) | 18:24 |
| vhosakot_ | bmaltais: nice +1 :) | 18:24 |
| inc0 | bmaltais: thank you! | 18:24 |
| bmaltais | Should have something ready soon for validation | 18:24 |
| sbezverk | sbezverk: why do not you check actual labels/taints on the node you try to remove it from? | 18:25 |
| vhosakot_ | jemcevoy: good to have a separate parition for logs as node might be unusable if /root is full. | 18:25 |
| sbezverk | rwellum: ^^^ | 18:25 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 18:29 |
| *** chas has joined #openstack-kolla | 18:31 | |
| jemcevoy | vhosakot_ Well it is a little late... Already in production... And I gave the majority of the boot drives to the LVM thin pool 'cuz I thought all the docker runtime stuff would go to the thin pool... MISTAKE... | 18:33 |
| vhosakot_ | jemcevoy: yeah, docker's runtime increases usage in /var/lib/docker in root partition I've seen. | 18:34 |
| vhosakot_ | especially the logging container, which is supposed to grow its volume in size fast due to all the logging happening :) | 18:35 |
| *** chas has quit IRC | 18:35 | |
| vhosakot_ | in staging, we just truncate logs once the logging volume reaches a threshold disk space. | 18:37 |
| *** harlowja has joined #openstack-kolla | 18:37 | |
| jemcevoy | vhosakot_ : Could I move the /var/lib/docker/volumes/kolla_logs/ and elasticsearch into a large ceph volumes on each controller? | 18:38 |
| *** masuberu has quit IRC | 18:39 | |
| rwellum | Which comment sbezverk ? the separate partition of the label/taint? | 18:39 |
| vhosakot_ | jemcevoy: hmm, haven't tried it... the docker volumes must reside locally on the docker host I think, instead of attaching to a remove volume like ceph over the network. | 18:40 |
| sbezverk | rwellum: label/taint | 18:40 |
| *** dave-mccowan has quit IRC | 18:41 | |
| *** lucasxu has quit IRC | 18:41 | |
| jemcevoy | I'll need to do some lab experiments next week... :-) vhosakot_ | 18:41 |
| *** itlinux has quit IRC | 18:41 | |
| vhosakot_ | +1 ;) | 18:42 |
| vhosakot_ | never heard of remote docker volume over the wire tbh | 18:42 |
| *** itlinux has joined #openstack-kolla | 18:42 | |
| *** ArminderSingh has joined #openstack-kolla | 18:43 | |
| *** lucasxu has joined #openstack-kolla | 18:43 | |
| *** harlowja has quit IRC | 18:43 | |
| vhosakot_ | jemcevoy: looks like someone did it using NFS server+client --> http://container42.com/2014/03/29/docker-quicktip-4-remote-volumes/ | 18:44 |
| *** zhuzeyu has quit IRC | 18:48 | |
| *** itlinux has quit IRC | 18:48 | |
| *** zhuzeyu has joined #openstack-kolla | 18:48 | |
| *** manheim_ has joined #openstack-kolla | 18:51 | |
| *** manheim_ has quit IRC | 18:55 | |
| *** Mannheim has joined #openstack-kolla | 18:57 | |
| *** lucasxu has quit IRC | 18:58 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment. https://review.openstack.org/466876 | 18:59 |
| *** powerd has joined #openstack-kolla | 19:02 | |
| *** zhubingbing has quit IRC | 19:02 | |
| *** zhubingbing has joined #openstack-kolla | 19:02 | |
| *** lucasxu has joined #openstack-kolla | 19:03 | |
| *** feisky has quit IRC | 19:07 | |
| *** blallau has joined #openstack-kolla | 19:09 | |
| *** goldyfruit has joined #openstack-kolla | 19:11 | |
| *** goldyfruit has quit IRC | 19:11 | |
| *** chas has joined #openstack-kolla | 19:12 | |
| *** iniazi has joined #openstack-kolla | 19:13 | |
| *** iniazi_ has quit IRC | 19:16 | |
| *** chas has quit IRC | 19:17 | |
| *** salv-orlando has joined #openstack-kolla | 19:18 | |
| *** salv-orlando has quit IRC | 19:28 | |
| *** chas has joined #openstack-kolla | 19:33 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 19:38 |
| *** chas has quit IRC | 19:38 | |
| *** egonzalez has joined #openstack-kolla | 19:44 | |
| *** spsurya has quit IRC | 19:44 | |
| *** eaguilar has joined #openstack-kolla | 19:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements https://review.openstack.org/453309 | 19:46 |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Added pointer to new file in index.rst https://review.openstack.org/466881 | 19:50 |
| *** chas has joined #openstack-kolla | 19:54 | |
| vhosakot_ | inc0: I see you removed tools/deploy_aio.sh in https://review.openstack.org/#/c/466007/. won't it break all the gates? :) is the idea to remove tools/deploy_aio.sh and use the new workflow in gate you have in https://review.openstack.org/#/c/466007/ ? | 19:55 |
| vhosakot_ | inc0: instead of removing tools/deploy_aio.sh, why not add something like deploy_multinode.sh? | 19:56 |
| *** salv-orlando has joined #openstack-kolla | 19:56 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Fixing small typos. https://review.openstack.org/466881 | 19:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla stable/ocata: Updated from global requirements https://review.openstack.org/464807 | 19:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible stable/ocata: Updated from global requirements https://review.openstack.org/452870 | 19:57 |
| *** chas has quit IRC | 19:59 | |
| egonzalez | please, review to fix source builds https://review.openstack.org/#/c/466639/ | 20:08 |
| *** powerd has quit IRC | 20:08 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Fixing small typos. https://review.openstack.org/466881 | 20:10 |
| bmaltais | I think I made a faux pas in gerrit as this is my 1st experience... | 20:11 |
| bmaltais | I commited a change over a change instead of doing a --amend | 20:12 |
| bmaltais | Now some are asking why I am not making patch on original commit ;-( | 20:12 |
| bmaltais | How do I fix this error? | 20:12 |
| *** chas has joined #openstack-kolla | 20:15 | |
| egonzalez | bmaltais what I use to do when screw up git things is remove local branch and fetch gerrit change, then work on it again with commit --amend , i do not complicate too much with git commands unless i have so much work on the wrong change | 20:17 |
| *** chas has quit IRC | 20:19 | |
| *** stefan_bo has joined #openstack-kolla | 20:22 | |
| *** stefan_bo has quit IRC | 20:27 | |
| vhosakot_ | bmaltais: yes, without --amend, "git review" will generate multiple reviews in gerrit, one review for each commit | 20:29 |
| vhosakot_ | bmaltais: there is way to undo the commits and move back to HEAD using "git reset" | 20:30 |
| *** chas has joined #openstack-kolla | 20:36 | |
| *** ducttape_ has quit IRC | 20:39 | |
| *** salv-orlando has quit IRC | 20:39 | |
| *** ducttape_ has joined #openstack-kolla | 20:40 | |
| rwellum | sorry sbezverk caught up in long meeting. I'm trying to untaint a new cluster. Default behaviour for an AIO. | 20:40 |
| *** salv-orlando has joined #openstack-kolla | 20:40 | |
| *** chas has quit IRC | 20:41 | |
| *** blallau has quit IRC | 20:42 | |
| *** Pavo has quit IRC | 20:43 | |
| *** salv-orlando has quit IRC | 20:44 | |
| *** ddyer has quit IRC | 20:49 | |
| *** eaguilar has quit IRC | 20:51 | |
| inc0 | vhosakot_: idea is that multinode and aio aren't that different | 20:53 |
| inc0 | I'm collapsing everything to single, ansible-driven, workflow | 20:53 |
| *** ddyer has joined #openstack-kolla | 20:53 | |
| inc0 | and I got it working already, issue I'm struggling right now is managing failures and downloading logs from everywhere | 20:53 |
| *** eaguilar has joined #openstack-kolla | 20:54 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment. https://review.openstack.org/466881 | 20:55 |
| vhosakot_ | inc0: cool, so will deploy_aoi.sh be removed? | 20:55 |
| vhosakot_ | aio* | 20:55 |
| inc0 | vhosakot_: there is no reason to keep it | 20:56 |
| vhosakot_ | inc0: well, gate uses it daily :) so, the gate will use the new apporach using your ansible playbooks to deploy in gate instead of deploy_aio.sh? | 20:56 |
| inc0 | yup | 20:57 |
| *** chas has joined #openstack-kolla | 20:57 | |
| vhosakot_ | inc0: cool, got it, thanks! | 20:57 |
| inc0 | lots of that stuff, like deploy_aio or setup_node are leftovers from early Kolla | 20:57 |
| inc0 | now we have things like boostrap-servers that can take care of most of it | 20:58 |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment. https://review.openstack.org/466881 | 20:58 |
| inc0 | bmaltais: soo, one thing about gerrit:) | 20:58 |
| inc0 | one commit == one change in gerrit | 20:59 |
| inc0 | so for example https://review.openstack.org/#/c/466876/ this is one commit and https://review.openstack.org/#/c/466881/ this is another set | 21:00 |
| inc0 | look at Change-Id, it's a bit messed up;) | 21:00 |
| *** ddyer has quit IRC | 21:00 | |
| inc0 | what you want to do is fix the feedback from review | 21:00 |
| inc0 | and do git commit --amend | 21:00 |
| inc0 | --amend flag means "edit last commit instead of creating new one" | 21:00 |
| *** chas has quit IRC | 21:02 | |
| *** ddyer has joined #openstack-kolla | 21:03 | |
| *** bmace__ has quit IRC | 21:04 | |
| *** bmace__ has joined #openstack-kolla | 21:04 | |
| *** dave-mccowan has joined #openstack-kolla | 21:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements https://review.openstack.org/453309 | 21:06 |
| egonzalez | inc0 can you review this change which fix source build gates https://review.openstack.org/#/c/466639/ | 21:08 |
| *** ducttape_ has quit IRC | 21:08 | |
| inc0 | done, I was looking at it myself | 21:09 |
| *** ducttape_ has joined #openstack-kolla | 21:09 | |
| egonzalez | thanks | 21:09 |
| *** dasTor_ has joined #openstack-kolla | 21:10 | |
| *** dasTor has quit IRC | 21:11 | |
| *** chas has joined #openstack-kolla | 21:18 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 21:22 |
| *** chas has quit IRC | 21:22 | |
| *** rhallisey has quit IRC | 21:22 | |
| *** fooliouno has joined #openstack-kolla | 21:24 | |
| *** jheroux has quit IRC | 21:30 | |
| *** ddyer has quit IRC | 21:30 | |
| *** ddyer has joined #openstack-kolla | 21:34 | |
| *** ddyer has quit IRC | 21:34 | |
| *** Pavo has joined #openstack-kolla | 21:36 | |
| *** ddyer has joined #openstack-kolla | 21:38 | |
| *** chas has joined #openstack-kolla | 21:39 | |
| *** chas has quit IRC | 21:43 | |
| *** eaguilar has quit IRC | 21:56 | |
| *** lucasxu has quit IRC | 21:58 | |
| *** chas has joined #openstack-kolla | 21:59 | |
| openstackgerrit | Merged openstack/kolla master: Fix aodh-base image https://review.openstack.org/466639 | 22:02 |
| vhosakot_ | inc0: great work, deploy_aio.sh is used for kolla gate as well, not just for kolla-ansible gate --> http://logs.openstack.org/39/466639/1/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/b363a9e/console.html#_2017-05-22_20_24_16_434607 | 22:04 |
| vhosakot_ | inc0: so, the kolla gate must also use the new approach in your review instead of deploy_aio.sh, right? | 22:04 |
| *** chas has quit IRC | 22:04 | |
| *** egonzalez has quit IRC | 22:06 | |
| *** Mannheim has quit IRC | 22:12 | |
| *** ducttape_ has quit IRC | 22:12 | |
| *** ducttape_ has joined #openstack-kolla | 22:12 | |
| *** itlinux has joined #openstack-kolla | 22:13 | |
| inc0 | vhosakot_: I think kolla gate should work, it uses kolla-ansible's gate code | 22:16 |
| vhosakot_ | inc0: ah cool, good both kolla and kolla-ansible use the new approach with ansible playbooks, thanks for the info. | 22:16 |
| inc0 | that's the plan:) | 22:17 |
| vhosakot_ | inc0: plan is all that is needed ;) | 22:17 |
| *** manheim_ has joined #openstack-kolla | 22:20 | |
| *** chas has joined #openstack-kolla | 22:20 | |
| *** Pavo has quit IRC | 22:20 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 22:21 |
| *** manheim__ has joined #openstack-kolla | 22:23 | |
| *** fooliouno has quit IRC | 22:25 | |
| *** chas has quit IRC | 22:25 | |
| *** manheim_ has quit IRC | 22:26 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instruction on shutting down and restarting complete kolla-ansible environment. https://review.openstack.org/466881 | 22:37 |
| *** manheim__ has quit IRC | 22:40 | |
| *** chas has joined #openstack-kolla | 22:41 | |
| *** itlinux has quit IRC | 22:42 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/464393 | 22:42 |
| *** chas has quit IRC | 22:46 | |
| *** celo_r has joined #openstack-kolla | 22:47 | |
| *** powerd has joined #openstack-kolla | 22:48 | |
| *** ducttape_ has quit IRC | 22:49 | |
| *** ducttape_ has joined #openstack-kolla | 22:50 | |
| *** ducttap__ has joined #openstack-kolla | 22:54 | |
| *** ducttape_ has quit IRC | 22:54 | |
| *** itlinux has joined #openstack-kolla | 23:01 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 23:02 |
| *** chas has joined #openstack-kolla | 23:02 | |
| *** itlinux has quit IRC | 23:05 | |
| *** chas has quit IRC | 23:06 | |
| *** ducttap__ has quit IRC | 23:08 | |
| *** powerd has quit IRC | 23:10 | |
| *** chas has joined #openstack-kolla | 23:23 | |
| *** chas has quit IRC | 23:27 | |
| *** stefan_bo has joined #openstack-kolla | 23:32 | |
| *** srwilkers has quit IRC | 23:34 | |
| *** rwsu has quit IRC | 23:34 | |
| *** rwsu has joined #openstack-kolla | 23:41 | |
| *** salv-orlando has joined #openstack-kolla | 23:41 | |
| *** spsurya has joined #openstack-kolla | 23:41 | |
| spsurya | morning | 23:42 |
| *** chas has joined #openstack-kolla | 23:44 | |
| *** chas has quit IRC | 23:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!