| *** jbadiapa has quit IRC | 00:07 | |
| *** igordc has quit IRC | 00:13 | |
| *** k_mouza has quit IRC | 00:16 | |
| *** k_mouza has joined #openstack-kolla | 00:17 | |
| *** k_mouza has quit IRC | 00:21 | |
| *** k_mouza has joined #openstack-kolla | 00:23 | |
| *** xinliang has joined #openstack-kolla | 00:28 | |
| *** schwicht has quit IRC | 00:35 | |
| *** schwicht has joined #openstack-kolla | 00:40 | |
| *** k_mouza has quit IRC | 00:44 | |
| *** k_mouza has joined #openstack-kolla | 00:58 | |
| *** zhanglong has joined #openstack-kolla | 01:02 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Template custom HAProxy service configuration https://review.opendev.org/698271 | 01:19 |
|---|---|---|
| *** schwicht has quit IRC | 01:21 | |
| *** goldyfruit_ has joined #openstack-kolla | 01:27 | |
| *** k_mouza has quit IRC | 01:28 | |
| *** zhanglong has quit IRC | 01:36 | |
| *** zhanglong has joined #openstack-kolla | 01:38 | |
| *** schwicht has joined #openstack-kolla | 01:49 | |
| *** dave-mccowan has joined #openstack-kolla | 02:00 | |
| *** goldyfruit_ has quit IRC | 02:15 | |
| *** goldyfruit_ has joined #openstack-kolla | 02:15 | |
| *** negronjl has quit IRC | 02:18 | |
| *** negronjl has joined #openstack-kolla | 02:21 | |
| *** goldyfruit_ has quit IRC | 02:40 | |
| *** ricolin has joined #openstack-kolla | 02:40 | |
| *** dave-mccowan has quit IRC | 02:55 | |
| *** zhanglong has quit IRC | 03:07 | |
| *** munimeha1 has quit IRC | 03:28 | |
| *** k_mouza has joined #openstack-kolla | 03:29 | |
| *** factor has joined #openstack-kolla | 03:31 | |
| *** k_mouza has quit IRC | 03:34 | |
| *** xinliang has quit IRC | 03:42 | |
| *** zhanglong has joined #openstack-kolla | 03:54 | |
| *** zhanglong has quit IRC | 04:46 | |
| *** skramaja has joined #openstack-kolla | 05:27 | |
| *** sri_ has joined #openstack-kolla | 06:02 | |
| sri_ | Morning | 06:03 |
| *** pcaruana has joined #openstack-kolla | 06:09 | |
| *** raukadah is now known as chkumar|ruck | 06:10 | |
| yoctozepto | morning | 06:20 |
| *** xinliang has joined #openstack-kolla | 06:28 | |
| openstackgerrit | Kevin Zhao proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 06:34 |
| openstackgerrit | Kevin Zhao proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 06:41 |
| osmanlicilegi | morning | 06:46 |
| *** dpawlik has joined #openstack-kolla | 07:00 | |
| *** dpawlik has quit IRC | 07:04 | |
| *** dpawlik has joined #openstack-kolla | 07:07 | |
| *** rgogunskiy has joined #openstack-kolla | 07:07 | |
| *** k_mouza has joined #openstack-kolla | 07:30 | |
| *** zainub_wahid has joined #openstack-kolla | 07:31 | |
| hrw | morning | 07:32 |
| *** k_mouza has quit IRC | 07:34 | |
| *** hjensas has quit IRC | 07:36 | |
| yoctozepto | born2bake: replied | 07:36 |
| yoctozepto | ah, he's gone | 07:36 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 07:37 |
| hrw | arm64 job in progress | 07:45 |
| yoctozepto | mgoddard, mnasiadka: https://review.opendev.org/696047 - let's discuss it, merge and release, otherwise we get unhappy Ubunters | 07:52 |
| patchbot | patch 696047 - kolla-ansible - Ensure hostname resolved by kolla configured /etc/... - 2 patch sets | 07:52 |
| *** lemko has joined #openstack-kolla | 07:53 | |
| *** jbadiapa has joined #openstack-kolla | 08:20 | |
| *** pcaruana has quit IRC | 08:22 | |
| mnasiadka | yoctozepto: find an unhappy Ubunter here :) | 08:31 |
| yoctozepto | mnasiadka: sure: born2bake | 08:32 |
| yoctozepto | sorry, wrong guy | 08:32 |
| yoctozepto | :D | 08:33 |
| yoctozepto | but there was one | 08:33 |
| yoctozepto | and there are on ask afraid to visit us on irc | 08:33 |
| yoctozepto | ;D | 08:33 |
| yoctozepto | tbarron, brtknr: I merged manila bugs, they are the same, better track as one | 08:34 |
| yoctozepto | let tripleo know about it | 08:34 |
| yoctozepto | brtknr: could you verify train? | 08:34 |
| *** rpittau|afk is now known as rpittau | 08:36 | |
| *** hjensas has joined #openstack-kolla | 08:42 | |
| hrw | mnasiadka, yoctozepto: help py2 to die? https://review.opendev.org/#/c/695168/ | 08:55 |
| patchbot | patch 695168 - kolla-ansible - Get rid of Python 2 support - 11 patch sets | 08:55 |
| *** strigazi has joined #openstack-kolla | 08:55 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 08:58 |
| *** zhanglong has joined #openstack-kolla | 09:08 | |
| mgoddard | morning | 09:17 |
| *** dougsz has joined #openstack-kolla | 09:17 | |
| hrw | arm64 not timed out! | 09:18 |
| mgoddard | yoctozepto: on https://review.opendev.org/696047, are you saying we should delay train for it? | 09:19 |
| patchbot | patch 696047 - kolla-ansible - Ensure hostname resolved by kolla configured /etc/... - 2 patch sets | 09:19 |
| mgoddard | what is the problem that it fixes for Ubunters? | 09:20 |
| *** k_mouza has joined #openstack-kolla | 09:21 | |
| *** k_mouza has quit IRC | 09:22 | |
| *** k_mouza has joined #openstack-kolla | 09:22 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 09:26 |
| *** k_mouza has quit IRC | 09:26 | |
| *** k_mouza has joined #openstack-kolla | 09:27 | |
| *** k_mouza has quit IRC | 09:28 | |
| *** k_mouza has joined #openstack-kolla | 09:29 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: skydive: mark as unbuildable on !x86 https://review.opendev.org/698430 | 09:33 |
| hrw | skydive change is dependency for arm64 ci now | 09:34 |
| yoctozepto | mgoddard: it seems ubuntu defaults to breaking /etc/hosts | 09:34 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 09:35 |
| mgoddard | yoctozepto: since when? | 09:35 |
| mgoddard | yoctozepto: and how? | 09:35 |
| mgoddard | yoctozepto: it won't in general know about every host in the baremetal group | 09:36 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 09:36 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add Debian/source aarch64 experimental job https://review.opendev.org/557659 | 09:36 |
| *** zainub_wahid has quit IRC | 09:37 | |
| *** dpawlik has quit IRC | 09:38 | |
| yoctozepto | mgoddard: no, that's why I commented we should really drop the host itself | 09:39 |
| yoctozepto | as it is what cause the havoc | 09:39 |
| yoctozepto | mgoddard: since no idea when, but ppl coming with this breakage recently | 09:40 |
| yoctozepto | could be from maas | 09:40 |
| mgoddard | yoctozepto: that makes a bit more sense | 09:40 |
| yoctozepto | mgoddard: yeah, if Chason does not come to fix it, I will take over it | 09:40 |
| mgoddard | I don't like the proposed patch | 09:40 |
| yoctozepto | mgoddard: -1 from me as well | 09:40 |
| mnasiadka | so what are we even discussing about? :D | 09:41 |
| mgoddard | would accept removal of this host, if it is idempotent | 09:41 |
| yoctozepto | mnasiadka: that fix is needed, but better | 09:41 |
| mnasiadka | yoctozepto: so go fix it, send a mail to chason if he's going to continue - he's not very active lately | 09:42 |
| mnasiadka | yoctozepto, mgoddard, hrw: changing subject - I could use some criticism on https://review.opendev.org/#/q/topic:bp/container-health-check | 09:42 |
| yoctozepto | mnasiadka: will do, just busy with other stuff now, he got notified by gerrit already | 09:43 |
| mnasiadka | yoctozepto: you know where gerrit notifications land normally, right? :) | 09:43 |
| yoctozepto | mnasiadka: Chason reacts upon them correctly :-) | 09:44 |
| hrw | mnasiadka: kolla one looks fine, +2 material etc. but can you explain in two sentences what it does? | 09:44 |
| mnasiadka | hrw: in the commit message or here? :) | 09:46 |
| mgoddard | hrw: lol. +2, what does it do? | 09:46 |
| hrw | mnasiadka: commit or both | 09:46 |
| *** factor has quit IRC | 09:46 | |
| hrw | mgoddard: the code itself does not affect how images work. will not break them for normal users. | 09:47 |
| *** factor has joined #openstack-kolla | 09:47 | |
| mnasiadka | well, in the commit message there's a link to what healthchecks do, the kolla change is only some scripts we use in kolla-ansible to check if the service hosted in the docker container is ,,healthy'' | 09:47 |
| yoctozepto | lol, quality reviews xD | 09:47 |
| mnasiadka | hm, I just noticed fluentd healthcheck is a bit awkward, let me look into that | 09:47 |
| mnasiadka | hrw: an example of working healthchecks: https://zuul.opendev.org/t/openstack/build/befc7727953949c996990fe62054721a/log/primary/logs/system_logs/docker-info.txt.gz#159 | 09:47 |
| mnasiadka | if the container is unhealthy - docker engine will restart it | 09:48 |
| mnasiadka | (hoping that solves the issue) | 09:48 |
| hrw | mnasiadka: thanks | 09:49 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Use Docker healthchecks for core services https://review.opendev.org/676389 | 09:49 |
| mnasiadka | fixing fluentd unhealthy ^^ | 09:50 |
| *** xinliang has quit IRC | 09:50 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:54 |
| hrw | mnasiadka: if (new_healthcheck[key1] * 1000000000) != current_healthcheck[key2]: - move from other config where time was in ns? | 09:54 |
| *** dpawlik has joined #openstack-kolla | 09:54 | |
| *** farhanjamil has joined #openstack-kolla | 09:55 | |
| hrw | mgoddard: is there a way to setup zuul jobs as separate set than zuul ones? | 09:55 |
| mnasiadka | hrw: well, Docker HealthConfig struct in Docker Engine is in nanoseconds, I don't think it's sensible for us to use nanoseconds in Ansible :) | 09:55 |
| hrw | mgoddard: arm64 CI works now, can be merged but I am afraid that with 8 nodes we will have lot of CI jobs waiting in queue | 09:56 |
| mnasiadka | hrw: if there's more elegant solution - I'll gladly hear about it, it's my first approach to python since python 2.3 :) | 09:56 |
| hrw | mgoddard: kind of opendev-zuul + opendev-zuul-arm64 | 09:56 |
| hrw | mnasiadka: # NOTE(mnasiadka) something about time? | 09:56 |
| mnasiadka | hrw: makes sense, will do :) | 09:57 |
| hrw | mnasiadka: and you can use 1e9 to make it easier to count | 09:57 |
| hrw | will comment | 09:57 |
| *** zainub_wahid has joined #openstack-kolla | 09:57 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:58 |
| *** jbadiapa has quit IRC | 09:59 | |
| hrw | mnasiadka: and below there is comment about time conversion ;D | 09:59 |
| mnasiadka | hrw, mgoddard, yoctozepto: do we need debian-ceph-ansible zuul job? currently with kolla-ceph we only have centos and ubuntu jobs | 10:00 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix permission error when qemu-kvm loads ceph.conf https://review.opendev.org/680740 | 10:00 |
| mgoddard | mnasiadka: we can't add every job for debian + ubuntu | 10:01 |
| mnasiadka | mgoddard: I know, asking if anybody sees a problem with no job for debian | 10:01 |
| mnasiadka | and it's still CentOS container on top of Ubuntu | 10:01 |
| mnasiadka | :) | 10:01 |
| mgoddard | it's not perfect, but our test coverage is far from 100% | 10:01 |
| mgoddard | ubuntu ~= debian | 10:02 |
| mnasiadka | yeah | 10:02 |
| mgoddard | if we pass debian aio that's a good start | 10:02 |
| mnasiadka | I'm working towards testing all services that integrate with Ceph in the CI job | 10:02 |
| mnasiadka | that's second alternative - we could test Ceph as AIO instead of multinode | 10:02 |
| *** gfidente has joined #openstack-kolla | 10:03 | |
| mnasiadka | we no longer need to test if ceph deployment works multinode - we have ceph-ansible for that | 10:04 |
| *** farhanjamil has quit IRC | 10:07 | |
| hrw | mnasiadka: commented | 10:09 |
| mnasiadka | hrw: I commented the ansible-lint related fixes change (https://review.opendev.org/#/c/694797/) but it might make sense to merge it as it is, and check for passwords in the ansible deploy logs as a separate task | 10:09 |
| patchbot | patch 694797 - kolla-ansible - Some ansible-lint fixes - 11 patch sets | 10:09 |
| mnasiadka | hrw: thanks | 10:09 |
| hrw | mnasiadka: no_log stuff? | 10:09 |
| mnasiadka | yeah, it was just a note - I'll add this to my ansible todos | 10:10 |
| mnasiadka | this change is not about this ;) | 10:10 |
| hrw | ok | 10:10 |
| mnasiadka | +w and off it goes | 10:10 |
| hrw | mnasiadka: healthchecks k-a patch - I was not sure how it goes with values in some places. you have abc_healthcheck and then abc_healthcheck={} is all. maybe just do |default:{} and consider it done? | 10:12 |
| *** shyamb has joined #openstack-kolla | 10:13 | |
| mnasiadka | hrw: I need to re-verify this, I think with introducing enable_container_healthchecks var it can go away now | 10:13 |
| hrw | mnasiadka: I like the idea of those healthchecks | 10:14 |
| mnasiadka | hrw: well, a lot of people voted on that feature in PTG, so I guess it makes sense :) | 10:16 |
| *** dpawlik has quit IRC | 10:19 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 10:21 |
| *** dpawlik has joined #openstack-kolla | 10:22 | |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Fix fernet_token_expiry precheck https://review.opendev.org/698441 | 10:23 |
| hrw | https://review.opendev.org/#/c/698430 needs another brave core ;D | 10:24 |
| patchbot | patch 698430 - kolla - skydive: mark as unbuildable on !x86 - 1 patch set | 10:24 |
| hrw | trivial change | 10:24 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 10:24 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Fix fernet_token_expiry precheck https://review.opendev.org/698441 | 10:25 |
| *** zhanglong has quit IRC | 10:25 | |
| hrw | bbl | 10:26 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Fix fernet_token_expiry precheck https://review.opendev.org/698441 | 10:28 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 10:28 |
| *** shyamb has quit IRC | 10:33 | |
| *** shyamb has joined #openstack-kolla | 10:48 | |
| *** sri_ has quit IRC | 10:50 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Use python3 for openstack clients https://review.opendev.org/696062 | 10:52 |
| openstackgerrit | yatin proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 11:03 |
| openstackgerrit | Merged openstack/kayobe master: Switch to generic package module https://review.opendev.org/691393 | 11:04 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Fix fernet_token_expiry precheck https://review.opendev.org/698441 | 11:05 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Refactor Apache httpd setup https://review.opendev.org/698255 | 11:05 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 11:05 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: work around mod_ssl certificate issue https://review.opendev.org/698445 | 11:05 |
| *** schwicht has quit IRC | 11:11 | |
| *** k_mouza has quit IRC | 11:17 | |
| *** k_mouza has joined #openstack-kolla | 11:17 | |
| *** dpawlik has quit IRC | 11:21 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 11:24 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 11:25 |
| mnasiadka | mgoddard: that's interesting - https://zuul.opendev.org/t/openstack/build/a4e8f51899f24a8c89a0c9384a2d4533/log/primary/logs/ansible/upgrade#10988 | 11:28 |
| mnasiadka | :D | 11:28 |
| mgoddard | mnasiadka: that is odd. strange that it got that far before failing | 11:29 |
| mnasiadka | yeah | 11:29 |
| *** shyam89 has joined #openstack-kolla | 11:31 | |
| *** shyamb has quit IRC | 11:31 | |
| *** pcaruana has joined #openstack-kolla | 11:36 | |
| *** shyam89 has quit IRC | 11:38 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Refactor Apache httpd setup https://review.opendev.org/698255 | 11:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: work around mod_ssl certificate issue https://review.opendev.org/698445 | 11:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 11:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: work around mod_ssl certificate issue https://review.opendev.org/698445 | 11:41 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 11:41 |
| *** sri_ has joined #openstack-kolla | 11:51 | |
| *** sean-k-mooney is now known as skm | 11:55 | |
| *** k_mouza has quit IRC | 11:55 | |
| *** skm is now known as sean-k-mooney | 11:55 | |
| *** k_mouza has joined #openstack-kolla | 11:56 | |
| mnasiadka | analyzing the keystone error 500 thing (especially in the healthchecks change), it seems only one out of three keystones is in healthy state | 11:57 |
| mnasiadka | and this only happens in multinode it seems | 11:57 |
| yoctozepto | mnasiadka: deprecate mariadb | 11:57 |
| mnasiadka | yoctozepto: actually I don't think it's mariadb's fault | 11:58 |
| yoctozepto | mnasiadka: well, everything is bound to keystone | 11:58 |
| yoctozepto | and keystone is bound to mariadb | 11:58 |
| yoctozepto | so | 11:58 |
| yoctozepto | :D | 11:58 |
| yoctozepto | mgoddard, mnasiadka: yeah, the lack of basic ansible vars errors are just lovely | 11:59 |
| yoctozepto | wonder if the nodes are really stable or we get some memory corruption (or we found the reason of ceph-ansible deprecation) | 12:00 |
| *** dpawlik has joined #openstack-kolla | 12:00 | |
| mnasiadka | yoctozepto: well, this was in hrw's ansible-lint related change :D | 12:02 |
| yoctozepto | mnasiadka: they happen very rarily but do | 12:02 |
| yoctozepto | as if some internal state broke for some reason | 12:03 |
| yoctozepto | could be some tricky race condition as well | 12:03 |
| *** spsurya has joined #openstack-kolla | 12:05 | |
| *** dpawlik has quit IRC | 12:05 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test keystone wsgi LogLevel debug https://review.opendev.org/698452 | 12:08 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:13 |
| *** vmixor has joined #openstack-kolla | 12:13 | |
| *** dpawlik has joined #openstack-kolla | 12:20 | |
| *** dougsz has quit IRC | 12:21 | |
| yoctozepto | mnasiadka: I'm afraid you might need strace | 12:21 |
| *** hamdyk has joined #openstack-kolla | 12:21 | |
| yoctozepto | what came to my mind, might be keystone is doing heavy locking with each action | 12:22 |
| yoctozepto | hence only one is deemed healthy | 12:22 |
| yoctozepto | and our round-robin scheduling increases failure ratio in general | 12:22 |
| yoctozepto | as in those funny failures when you always get the 3rd node to fail :-) | 12:23 |
| yoctozepto | i.e. do multiple keystone only for HA, no scaling whatsoever | 12:23 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Use Docker healthchecks for core services https://review.opendev.org/676389 | 12:23 |
| hamdyk | Hi all, I want to add neutron plugin (networking-mlnx) to neutron-base image. For binary it's no problem, but for source, how can I generate and upload the tar files to add it to kolla/common/config.py ? | 12:24 |
| *** dpawlik has quit IRC | 12:25 | |
| tbarron | yoctozepto: thanks for combining the bugs and giving triplo folks a heads up | 12:27 |
| mnasiadka | hamdyk: just add it to kolla/common/config.py, networking-mlnx is publishing tarballs on tarballs.openstack.org, right? | 12:28 |
| mnasiadka | hamdyk: and is there a reason you need it in neutron-base? neutron-server will not suffice? | 12:28 |
| tbarron | yoctozepto: we (manila) are going to need to work with Ceph folks proper since the issue is in this ceph-volume-library which | 12:29 |
| tbarron | yoctozepto: although python is part of Ceph itself rather than the manila driver proper | 12:29 |
| tbarron | yoctozepto: the ceph folks are migrating clients off of it to a rest interface so that e.g. golang clients can use the same capability without pythohn bindings | 12:29 |
| tbarron | yoctozepto: but we've got to get this fixed anyways | 12:29 |
| hamdyk | mnasiadka: not sure about the publishing, will check | 12:30 |
| tbarron | brtknr: ^^ just fyi | 12:30 |
| hamdyk | mnasiadka: we need the plugin to be used in neutron-server, neuton-dhcp and neutron-l3 | 12:30 |
| *** dpawlik has joined #openstack-kolla | 12:33 | |
| mnasiadka | hamdyk: sure, just asking - write the explanation in the commit message ;-) | 12:34 |
| yoctozepto | tbarron: ack, so this affects everything up to ussuri, eh? | 12:35 |
| hamdyk | mnasiadka: so I can see http://tarballs.openstack.org/networking-mlnx/, now all I need to do is to modify kolla/common/config.py and add the package with prefix "neutron-base-plugin-" ? | 12:35 |
| mnasiadka | hamdyk: seems so | 12:36 |
| hamdyk | mnasiadka: great, will try it | 12:36 |
| hamdyk | mnasiadka: thnx :) | 12:36 |
| *** negronjl has quit IRC | 12:40 | |
| tbarron | yoctozepto: well it affects whatever openstack release you use if you use post-luminous so far as I can tell atm, so yes | 12:41 |
| *** negronjl has joined #openstack-kolla | 12:41 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:43 |
| tbarron | yoctozepto: assuming that the deployment bakes ceph versions into container images (not all do of courxe) | 12:44 |
| *** k_mouza_ has joined #openstack-kolla | 12:44 | |
| tbarron | yoctozepto: upstream manila CI does not and the devkstack-ceph-plugin wax still deploying luminous so the | 12:45 |
| tbarron | issue is not seen there even on master | 12:45 |
| yoctozepto | tbarron: I see, thanks for your explanation | 12:45 |
| tbarron | yoctozepto: but in my downstream job we use tripleo with kolla images (but lag behind upstream) so I'm motivated to get a fix :) | 12:46 |
| *** k_mouza has quit IRC | 12:47 | |
| *** k_mouza_ has quit IRC | 12:49 | |
| *** k_mouza has joined #openstack-kolla | 12:49 | |
| hrw | folks: RDO has meeting just before our one. I will log but not being able to attend. Probably have to skip our one too today | 12:54 |
| hrw | yoctozepto: skydive - you should -1 https://review.opendev.org/#/c/696709/ when it was time for it ;D | 12:57 |
| patchbot | patch 696709 - kolla - Remove skydive rpm installation (MERGED) - 1 patch set | 12:57 |
| hrw | yoctozepto: will handle | 12:57 |
| yoctozepto | hrw: hmm, that did not change buildability | 12:58 |
| yoctozepto | but you are right about releasenote | 12:58 |
| yoctozepto | mnasiadka: ^ | 12:58 |
| yoctozepto | hrw: where is that rdo channel? :D | 12:58 |
| yoctozepto | ok, here #rdo | 12:59 |
| mnasiadka | hrw: usually they had theirs at the same time | 12:59 |
| mnasiadka | I'm normally following both and complaining over something on #rdo | 13:00 |
| yoctozepto | mnasiadka: may you complain about manila today | 13:00 |
| yoctozepto | or really just raise their awareness | 13:00 |
| mnasiadka | well, it's interesting because RDO is testing some setups, right? | 13:01 |
| yoctozepto | well, tripleo too | 13:01 |
| yoctozepto | probably not manila share | 13:01 |
| yoctozepto | or not with the latest ceph | 13:01 |
| yoctozepto | for whatever reason | 13:01 |
| hrw | mnasiadka: hm. maybe right. I had to calculate GMT offset wrong again | 13:01 |
| mnasiadka | yoctozepto: we haven't tested, we are not to blame :D | 13:01 |
| yoctozepto | mnasiadka: surely :D | 13:02 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: skydive: mark as unbuildable on !x86 https://review.opendev.org/698430 | 13:06 |
| mnasiadka | yoctozepto: coming back to "option httpchk" discussion, it seems most services should support healthchecks - https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html | 13:07 |
| hrw | yoctozepto: here you have | 13:07 |
| yoctozepto | mnasiadka: and you are going for /healthcheck or / ? | 13:07 |
| mnasiadka | yoctozepto: I'll check on some deployment and I guess it makes sense to start doing real healthchecks on haproxy level too | 13:08 |
| yoctozepto | hrw: thx | 13:09 |
| yoctozepto | mnasiadka: well, currently haproxy has issues with its script | 13:09 |
| yoctozepto | it seems to destabilize rather than stabilize the situation | 13:09 |
| yoctozepto | I mean the keepalived check | 13:09 |
| *** zainub_wahid has quit IRC | 13:10 | |
| *** k_mouza has quit IRC | 13:11 | |
| *** schwicht has joined #openstack-kolla | 13:12 | |
| mnasiadka | yoctozepto: you mean the check script that pollutes haproxy logs? :) | 13:16 |
| *** goldyfruit_ has joined #openstack-kolla | 13:17 | |
| yoctozepto | mnasiadka: possibly, at least the one that breaks otherwise valid jobs | 13:26 |
| *** k_mouza has joined #openstack-kolla | 13:26 | |
| mnasiadka | yoctozepto: https://github.com/openstack/kolla/blob/master/docker/keepalived/check_alive.sh - that one? or I'm again sleeping on the job? :) | 13:27 |
| *** schwicht has quit IRC | 13:28 | |
| yoctozepto | mnasiadka: that one | 13:29 |
| mnasiadka | yoctozepto: how does it break jobs? | 13:29 |
| *** wuchunyang has quit IRC | 13:33 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 13:35 |
| hrw | https://review.opendev.org/#/c/662456 https://review.opendev.org/#/c/557659 - anyone? | 13:35 |
| patchbot | patch 662456 - kolla - Add CentOS/source aarch64 experimental job - 32 patch sets | 13:35 |
| patchbot | patch 557659 - kolla - Add Debian/source aarch64 experimental job - 57 patch sets | 13:35 |
| openstackgerrit | Hamdy Khader proposed openstack/kolla master: Add Networking-MLNX to Neutron-Base packages https://review.opendev.org/689110 | 13:39 |
| *** dougsz has joined #openstack-kolla | 13:40 | |
| openstackgerrit | Hamdy Khader proposed openstack/kolla master: Add Networking-MLNX to Neutron-Base packages https://review.opendev.org/689110 | 13:41 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Use python3 for openstack clients https://review.opendev.org/696062 | 13:45 |
| openstackgerrit | Hamdy Khader proposed openstack/kolla master: Add Networking-MLNX to Neutron-Base packages https://review.opendev.org/689110 | 13:50 |
| yoctozepto | mnasiadka: timeouts and makes vip address move away | 13:55 |
| yoctozepto | there are two issues, one is keepalived behaving weirdly on timeout | 13:55 |
| yoctozepto | second is the timeout itself :-) | 13:55 |
| yoctozepto | I have this reported somewhere | 13:56 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Fix fernet_token_expiry precheck https://review.opendev.org/698441 | 13:58 |
| *** schwicht has joined #openstack-kolla | 13:59 | |
| *** cah_link has joined #openstack-kolla | 13:59 | |
| *** goldyfruit_ has quit IRC | 14:02 | |
| *** cah_link has quit IRC | 14:03 | |
| *** k_mouza has quit IRC | 14:07 | |
| *** k_mouza has joined #openstack-kolla | 14:07 | |
| osmanlicilegi | hrw: can you have a look at https://review.opendev.org/#/c/694799/ when you have some time | 14:21 |
| patchbot | patch 694799 - kolla - WIP: Upgrade RabbitMQ to 3.8 - 10 patch sets | 14:21 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Remove fernet_token_expiry precheck https://review.opendev.org/698441 | 14:30 |
| mnasiadka | yoctozepto: so lets change it to killall -0 haproxy | 14:30 |
| *** hamdyk has quit IRC | 14:34 | |
| *** watersj has joined #openstack-kolla | 14:35 | |
| hrw | osmanlicilegi: done. -1 | 14:37 |
| yoctozepto | mnasiadka: what does it do? | 14:37 |
| *** schwicht has quit IRC | 14:38 | |
| mnasiadka | yoctozepto: the same what docker does - checks if haproxy is running | 14:38 |
| mnasiadka | yoctozepto: probably better to disable it in CI for sure :) | 14:40 |
| mnasiadka | (The check) | 14:40 |
| *** hjensas has quit IRC | 14:40 | |
| yoctozepto | mnasiadka: ah, I read it again and now get what you meant :D | 14:43 |
| *** sri_ has quit IRC | 14:44 | |
| yoctozepto | well, it might be more reasonable to rely on a simpler check either way | 14:45 |
| yoctozepto | no need to stress haproxy with extra requests via the unix socket | 14:46 |
| yoctozepto | still, it's interesting why it fails to respond in several seconds, could be related to general load | 14:46 |
| yoctozepto | never had it locally | 14:46 |
| yoctozepto | see: https://bugs.launchpad.net/kolla-ansible/+bug/1847727 | 14:48 |
| openstack | Launchpad bug 1847727 in kolla-ansible "CI: failing on CRITICAL may be still too strong" [Medium,Triaged] | 14:48 |
| yoctozepto | https://review.opendev.org/#/c/690702/ | 14:48 |
| patchbot | patch 690702 - kolla-ansible - DNM: haproxy timeout - raise value for server - 3 patch sets | 14:48 |
| yoctozepto | and | 14:48 |
| yoctozepto | https://review.opendev.org/#/c/690920/ | 14:48 |
| patchbot | patch 690920 - kolla-ansible - DNM: keepalived testing - 10 patch sets | 14:48 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Start testing CentOS 8 hosts https://review.opendev.org/693544 | 14:54 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Deploy CentOS 8 containers https://review.opendev.org/697690 | 14:54 |
| mgoddard | meeting in 5 | 14:54 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 14:54 |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne ^ | 14:55 |
| mnasiadka | yoctozepto: yeah, some things only happen in the CI, right? :) | 14:57 |
| yoctozepto | mnasiadka: unfortunately! | 14:58 |
| *** wuchunyang has joined #openstack-kolla | 14:58 | |
| *** chason has joined #openstack-kolla | 14:59 | |
| * hrw off | 14:59 | |
| *** chason has left #openstack-kolla | 15:00 | |
| yoctozepto | meeting time | 15:01 |
| yoctozepto | mgoddard | 15:01 |
| mgoddard | yessir | 15:01 |
| mgoddard | #startmeeting | 15:01 |
| openstack | mgoddard: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 15:01 |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Dec 11 15:01:21 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| openstack | The meeting name has been set to 'kolla' | 15:01 |
| mgoddard | #topic rollcall | 15:01 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| mgoddard | \o | 15:01 |
| jovial[m] | o/ | 15:01 |
| generalfuzz | o/ | 15:01 |
| *** chason has joined #openstack-kolla | 15:01 | |
| osmanlicilegi | o/ | 15:01 |
| yoctozepto | o/ | 15:01 |
| chason | o/ | 15:02 |
| *** graysonh has joined #openstack-kolla | 15:02 | |
| mnasiadka | o/ | 15:02 |
| *** scottsol has joined #openstack-kolla | 15:02 | |
| *** wuchunyang has quit IRC | 15:02 | |
| *** lemko has quit IRC | 15:03 | |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:03 |
| mgoddard | * Announcements | 15:03 |
| mgoddard | * Review action items from last meeting | 15:03 |
| mgoddard | * CI status | 15:03 |
| mgoddard | * Train release planning | 15:03 |
| mgoddard | * Ussuri release planning | 15:03 |
| mgoddard | ** Support for Train redeployment on C8, we are already diverging with support for Train | 15:03 |
| mgoddard | * Review priorities | 15:03 |
| mgoddard | * Should we check for supported OS distro & release in prechecks? | 15:03 |
| mgoddard | * CI scenarios | 15:03 |
| mgoddard | #topic announcements | 15:03 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:03 | |
| mgoddard | #info Kayobe RC1 planned this week | 15:03 |
| mgoddard | #info kolla RC2 releases proposed | 15:04 |
| mgoddard | I do wonder if the release team have become a bottleneck. Those patches have been up for a couple days | 15:04 |
| mgoddard | *patch | 15:04 |
| mgoddard | Any others? | 15:05 |
| mnasiadka | they announced some time ago they are happy to teach new members... so maybe that's the case | 15:05 |
| yoctozepto | yea, it's happening again | 15:05 |
| yoctozepto | they were delayed previously too | 15:05 |
| yoctozepto | mnasiadka: we should go get taught :-) | 15:05 |
| mgoddard | seems to me like teams should have control over their own releases | 15:06 |
| mgoddard | similar to the recent stable branch discussion | 15:06 |
| mgoddard | anyway | 15:07 |
| mgoddard | #topic Review action items from last meeting | 15:07 |
| *** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:07 | |
| mgoddard | We had two | 15:07 |
| mgoddard | mgoddard to update kayobe train release etherpad | 15:07 |
| mgoddard | osmanlicilegi to mark senlin images as unbuildable for ubuntu & debian binary | 15:07 |
| mgoddard | I did mine | 15:07 |
| mgoddard | osmanlicilegi did his | 15:07 |
| mgoddard | boom | 15:07 |
| osmanlicilegi | yep | 15:07 |
| mgoddard | #topic CI status | 15:07 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:08 | |
| mgoddard | thanks osmanlicilegi :) | 15:08 |
| osmanlicilegi | yw | 15:08 |
| mgoddard | CI looking good AFAIK, although we are running train tarballs on master | 15:08 |
| yoctozepto | only for c7 | 15:08 |
| mgoddard | true | 15:09 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 15:09 |
| mgoddard | #topic Train release planning | 15:09 |
| *** openstack changes topic to "Train release planning (Meeting topic: kolla)" | 15:09 | |
| mgoddard | Kolla is looking good at this point, just waiting for release handle cranking | 15:09 |
| *** schwicht has joined #openstack-kolla | 15:09 | |
| mgoddard | jovial[m]: how about kayobe? | 15:10 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Add kayobe as openstack project for release notes https://review.opendev.org/697701 | 15:10 |
| jovial[m] | Still need the nova cells work and anible version bump patches to be completed. | 15:11 |
| mgoddard | jovial[m]: and this one: https://review.opendev.org/#/c/694616/ | 15:11 |
| jovial[m] | Other than that it's just reviews we need. | 15:11 |
| patchbot | patch 694616 - kayobe - Use OpenStack Train release - 2 patch sets | 15:11 |
| mgoddard | Release tracker here: https://etherpad.openstack.org/p/kayobe-train-release | 15:12 |
| jovial[m] | Oh yeah, I will update that one shortly. | 15:12 |
| mgoddard | #link https://etherpad.openstack.org/p/kayobe-train-release | 15:12 |
| *** chason has quit IRC | 15:13 | |
| mgoddard | 7 patches left to review | 15:13 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 15:13 |
| mgoddard | let's keep pushing on it | 15:13 |
| mgoddard | #topic Ussuri release planning | 15:14 |
| *** openstack changes topic to "Ussuri release planning (Meeting topic: kolla)" | 15:14 | |
| mgoddard | Onto U | 15:14 |
| yoctozepto | onto U! | 15:14 |
| *** rgogunskiy has quit IRC | 15:14 | |
| mgoddard | I've been spending some time on centos 8 recently | 15:14 |
| mgoddard | current status is we have an aio centos8/source job passing in CI | 15:15 |
| yoctozepto | hurrary | 15:15 |
| yoctozepto | hurray* | 15:15 |
| mgoddard | it required a few hacks & fudges, trying to extract the correct pieces into other patches to get them merged | 15:15 |
| mnasiadka | so it's in good progress | 15:16 |
| mnasiadka | although we're missing ceph client libraries, right? | 15:16 |
| mgoddard | yatin pointed out that the main delorean RDO repo is now available, so I've just pushed a patch to test centos8/binary aio | 15:16 |
| *** dpawlik has quit IRC | 15:16 | |
| mgoddard | mnasiadka: yes | 15:16 |
| mgoddard | and a few other things | 15:16 |
| mnasiadka | ok, around ceph - we should get it built in a couple of weeks, there's a group that is pushing all the deps to EPEL el8 | 15:16 |
| mnasiadka | and there's only a couple left | 15:17 |
| mnasiadka | mainly pecan and libs depending on pecan | 15:17 |
| mgoddard | nice | 15:18 |
| yoctozepto | very good | 15:18 |
| mgoddard | we know that we need to support c7 & c8 on train, so bear that in mind when writing and reviewing these patches | 15:18 |
| mgoddard | (backport) | 15:18 |
| mnasiadka | we already know that backporting will be a bit painful, due to some changes in kolla (like repo enablement on a different level) | 15:19 |
| mgoddard | I think the most sane way to support that is to support both in master, then remove c7 | 15:19 |
| mgoddard | i.e. parallel build & deploy jobs | 15:19 |
| mnasiadka | yeah, that's probably a good idea to follow that path | 15:20 |
| yoctozepto | yea, we already broke qinling now | 15:20 |
| yoctozepto | sorry, senlin* | 15:20 |
| mgoddard | I'm not sure what we do about publishing | 15:20 |
| mgoddard | should we publish images with a base_distro of centos8? | 15:20 |
| mnasiadka | mgoddard: I think we have no other option | 15:21 |
| mgoddard | true for train, but we could avoid it on master if we wish | 15:21 |
| mgoddard | maybe it doesn't matter | 15:21 |
| mnasiadka | I think a sane approach would be to have base_distro = centos8, and deprecate/remove centos (as in centos7) | 15:22 |
| mnasiadka | less people complaining maybe | 15:22 |
| mnasiadka | but I guess we'll work it out on the way | 15:22 |
| mgoddard | hmm, rather than switching back to centos on master? | 15:22 |
| mnasiadka | maybe it doesn't make any sense, and we should switch back to centos on master | 15:23 |
| openstackgerrit | Merged openstack/kayobe master: Add example to Bridges and VLANs section https://review.opendev.org/693368 | 15:23 |
| yoctozepto | centos8 sounds nice | 15:23 |
| yoctozepto | we might soon require that for ubuntu20 | 15:23 |
| mgoddard | would require a lot of conditional logic changes | 15:23 |
| yoctozepto | encoding distro version sounds nice | 15:23 |
| mnasiadka | my worry is that we'll get centos8 master tagged images we would need to remove from docker hub | 15:23 |
| yoctozepto | mgoddard: hmm, could be | 15:24 |
| yoctozepto | mnasiadka: yea, going back and forth sounds bad | 15:24 |
| yoctozepto | my idea is to stay | 15:24 |
| yoctozepto | rethink logic to accommodate it | 15:24 |
| mgoddard | or we encode the OS version in the image name | 15:24 |
| mnasiadka | like centos-8-source-xxx ? | 15:25 |
| mgoddard | or kolla/centos-8-source-base:train | 15:25 |
| mgoddard | yeah | 15:25 |
| mnasiadka | well, we have time to rethink the pros and cons of both solutions I think | 15:25 |
| mgoddard | or kolla/centos-source-base:train-centos8 | 15:25 |
| mgoddard | as a one-off | 15:25 |
| mnasiadka | yeah, the tag thing as a one off should probably work | 15:26 |
| mnasiadka | we could shorten it to train-c8 | 15:26 |
| mnasiadka | :) | 15:26 |
| mgoddard | yeah | 15:26 |
| mgoddard | that's probably best | 15:26 |
| mnasiadka | yup | 15:26 |
| mgoddard | we need something we can decide on at runtime in ansible, based on ansible_distro_major_version | 15:27 |
| mgoddard | for mixed c7/8 | 15:27 |
| mgoddard | ok, we have some things to think about | 15:27 |
| mgoddard | while we are on this topic, we have a subtopic proposed | 15:28 |
| mgoddard | Support for Train redeployment on C8, we are already diverging with support for Train | 15:28 |
| mgoddard | yoctozepto: is that you? | 15:28 |
| yoctozepto | mgoddard: yea | 15:28 |
| yoctozepto | so we need to think about it when reviewing | 15:28 |
| yoctozepto | and also need some mechanism to support both ways | 15:28 |
| yoctozepto | as we are going to lose train re-deployability | 15:29 |
| yoctozepto | on c8 | 15:29 |
| mgoddard | you mean migrating an existing train c7 cloud to c8? | 15:30 |
| yoctozepto | mhm | 15:30 |
| yoctozepto | now senlin would fail to redeploy because of the logic change | 15:30 |
| yoctozepto | it could be really hard to allow both releases if more projects start diverging | 15:30 |
| mgoddard | but we'd be using stable/train k-a to do this | 15:30 |
| yoctozepto | ah | 15:31 |
| yoctozepto | did not get that | 15:31 |
| yoctozepto | :D | 15:31 |
| mgoddard | at least that was the generally preferred option I think | 15:31 |
| yoctozepto | then all is fine | 15:31 |
| *** negronjl has quit IRC | 15:31 | |
| mnasiadka | but maybe it would make sense to have a change with zuul job that does the migration? | 15:31 |
| mgoddard | yes, we need that | 15:31 |
| mgoddard | but on stable/train | 15:32 |
| mnasiadka | of course | 15:32 |
| mgoddard | I think we would have a multinode with mixed c7/8 | 15:33 |
| mgoddard | deploy to c7, then add the c8 node to the inventory and deploy again | 15:34 |
| yoctozepto | so we need to support c8 as host on train | 15:34 |
| mnasiadka | well, host can be c7, containers can be c8 :) | 15:34 |
| yoctozepto | not sure we do already | 15:34 |
| mgoddard | yes | 15:34 |
| *** negronjl has joined #openstack-kolla | 15:34 | |
| mgoddard | no, we do not | 15:34 |
| yoctozepto | :D | 15:34 |
| mgoddard | there will be much backporting | 15:34 |
| yoctozepto | mhm | 15:34 |
| mnasiadka | I guess we need to support all three mixes for the migration/upgrade process? c7 on c7, c8 on c7, c8 on c8? | 15:35 |
| mnasiadka | or c7 on c8? :) | 15:35 |
| mgoddard | mnasiadka: no - tripleo isn't going to mix host and container versions | 15:35 |
| mgoddard | so we shoulnd't either | 15:36 |
| mgoddard | c7 host runs c7 containers | 15:36 |
| mnasiadka | mgoddard: well, they control the OS with Heat stack, we need to state what we support in the docs | 15:36 |
| mgoddard | new c8 hosts come up with c8 containers | 15:36 |
| mnasiadka | that's why I'm asking | 15:36 |
| mgoddard | that's what I meant earlier with ansible_distro_major_version - we will select the container tag based on the host | 15:36 |
| yoctozepto | c8 on c7 unsupported | 15:37 |
| mnasiadka | makes sense | 15:37 |
| mnasiadka | so we don't expose that to the user in any other way, we deploy c8 containers on c8 host | 15:37 |
| mgoddard | we should probably write this stuff down somewhere :) | 15:37 |
| mnasiadka | yeah, most probably - we need a draft of a proposed migration process and a zuul job that tests it :) | 15:38 |
| mgoddard | it'll be a variable. they can shoot themselves in the foot if they want | 15:38 |
| yoctozepto | :D | 15:38 |
| mnasiadka | good, I love users shooting in the foot | 15:38 |
| mgoddard | openstack_release & openstack_release_centos8? | 15:39 |
| mgoddard | or maybe it needs to be configurable for each container? | 15:39 |
| mgoddard | hmm, perhaps base_distro would be easier from ansible perspective | 15:39 |
| mgoddard | would be a lot easier if they just supported upgrades from 7 to 8... | 15:40 |
| mgoddard | Any other thoughts on this one? | 15:40 |
| mnasiadka | by the way, the centos 8 containers will be based on Stream, right? | 15:41 |
| mgoddard | mnasiadka: I think so | 15:41 |
| mnasiadka | yeah centos:8 is stream | 15:41 |
| mnasiadka | no more stupid questions, I guess as homework we all need to rethink the base_distro vs tag for next meeting | 15:42 |
| mgoddard | yeah | 15:43 |
| mgoddard | #topic Should we check for supported OS distro & release in prechecks? | 15:43 |
| *** openstack changes topic to "Should we check for supported OS distro & release in prechecks? (Meeting topic: kolla)" | 15:43 | |
| mgoddard | I noticed that OSA has some checks on the OS distro & release against a supported set | 15:44 |
| mnasiadka | might make sense, do we plan to check the docker engine version as well? | 15:44 |
| mgoddard | I also notice in bug reports people using untested ubuntu & kolla release combos | 15:44 |
| yoctozepto | +1 | 15:45 |
| mnasiadka | especially when we merge healthchecks, anything prior to docker 1.12 will not work | 15:45 |
| mgoddard | we could check docker engine version. I'm not sure we have much of a requirement on it | 15:45 |
| osmanlicilegi | +1 | 15:45 |
| mnasiadka | we have some requirements on docker API versions rather | 15:45 |
| mgoddard | hopefully no one is running anything earlier than 1.12 with ussuri :) | 15:45 |
| mnasiadka | we could check for that | 15:45 |
| mnasiadka | mgoddard: who knows... who knows :) | 15:46 |
| mnasiadka | especially people might not be aware about the maintenance cycles for docker-ce | 15:46 |
| mgoddard | how strict should we be with centos versions? | 15:46 |
| *** chkumar|ruck is now known as raukadah | 15:47 | |
| mgoddard | should we just say we support c7, or 7.7? | 15:47 |
| mgoddard | same with debian/ubuntu | 15:47 |
| mnasiadka | I think it should be c7 - c7 GA is supported normally until Q4 2020 | 15:47 |
| mnasiadka | well, with Ubuntu it's easy - we say Bionic, right? | 15:47 |
| mgoddard | I think so | 15:48 |
| mnasiadka | and Debian Buster I think? | 15:48 |
| mgoddard | yes | 15:48 |
| yoctozepto | image must match host | 15:48 |
| mgoddard | yeah | 15:48 |
| yoctozepto | otherwise very limited support | 15:48 |
| mgoddard | so does anyone want to pick this up? | 15:49 |
| yoctozepto | i.e. not necessarily rejecting patches | 15:49 |
| yoctozepto | but not caring much | 15:49 |
| osmanlicilegi | I can take it if nobody wants it :) I'm not sure if I can catch ussuri timelines but I'll try | 15:50 |
| mnasiadka | mgoddard: around picking up, maybe we could think about having a bucket list (or a more scary version - kanban board) - and each one of us with spare cycles could take something - at least I have a tendency of forgetting about things :) | 15:51 |
| mgoddard | I've added it to the whiteboard | 15:51 |
| mgoddard | https://blueprints.launchpad.net/kolla-ansible/+spec/os-prechecks | 15:51 |
| mnasiadka | goodie | 15:52 |
| mgoddard | always a good place to pick things up from | 15:53 |
| mgoddard | I'll put your name on the blueprint osmanlicilegi, remove it if you think you won't have time | 15:54 |
| mnasiadka | there are some... old blueprints with high priority - can we sort of think about cleaning it up? | 15:54 |
| osmanlicilegi | mgoddard: ok | 15:54 |
| mgoddard | mnasiadka: which ones? | 15:55 |
| mnasiadka | mgoddard: example - https://blueprints.launchpad.net/kolla-ansible/+spec/sanity-check-container | 15:55 |
| mnasiadka | it's there since Newton or longer :) | 15:55 |
| mgoddard | oh right, I thought you meant on the whiteboard | 15:55 |
| mgoddard | yeah, LP is a mess | 15:56 |
| mnasiadka | naah, whiteboard is maintained :) | 15:56 |
| mgoddard | I occasionally tidy it up until I get bored | 15:56 |
| mnasiadka | mgoddard: maybe let's just remove priority tag from those things we don't do in Ussuri? | 15:56 |
| mgoddard | +1 | 15:56 |
| mnasiadka | ok, I'll do that, because this mess hurts :) | 15:57 |
| yoctozepto | meh, just filter for ussuri :D | 15:57 |
| mgoddard | #action mnasiadka to remove priority from inactive blueprints | 15:58 |
| mgoddard | #topic Open discussion | 15:58 |
| *** openstack changes topic to "Open discussion (Meeting topic: kolla)" | 15:58 | |
| mgoddard | Anything else to cover? | 15:59 |
| yoctozepto | nah, /me overloaded | 15:59 |
| osmanlicilegi | mgoddard: I have a question for https://blueprints.launchpad.net/kolla-ansible/+spec/docker-image-pruning. should I go with ansible docker module or kolla_docker.py ? | 15:59 |
| yoctozepto | good question | 15:59 |
| osmanlicilegi | maybe for the v cycle kolla_docker.py will retire :] | 16:00 |
| mnasiadka | so, if kolla_docker.py lacks this functionality - I would go for upstream module | 16:00 |
| mgoddard | osmanlicilegi: it still has some things docker_container does not | 16:00 |
| yoctozepto | make it so that it's easy to go either way :D | 16:00 |
| mnasiadka | unless upstream module lacks some functionality, so go for extending kolla_docker.py :) | 16:00 |
| yoctozepto | looking at current shape of docker | 16:01 |
| yoctozepto | i would not be surprised it's "deprecated" in a year | 16:01 |
| mgoddard | https://docs.ansible.com/ansible/latest/modules/docker_image_module.html | 16:01 |
| mgoddard | I can't see prune on there | 16:01 |
| yoctozepto | probably getting a fork and more momentum toward podman and alike | 16:01 |
| mnasiadka | yoctozepto: and what all those Docker Desktop users will do? :) | 16:01 |
| yoctozepto | mnasiadka: cry just a little | 16:01 |
| osmanlicilegi | mgoddard: https://docs.ansible.com/ansible/latest/modules/docker_prune_module.html | 16:02 |
| mnasiadka | yoctozepto: Windows users cry louder than others :) | 16:02 |
| yoctozepto | https://www.youtube.com/watch?v=ZBpWdwXzpMk | 16:02 |
| mgoddard | osmanlicilegi: oh, nice. Use that. We'll need to bump ansible minimum to 2.8, but I think we planned to do that in U anyway | 16:02 |
| osmanlicilegi | ok sir :] | 16:02 |
| mnasiadka | yes, bump up Ansible :) | 16:03 |
| yoctozepto | yes, bump as far as eyes can see | 16:03 |
| yoctozepto | we are past 5pm | 16:03 |
| mgoddard | ok, we're at time | 16:03 |
| yoctozepto | let's wrap it up | 16:03 |
| mgoddard | Thank you all | 16:04 |
| mgoddard | #endmeeting | 16:04 |
| *** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:04 | |
| openstack | Meeting ended Wed Dec 11 16:04:12 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:04 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-12-11-15.01.html | 16:04 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-12-11-15.01.txt | 16:04 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-12-11-15.01.log.html | 16:04 |
| yoctozepto | thanks mgoddard | 16:04 |
| yoctozepto | mgoddard, mnasiadka: hrw was right, rdo had meeting one hour before us :D | 16:04 |
| hrw | elo again | 16:04 |
| yoctozepto | weird approach to tz | 16:04 |
| mgoddard | yes, I remember him reading logs last week | 16:05 |
| mnasiadka | yoctozepto: maybe it was at the same time before time change in PL :) | 16:05 |
| hrw | speaking of docker... can we just say "19.03 is minimum supported, deal with it"? | 16:05 |
| *** negronjl has quit IRC | 16:05 | |
| hrw | and not as 'in some release in future' but today/tomorrow/notlater | 16:06 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Drop python 2 support from action plugins https://review.opendev.org/696063 | 16:07 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Get rid of Python 2 support https://review.opendev.org/695168 | 16:07 |
| *** negronjl has joined #openstack-kolla | 16:08 | |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: WIP: Adds support for Ansible 2.8, bump mimimum to 2.6 https://review.opendev.org/697702 | 16:10 |
| *** scottsol has quit IRC | 16:10 | |
| mgoddard | hrw: I don't think we have any dependencies on that version specifically | 16:10 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: OVN Support https://review.opendev.org/696841 | 16:11 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Use OpenStack Train release https://review.opendev.org/694616 | 16:11 |
| hrw | mgoddard: according to docker devs we need 19.03.4+ to have ver working with kernels which have overlayfs fix (so 4.19/debian and all stable branches) | 16:12 |
| hrw | mgoddard: anyway we should define one version and keep it to cut all those discussions 'will it work on someone's 1.12 docker' | 16:12 |
| watersj | bridge_mappings = physnet1:br-ex, physnet2:br-ex2, what creates the ovs bridge device? Do I need to enter the neutron_openvswitch_agent and create br-ex2 1st? | 16:12 |
| hrw | 1.12 is several years old one | 16:12 |
| hrw | k-a installs docker-ce in bootstrap-servers | 16:13 |
| mnasiadka | yeah, there are overlay2 fixes in 19.03 for newer kernels, might help us just point to that version | 16:14 |
| yoctozepto | yay | 16:14 |
| mgoddard | we don't require users to run bootstrap-servers | 16:15 |
| yoctozepto | mnasiadka: not sure, they clam it's 15 UTC | 16:15 |
| hrw | mgoddard: but we require them to run docker? | 16:15 |
| mnasiadka | mgoddard: that's another thing, we need to manage the deps somehow :) | 16:16 |
| mnasiadka | for a sane approach, now if somebody comes in with Docker 1.0 he can expect it to work :) | 16:16 |
| *** dougsz has quit IRC | 16:16 | |
| mgoddard | I'm up for a minimum, but not necessarily too new | 16:16 |
| hrw | mgoddard: 18.06, 18.09 or 19.03? there are no other version available. | 16:17 |
| yoctozepto | 19.03 is not too new | 16:18 |
| yoctozepto | and expect latest patch | 16:18 |
| hrw | osmanlicilegi: do you plan any rabbitmq patch chcanges today? I would like to change linaro repos there to use https (and only that) | 16:19 |
| *** dougsz has joined #openstack-kolla | 16:21 | |
| *** k_mouza has quit IRC | 16:25 | |
| *** k_mouza_ has joined #openstack-kolla | 16:25 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: WIP: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 16:28 |
| hrw | osmanlicilegi: only Linaro repos changed | 16:28 |
| *** k_mouza has joined #openstack-kolla | 16:30 | |
| *** k_mouza_ has quit IRC | 16:32 | |
| openstackgerrit | Merged openstack/kolla master: skydive: mark as unbuildable on !x86 https://review.opendev.org/698430 | 16:41 |
| noxoid | well this is scary https://bugs.launchpad.net/keystone/+bug/1855080 | 16:42 |
| openstack | Launchpad bug 1855080 in OpenStack Security Advisory "Credentials API allows listing and retrieving of all users credentials" [High,Confirmed] - Assigned to Gage Hugo (gagehugo) | 16:42 |
| *** ricolin has quit IRC | 16:46 | |
| mgoddard | that is quite ridiculous | 16:55 |
| noxoid | right? i just confirmed any authorized user on my kolla stein deployment can see everyone elses ec2 (s3) credentials | 16:55 |
| yoctozepto | mgoddard, mnasiadka: https://review.opendev.org/696047 | 16:59 |
| patchbot | patch 696047 - kolla-ansible - Ensure hostname resolved by kolla configured /etc/... - 2 patch sets | 16:59 |
| yoctozepto | new comment from me | 16:59 |
| yoctozepto | noxoid: sh*t | 17:00 |
| noxoid | i just shoved out a change to set enforce_scope=true and im having people test it | 17:00 |
| noxoid | at first glance it fixes the problem but kolla was reporting 403s when trying to call list_services | 17:01 |
| *** wuchunyang has joined #openstack-kolla | 17:03 | |
| yoctozepto | mgoddard, mnasiadka: I see no other option than to either have idempotency and scalability issues, or completely override /etc/hosts | 17:06 |
| openstackgerrit | Merged openstack/kayobe master: Add kayobe as openstack project for release notes https://review.opendev.org/697701 | 17:07 |
| *** wuchunyang has quit IRC | 17:08 | |
| mnasiadka | yoctozepto: there is third one - failed_when :) | 17:09 |
| yoctozepto | mnasiadka: when what? | 17:09 |
| yoctozepto | rmq already fails :-) | 17:09 |
| mnasiadka | When hostname is in the same line with 127.0.* | 17:10 |
| yoctozepto | ideally we should have consenting adults who use proper DNS | 17:10 |
| yoctozepto | mnasiadka: it's not about localhost | 17:10 |
| yoctozepto | and/or loopback | 17:11 |
| yoctozepto | see http://paste.openstack.org/show/787463/ | 17:11 |
| yoctozepto | that's mess from maas | 17:11 |
| yoctozepto | 172.16.103.100 ctrl1.lab3.stack ctrl1 | 17:11 |
| yoctozepto | 172.16.103.100 controller1.lab3.stack controller1 | 17:11 |
| yoctozepto | must fail | 17:11 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Support internal API TLS encryption https://review.opendev.org/696371 | 17:13 |
| *** rpittau is now known as rpittau|afk | 17:13 | |
| *** wuchunyang has joined #openstack-kolla | 17:14 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Use mariabackup for database backups https://review.opendev.org/696129 | 17:17 |
| *** wuchunyang has quit IRC | 17:19 | |
| *** yan0s has quit IRC | 17:26 | |
| osmanlicilegi | hrw: I was on the way to home. thanks for the update! | 17:32 |
| mnasiadka | yoctozepto: then we will always hit that... either check for weirdness and fail or template out | 17:33 |
| mgoddard | mnasiadka: true, we could do the getent check for some/all hosts to check that our change applied correctly | 17:35 |
| mnasiadka | mgoddard: there you go - we have a solution :) | 17:39 |
| *** wuchunyang has joined #openstack-kolla | 17:40 | |
| *** igordc has joined #openstack-kolla | 17:42 | |
| *** wuchunyang has quit IRC | 17:44 | |
| *** k_mouza has quit IRC | 17:45 | |
| *** k_mouza has joined #openstack-kolla | 17:48 | |
| *** k_mouza has quit IRC | 17:49 | |
| *** wuchunyang has joined #openstack-kolla | 17:50 | |
| yoctozepto | hmm, that makes sense | 17:52 |
| *** dougsz has quit IRC | 17:53 | |
| yoctozepto | prechecks for rabbitmq would make most sense? | 17:53 |
| yoctozepto | mgoddard, mnasiadka: ^ | 17:53 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Remove fernet_token_expiry precheck https://review.opendev.org/698441 | 17:54 |
| mnasiadka | yoctozepto: might be it makes more sense there | 17:54 |
| yoctozepto | I felt like we had one already | 17:55 |
| yoctozepto | - name: Check if all rabbit hostnames are resolvable | 17:55 |
| *** wuchunyang has quit IRC | 17:55 | |
| yoctozepto | hah, it checks only resolvability, not its correctness lol | 17:55 |
| *** gfidente has quit IRC | 17:56 | |
| *** wuchunyang has joined #openstack-kolla | 17:59 | |
| mgoddard | yoctozepto: you sure? | 18:00 |
| yoctozepto | mgoddard: yeah, it doesn't check the sacred address -> hostname mapping | 18:01 |
| yoctozepto | only hostname -> address which is also relevant but not enough | 18:01 |
| yoctozepto | for rabbitmq and other stuff in erlang | 18:01 |
| *** wuchunyang has quit IRC | 18:04 | |
| *** wuchunyang has joined #openstack-kolla | 18:10 | |
| *** igordc has quit IRC | 18:13 | |
| *** igordc has joined #openstack-kolla | 18:14 | |
| yoctozepto | odd, I could not reproduce breakage with the linked paste, very weird | 18:14 |
| *** wuchunyang has quit IRC | 18:16 | |
| yoctozepto | mnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/1785569 your rmq bug - is this still relevant? | 18:22 |
| openstack | Launchpad bug 1785569 in kolla-ansible "Scale out of rabbitmq is broken" [Undecided,New] | 18:22 |
| mnasiadka | yoctozepto: I think we stopped using clusterer long time ago, but I don’t know if that got fixed by that :) | 18:24 |
| *** igordc has quit IRC | 18:25 | |
| yoctozepto | mnasiadka: I think it deserves invalidation anyway | 18:25 |
| *** rgogunskiy has joined #openstack-kolla | 18:28 | |
| *** rgogunskiy has quit IRC | 18:32 | |
| *** k_mouza has joined #openstack-kolla | 18:39 | |
| *** k_mouza has quit IRC | 18:43 | |
| *** henriqueof has joined #openstack-kolla | 18:50 | |
| *** murphyslawbbs has joined #openstack-kolla | 18:53 | |
| *** wuchunyang has joined #openstack-kolla | 18:56 | |
| *** wuchunyang has quit IRC | 19:05 | |
| mnasiadka | yoctozepto: yup | 19:11 |
| mnasiadka | yoctozepto: seems manila works when using ceph Nautilus (when ceph client is Nautilus as well) - https://zuul.opendev.org/t/openstack/build/8609f6cf17134c34852f3cb00519aa7f/log/primary/logs/ansible/test-ceph | 19:13 |
| *** wuchunyang has joined #openstack-kolla | 19:13 | |
| *** spsurya has quit IRC | 19:14 | |
| *** hjensas has joined #openstack-kolla | 19:16 | |
| *** wuchunyang has quit IRC | 19:17 | |
| yoctozepto | hmm, why does not it elsewhere? | 19:21 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add Ceph-Ansible jobs https://review.opendev.org/676376 | 19:21 |
| mnasiadka | yoctozepto: maybe it's related to shares created before upgrade? | 19:22 |
| mnasiadka | 189 patch sets... I hope I won't hit 200 :D | 19:22 |
| yoctozepto | hmm, interesting thought | 19:22 |
| yoctozepto | Why not | 19:23 |
| yoctozepto | You will bump up infra's stats | 19:23 |
| *** wuchunyang has joined #openstack-kolla | 19:24 | |
| *** goldyfruit_ has joined #openstack-kolla | 19:26 | |
| *** wuchunyang has quit IRC | 19:29 | |
| *** wuchunyang has joined #openstack-kolla | 19:33 | |
| *** wuchunyang has quit IRC | 19:37 | |
| *** wuchunyang has joined #openstack-kolla | 19:39 | |
| *** wuchunyang has quit IRC | 19:43 | |
| *** wuchunyang has joined #openstack-kolla | 19:48 | |
| *** wuchunyang has quit IRC | 19:55 | |
| *** murphyslawbbs has quit IRC | 20:20 | |
| *** wuchunyang has joined #openstack-kolla | 20:20 | |
| *** murphyslawbbs has joined #openstack-kolla | 20:25 | |
| *** murphyslawbbs has quit IRC | 20:26 | |
| noxoid | ah i think i see why i got 403's after enabling enforce_scope. auth config needs system_scope:all | 20:27 |
| *** wuchunyang has quit IRC | 20:32 | |
| *** wuchunyang has joined #openstack-kolla | 20:39 | |
| *** born2bake has joined #openstack-kolla | 20:41 | |
| *** wuchunyang has quit IRC | 20:44 | |
| *** wuchunyang has joined #openstack-kolla | 20:44 | |
| *** born2bake has quit IRC | 20:47 | |
| *** wuchunyang has quit IRC | 20:52 | |
| *** wuchunyang has joined #openstack-kolla | 20:56 | |
| *** wuchunyang has quit IRC | 21:00 | |
| *** born2bake has joined #openstack-kolla | 21:01 | |
| born2bake | guys when configuring external ceph... cinder_rbd_secret_uuid or rbd_secret_uuid shall be used in cinder-volumes.conf file? P.S. Ubuntu/Stein/source | 21:01 |
| *** wuchunyang has joined #openstack-kolla | 21:24 | |
| yoctozepto | cinder one | 21:28 |
| *** wuchunyang has quit IRC | 21:38 | |
| *** pcaruana has quit IRC | 21:38 | |
| *** vmixor has quit IRC | 21:41 | |
| *** wuchunyang has joined #openstack-kolla | 21:44 | |
| born2bake | okay, updated my bug report. Also, due to the fact that magnum and octavia are not working for me, is there anyone who tried to deploy rancher with openstack cloud provider connected? | 21:48 |
| watersj | how ans when are ovs bridges created? At container bootstrapping? | 21:50 |
| *** k_mouza has joined #openstack-kolla | 21:50 | |
| *** wuchunyang has quit IRC | 21:51 | |
| *** henriqueof has quit IRC | 21:52 | |
| *** henriqueof has joined #openstack-kolla | 21:52 | |
| *** graysonh has quit IRC | 21:54 | |
| *** k_mouza has quit IRC | 21:55 | |
| *** wuchunyang has joined #openstack-kolla | 21:58 | |
| *** watersj has quit IRC | 21:59 | |
| *** wuchunyang has quit IRC | 22:02 | |
| *** wuchunyang has joined #openstack-kolla | 22:13 | |
| *** wuchunyang has quit IRC | 22:18 | |
| *** wuchunyang has joined #openstack-kolla | 22:23 | |
| *** wuchunyang has quit IRC | 22:25 | |
| *** wuchunyang has joined #openstack-kolla | 22:30 | |
| *** wuchunyang has quit IRC | 22:35 | |
| *** wuchunyang has joined #openstack-kolla | 22:36 | |
| *** wuchunyang has quit IRC | 22:43 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: WIP: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 22:49 |
| *** wuchunyang has joined #openstack-kolla | 23:05 | |
| *** schwicht has quit IRC | 23:09 | |
| *** wuchunyang has quit IRC | 23:13 | |
| *** wuchunyang has joined #openstack-kolla | 23:17 | |
| *** wuchunyang has quit IRC | 23:25 | |
| *** wuchunyang has joined #openstack-kolla | 23:26 | |
| *** wuchunyang has quit IRC | 23:30 | |
| *** wuchunyang has joined #openstack-kolla | 23:33 | |
| *** schwicht has joined #openstack-kolla | 23:39 | |
| *** wuchunyang has quit IRC | 23:42 | |
| *** schwicht has quit IRC | 23:46 | |
| *** born2bake has quit IRC | 23:49 | |
| *** schwicht has joined #openstack-kolla | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!