| *** AjiArya has quit IRC | 00:04 | |
| *** mcnugit has joined #openstack-kolla | 00:34 | |
| *** mcnugit has quit IRC | 01:21 | |
| *** watersj has quit IRC | 01:53 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image https://review.opendev.org/708273 | 02:13 |
|---|---|---|
| *** ricolin has quit IRC | 02:25 | |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 | 02:34 |
| *** Tony31 has joined #openstack-kolla | 02:38 | |
| *** igordc has quit IRC | 02:45 | |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:Add support for encrypting glance/heat api https://review.opendev.org/707131 | 02:52 |
| *** openstackstatus has joined #openstack-kolla | 02:56 | |
| *** ChanServ sets mode: +v openstackstatus | 02:56 | |
| *** ricolin has joined #openstack-kolla | 03:02 | |
| *** AjiArya has joined #openstack-kolla | 03:39 | |
| AjiArya | Hello, I'm having issue while creating an instance. It's stuck on spawning state | 03:41 |
| AjiArya | Here's the log: http://paste.openstack.org/show/789775/ | 03:42 |
| *** evrardjp has quit IRC | 03:42 | |
| AjiArya | And also this | 03:43 |
| AjiArya | [root@maas-node0 ~]# tailf -n 40 /var/log/kolla/nova/nova-conductor.log 2020-02-20 02:22:43.934 7 INFO oslo_service.service [-] Starting 4 workers2020-02-20 02:22:43.949 29 INFO nova.service [-] Starting conductor node (version 20.1.0)2020-02-20 02:22:43.957 30 INFO nova.service [-] Starting conductor node (version 20.1.0)2020-02-20 02:22:43.959 31 | 03:43 |
| AjiArya | INFO nova.service [-] Starting conductor node (version 20.1.0)2020-02-20 02:22:43.987 32 INFO nova.service [-] Starting conductor node (version 20.1.0)2020-02-20 03:37:31.305 31 ERROR nova.scheduler.utils [req-f3d08764-49d0-4e51-a63e-3eb9153be15a e94c28caa3794020a0f956a79f74a4d1 2e3282a986bd475e877ead0d52314d36 - default default] [instance: | 03:43 |
| AjiArya | eb61ceb9-f899-48f6-a01e-4da3038a9eab] Error from last host: maas-node0.hujan.io (node maas-node0.hujan.io): [u'Traceback (most recent call last):\n', u' File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2143, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File | 03:43 |
| AjiArya | "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2519, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance eb61ceb9-f899-48f6-a01e-4da3038a9eab was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm | 03:43 |
| AjiArya | \n']2020-02-20 03:37:34.249 31 INFO nova.compute.rpcapi [req-df582a54-0c3a-406c-b039-494069f69a74 - - - - -] Automatically selected compute RPC version 5.3 from minimum service version 40 | 03:43 |
| AjiArya | Oops sorry. http://paste.openstack.org/show/789776/ | 03:44 |
| *** evrardjp has joined #openstack-kolla | 03:45 | |
| AjiArya | I'm using multinode deployment | 04:00 |
| *** Tony31 has quit IRC | 04:10 | |
| emccormick | Sounds like RabbitMQ hell | 04:41 |
| *** dave-mccowan has joined #openstack-kolla | 04:41 | |
| emccormick | or something else disguised as RabbitMQ hell | 04:42 |
| emccormick | messed up networking to your storage backend can present itself that way sometimes | 04:42 |
| emccormick | AjiArya: Hmm actually now see your pastebin | 04:48 |
| emccormick | Are you actually running on a KVM-based host? Not doing nested virtualization or anything like that? | 04:48 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:Add support for encrypting glance/heat api https://review.opendev.org/707131 | 05:34 |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-kolla | 05:34 | |
| *** dave-mccowan has quit IRC | 05:47 | |
| *** mcnugit has joined #openstack-kolla | 06:21 | |
| AjiArya | I'm doing nested virtualization | 06:28 |
| AjiArya | I'm running my node using KVM-based host | 06:30 |
| AjiArya | nested virtualization featured has been activated | 06:31 |
| *** priteau has joined #openstack-kolla | 06:31 | |
| *** mlukas has joined #openstack-kolla | 06:40 | |
| *** zhanglong has joined #openstack-kolla | 06:51 | |
| *** k_mouza has joined #openstack-kolla | 06:52 | |
| *** k_mouza has quit IRC | 06:56 | |
| openstackgerrit | Yongjun Bai proposed openstack/kolla master: Add Apache packages to neutron server containers https://review.opendev.org/708810 | 07:06 |
| *** cah_link has joined #openstack-kolla | 07:09 | |
| *** ivve has quit IRC | 07:19 | |
| yoctozepto | morning | 07:33 |
| *** priteau has quit IRC | 07:34 | |
| *** zhanglong has quit IRC | 07:37 | |
| *** zhanglong has joined #openstack-kolla | 07:38 | |
| *** rgogunskiy has joined #openstack-kolla | 07:44 | |
| *** zhanglong has quit IRC | 07:47 | |
| mnasiadka | morning | 07:48 |
| *** zhanglong has joined #openstack-kolla | 07:48 | |
| mnasiadka | yoctozepto, mgoddard: thanks for reviewing OVN change, now I have some stuff to do there ;) | 07:48 |
| *** bengates has joined #openstack-kolla | 07:49 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 | 07:49 |
| *** jbadiapa has joined #openstack-kolla | 07:52 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 07:54 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla master: Add Apache httpd setup to heat-api-cfn https://review.opendev.org/708816 | 07:59 |
| *** mcnugit has quit IRC | 08:05 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Change /run bind mount for neutron/openvswitch https://review.opendev.org/707375 | 08:09 |
| *** tonythomas has joined #openstack-kolla | 08:10 | |
| yoctozepto | mnasiadka: maybe debian for linuxbridge? | 08:21 |
| mnasiadka | yoctozepto: seems ebtables broute support is not going to be soon in centos8/rhel8 - so it will be broken | 08:22 |
| mnasiadka | So we could comment that job out and mark with FIXME, and add debian | 08:22 |
| yoctozepto | mnasiadka: agreed | 08:22 |
| mnasiadka | yoctozepto: RHOSP does not support linuxbridge at all, so they don't really care | 08:22 |
| yoctozepto | mnasiadka: well, linuxbridge is not really future | 08:23 |
| yoctozepto | mnasiadka: ovs, ovn, mix up dpdk... you get the point | 08:23 |
| yoctozepto | vs linux bridge :-) | 08:23 |
| mnasiadka | yoctozepto: but there are some people using it because of it's simplicity | 08:23 |
| yoctozepto | mnasiadka: yeah, but it does not kill one to learn a bit about ovs ;-) anyways, if platform does not support it because no, then we can't be bothered anymore | 08:24 |
| mnasiadka | yoctozepto: and there are claims (rightful ones) about LB being more performant than OVS (but that's obvious with the number of features OVS has) | 08:24 |
| kevinz | Hi Kolla, we are testing Rocky to Stein upgrade, and meet a weird issue:https://bugs.launchpad.net/kolla-ansible/+bug/1863982 | 08:25 |
| openstack | Launchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] | 08:25 |
| yoctozepto | mnasiadka: well, otoh you have proofs that ovs+dpdk beats lb in small packet scale | 08:26 |
| *** rpittau|afk is now known as rpittau | 08:26 | |
| mnasiadka | if you have cpus you can spare for dpdk permanent polling - sure | 08:26 |
| kevinz | After upgrade, router ns disappear, I suspect the issue is related with l3-agent. Do anyone meet the issue before? | 08:26 |
| yoctozepto | mnasiadka: indeed that's the case they miss :-) | 08:26 |
| mnasiadka | kevinz: well, router should recreate namespaces on restart - might need to run neutron-ovs-cleanup script (or named somehow similar) and then restart router | 08:27 |
| yoctozepto | kevinz: hmm, now that you mention it, I believe ppl said they had to restart l3 agent after upgrades | 08:27 |
| yoctozepto | there should be a bug report about that as well | 08:27 |
| yoctozepto | someone probably did not describe it in terms of namespaces, but lack of functionality :-) | 08:28 |
| mnasiadka | yoctozepto: but it's getting a lot better (I mean standard OVS performance) - on recent kernels | 08:28 |
| mnasiadka | yoctozepto: there was a bug report about us not running neutron-ovs-cleanup script on shutdown | 08:28 |
| mnasiadka | yoctozepto: and one of my team mates raised it like 3-4 years ago :D | 08:28 |
| yoctozepto | mnasiadka: could be as well | 08:28 |
| yoctozepto | mnasiadka: oh well | 08:28 |
| yoctozepto | mnasiadka: anyways, we work on networking this cycle, so might be nice to look at it again | 08:29 |
| kevinz | oooh, what an old bug.. | 08:29 |
| mnasiadka | yoctozepto: well, question how to run a script on docker stop/kill signal :) | 08:30 |
| mnasiadka | or run it before we start router | 08:30 |
| mnasiadka | that should be easier | 08:30 |
| yoctozepto | mnasiadka: exactly, but I wonder why 2nd restart of router by itself works | 08:30 |
| yoctozepto | mnasiadka: just not our automatic | 08:30 |
| mnasiadka | yoctozepto: https://bugs.launchpad.net/kolla/+bug/1617188 | 08:30 |
| openstack | Launchpad bug 1617188 in kolla "neutron-ovs-cleanup missing" [Undecided,Expired] | 08:30 |
| yoctozepto | mnasiadka: folks did not need to clean | 08:30 |
| mnasiadka | yoctozepto: we did, but we had like hundreds of routers :D | 08:31 |
| mnasiadka | not working for that company, nor the customer anymore - and I don't think they still use OpenStack | 08:32 |
| yoctozepto | mnasiadka: oh well, I mean current issues | 08:32 |
| yoctozepto | mnasiadka: so it might be something else | 08:32 |
| yoctozepto | mnasiadka: or maybe it tries to clean itself up in some case and not the other | 08:32 |
| mnasiadka | god only knows | 08:33 |
| osmanlicilegi | morning | 08:34 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 08:35 |
| mnasiadka | yoctozepto: debian added, centos8 removed | 08:36 |
| mnasiadka | well, centos7 will be removed soon, but for now it can be there | 08:36 |
| yoctozepto | mnasiadka: yeah, we'll remove it all at once | 08:37 |
| mnasiadka | yoctozepto: ok, ceph stuff solved itself :) | 08:39 |
| mnasiadka | (see ML) | 08:39 |
| yoctozepto | mnasiadka: nothing yet | 08:40 |
| kevinz | yoctozepto mnasiadka: now the upgrade procedure is: restart neutron-server -> neutron-ovs-agent -> neutron-dhcp-agent -> neutron-l3-agent -> neutron-metadata-agent. we should insert the ovs-cleanup before restart neutron-server? | 08:41 |
| yoctozepto | mnasiadka: haha, feeling bad for forgetting | 08:41 |
| yoctozepto | mnasiadka: but I had this feeling that it worked | 08:41 |
| *** gfidente has joined #openstack-kolla | 08:41 | |
| kevinz | that means we do ovs-cleanup at the "old" neutron-server container? | 08:42 |
| yoctozepto | kevinz: it's for ovs, so after stopping all agent services on the ovs node | 08:42 |
| mnasiadka | kevinz: it doesn't matter if before or after - but namespaces are created/managed by ovs/dhcp/l3 agents | 08:42 |
| yoctozepto | mnasiadka: I believe it is wrong to clean when it's running though | 08:44 |
| mnasiadka | yoctozepto: for sure | 08:44 |
| mnasiadka | yoctozepto: we should be rather first stopping everything on a host, running cleanup, then starting | 08:44 |
| mnasiadka | if it helps in anything | 08:44 |
| mnasiadka | I mean if the cleanup helps | 08:44 |
| yoctozepto | mnasiadka: indeed "if" | 08:45 |
| yoctozepto | mnasiadka: and thanks for rephrasing myself :-) | 08:45 |
| kevinz | well, sure, I will try to see if this helps:-) | 08:45 |
| yoctozepto | mnasiadka: oopsie, you lost one file in lb zuul job | 08:45 |
| yoctozepto | mnasiadka: that's what you get for editing and not checking out locally | 08:46 |
| mnasiadka | argh, right | 08:46 |
| mnasiadka | I wanted to be quick | 08:46 |
| mnasiadka | :) | 08:46 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 08:47 |
| *** dswebb has joined #openstack-kolla | 08:51 | |
| *** ivve has joined #openstack-kolla | 08:53 | |
| *** mlukas has quit IRC | 08:54 | |
| *** dougsz has joined #openstack-kolla | 08:59 | |
| mnasiadka | kevinz: well, just spoke with the Neutron PTL - other deployment tools run ovn-cleanup on host reboots, not the upgrade - it shouldn't be that a router namespace just got lost in space | 08:59 |
| mnasiadka | yoctozepto: ^^ | 09:00 |
| yoctozepto | mnasiadka: makes sensse | 09:03 |
| *** jaicaa has quit IRC | 09:05 | |
| mnasiadka | ovs-cleanup, not ovn-cleanup - but I'm in a different world :D | 09:05 |
| *** jaicaa has joined #openstack-kolla | 09:07 | |
| kevinz | mnasiadka, thx, so we don't need to run ovs-cleanup when upgrade? | 09:09 |
| mnasiadka | kevinz: no, it would be good if you would raise a bug on neutron side, describe what you did and what went wrong. | 09:09 |
| mnasiadka | you can also mark kolla-ansible in that bug report, so we can collaborate if needed | 09:10 |
| kevinz | sure, I will raise now | 09:10 |
| *** carli has joined #openstack-kolla | 09:10 | |
| kevinz | this issue is more interested.. | 09:11 |
| *** k_mouza has joined #openstack-kolla | 09:16 | |
| *** zhanglong has quit IRC | 09:19 | |
| *** priteau has joined #openstack-kolla | 09:20 | |
| *** zhanglong has joined #openstack-kolla | 09:20 | |
| paddyoneill|afk | good morning | 09:22 |
| *** paddyoneill|afk is now known as paddyoneill | 09:22 | |
| mgoddard | morning | 09:23 |
| paddyoneill | i am working on integration gpfs into kolla-ansible as a cinder backend using the GPFSRemoteDriver | 09:23 |
| paddyoneill | as it works well with cinder being in a container. Only bit I have left to figure out is how to add a ssh key to the cinder container so the cinder user can ssh to the gpfs cluster to access the gpfs commands | 09:24 |
| paddyoneill | currently I manually copy the key file in, but I assume there is a way to handle this in kolla-ansible, but I can't find the relavent role / task. Any pointers? | 09:25 |
| dswebb | hi all, any idea on a timeline for when nova 18.2.4 images will be built? | 09:27 |
| dswebb | (just asking in regards to: https://nvd.nist.gov/vuln/detail/CVE-2015-9543) | 09:28 |
| *** k_mouza_ has joined #openstack-kolla | 09:35 | |
| mgoddard | hi dswebb: Anyone can propose a change to versions of projects in images. Use the tools/version-check.py script on the stable/rocky branch | 09:35 |
| dswebb | ok, will do | 09:35 |
| dswebb | cheers | 09:36 |
| *** k_mouza has quit IRC | 09:38 | |
| dswebb | ah, looks like it isn't really released upstream yet anyways. | 09:59 |
| openstackgerrit | Merged openstack/kolla-ansible master: Deprecate XenAPI integrations https://review.opendev.org/708743 | 10:02 |
| openstackgerrit | Merged openstack/kolla-ansible master: Deprecate VMware integrations https://review.opendev.org/708741 | 10:03 |
| mgoddard | yoctozepto: all your renos are under ansible/ | 10:13 |
| mgoddard | was wondering why all the jobs ran | 10:13 |
| openstackgerrit | Merged openstack/kolla-ansible master: Clean up some kolla-kubernetes cruft in OVS https://review.opendev.org/708710 | 10:14 |
| yoctozepto | mgoddard: bah, revert! | 10:18 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Revert "Deprecate VMware integrations" https://review.opendev.org/708839 | 10:19 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Revert "Deprecate XenAPI integrations" https://review.opendev.org/708841 | 10:20 |
| yoctozepto | mgoddard: or move? what do you think is better? | 10:20 |
| mgoddard | yoctozepto: move | 10:20 |
| yoctozepto | mgoddard: ack, fixing | 10:20 |
| *** AjiArya has quit IRC | 10:34 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Deprecate Hyper-V integrations https://review.opendev.org/708740 | 10:37 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix reno paths of two recent renos related to deprecations https://review.opendev.org/708847 | 10:40 |
| *** zhanglong has quit IRC | 10:42 | |
| *** zhanglong has joined #openstack-kolla | 10:44 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Deprecate deployment of MongoDB https://review.opendev.org/708508 | 10:44 |
| yoctozepto | ^ we bad at stopping the mess | 10:46 |
| yoctozepto | reno could be a bit smarter | 10:46 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix reno paths of three recent renos related to deprecations https://review.opendev.org/708847 | 10:47 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 10:53 |
| yoctozepto | mnasiadka: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7b8/708318/14/check/kolla-ansible-debian-source-linuxbridge/7b8c90f/primary/logs/kolla/all-ERROR.txt xDDD | 10:55 |
| mnasiadka | yoctozepto: awesome, debian is even better | 10:59 |
| mnasiadka | yoctozepto: I guess then I'll remove debian CI job from the change :D | 11:02 |
| mnasiadka | yoctozepto: unless you want to fix it by installing additional deps in the Dockerfile :) | 11:02 |
| yoctozepto | mnasiadka: we might want to | 11:07 |
| mnasiadka | yoctozepto: I'm pretty sure it should fail the same way when running ovs... | 11:08 |
| mnasiadka | yoctozepto: https://github.com/openstack/kolla/blob/ceae02f09d1c4e67013485c7323c11f8655c9c1e/docker/neutron/neutron-base/Dockerfile.j2#L113 | 11:09 |
| mnasiadka | that's even more interesting that iptables-save is missing | 11:09 |
| mnasiadka | ah, we're missing iptables-persistent package | 11:10 |
| mnasiadka | yoctozepto: but ping sysctl looks weird :) | 11:10 |
| yoctozepto | mnasiadka: ovs "works" in the base job | 11:12 |
| yoctozepto | mnasiadka: "works" https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1ad/708318/14/check/kolla-ansible-debian-source/1ad72c3/primary/logs/kolla/all-ERROR.txt | 11:13 |
| yoctozepto | mnasiadka: so it's b0rken all around | 11:14 |
| yoctozepto | mnasiadka: I wonder how it passes checks via cli | 11:15 |
| yoctozepto | status | ERROR | 11:16 |
| yoctozepto | is ok according to our tests | 11:16 |
| yoctozepto | very nice! | 11:16 |
| yoctozepto | ah, no, that one was failed in zuul | 11:17 |
| yoctozepto | hmm https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_69f/708217/4/check/kolla-ansible-debian-source/69fd21c/primary/logs/kolla/all-ERROR.txt | 11:18 |
| *** jopdorp has joined #openstack-kolla | 11:19 | |
| jopdorp | good afternoon everyone! | 11:19 |
| *** rpittau is now known as rpittau|bbl | 11:20 | |
| jopdorp | anyone wanna help me with a horizon deployment problem? | 11:21 |
| jopdorp | it seems not to be able to find the mysql module at the bootstrapping container step | 11:21 |
| yoctozepto | mnasiadka: looks like lb broke ovs as well? It's way to weeird ;D | 11:22 |
| jopdorp | https://pastebin.com/7AvhZPgR | 11:23 |
| jopdorp | that's the error with -vvvv | 11:23 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Refactor VXLAN overlay setup https://review.opendev.org/708217 | 11:24 |
| yoctozepto | jopdorp: it could be missing, horizon does not require a database by default | 11:25 |
| yoctozepto | jopdorp: have you changed some relevant setting? | 11:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 11:26 |
| jopdorp | yoctozepto, that might be, I enable murano | 11:26 |
| jopdorp | murano deployed fine though | 11:26 |
| jopdorp | so should I explicitly enable mysql in globals? | 11:26 |
| yoctozepto | jopdorp: but murano-dashboard (or -ui or whatever it's called) probably requires installing mysql client package in the container | 11:26 |
| yoctozepto | jopdorp: no, you probably need to fix/rebuild horizon image | 11:27 |
| jopdorp | right | 11:27 |
| jopdorp | okay | 11:27 |
| jopdorp | thank for the tip! | 11:27 |
| jopdorp | thanks* | 11:27 |
| jopdorp | could a source install instead of binary be enough? | 11:28 |
| jopdorp | or should i really change the code for the horizon role | 11:29 |
| yoctozepto | jopdorp: binary for murano? ;O | 11:33 |
| yoctozepto | jopdorp: best try source | 11:33 |
| jopdorp | okay | 11:33 |
| yoctozepto | jopdorp: murano is not the latest and greatest nowadays | 11:33 |
| jopdorp | I'll set kolla_install_type to source | 11:34 |
| jopdorp | I didn't do an install with my own docker registry yet | 11:34 |
| jopdorp | if it still doesn't work, that's what I'll try. want to do that anyway for our custom horizon theme | 11:34 |
| jopdorp | yoctozepto good to know | 11:35 |
| jopdorp | I might be committing fixes for it soon then | 11:35 |
| yoctozepto | jopdorp: thanks | 11:37 |
| jopdorp | yoctozepto: what about magnum? | 11:39 |
| *** zhanglong has quit IRC | 11:42 | |
| *** zhanglong has joined #openstack-kolla | 11:44 | |
| *** cah_link has quit IRC | 11:54 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Change /run bind mount for neutron/openvswitch https://review.opendev.org/707375 | 12:01 |
| openstackgerrit | Merged openstack/kolla-ansible master: Deprecate Hyper-V integrations https://review.opendev.org/708740 | 12:01 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 12:06 |
| *** kplant has joined #openstack-kolla | 12:13 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix reno paths of three recent renos related to deprecations https://review.opendev.org/708847 | 12:15 |
| yoctozepto | oh, finally | 12:19 |
| yoctozepto | jopdorp: magnum is fine, we plan to increase its testing | 12:20 |
| yoctozepto | jopdorp: it seems (much) more regular updates than murano | 12:20 |
| kplant | do many clouds actually use murano? | 12:20 |
| yoctozepto | kplant: how would we know? ;D | 12:24 |
| yoctozepto | kplant: it seemingly did not pass the test of time | 12:24 |
| kplant | word of mouth/project popularity | 12:24 |
| yoctozepto | zero, null, nada | 12:24 |
| yoctozepto | ;D | 12:24 |
| kplant | well there you go | 12:24 |
| mnasiadka | yoctozepto: well, I guess hrw has some production deployments using Debian? :) | 12:31 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla master: Add Apache packages to neutron server containers https://review.opendev.org/708810 | 12:32 |
| *** zhanglong has quit IRC | 12:33 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 12:35 |
| yoctozepto | mnasiadka: yeah, but he is not da deploya | 12:36 |
| yoctozepto | ;/ | 12:36 |
| yoctozepto | mnasiadka: wonder why it broke ovs too | 12:37 |
| mnasiadka | yoctozepto: I think it was already broken, I don't think it's related to my patch | 12:37 |
| yoctozepto | mnasiadka: no, I mean compare PS11 and PS14 | 12:38 |
| yoctozepto | mnasiadka: adding lb job broke ovs as well (from logs) | 12:38 |
| mnasiadka | yoctozepto: only on debian? | 12:38 |
| yoctozepto | mnasiadka: you think it just broke overnight? | 12:38 |
| mnasiadka | yoctozepto: it's master, could break overnight :) | 12:39 |
| yoctozepto | mnasiadka: yeah, it's very weird | 12:39 |
| yoctozepto | mnasiadka: could indeed | 12:39 |
| yoctozepto | mnasiadka: https://6a408c9644083aa020a9-6b466cd24fe60c196767959ed246eea2.ssl.cf5.rackcdn.com/708217/5/check/kolla-ansible-debian-source/c9ab461/primary/logs/kolla/all-ERROR.txt | 12:42 |
| yoctozepto | mnasiadka: it indeed broke overnight | 12:42 |
| yoctozepto | lolz | 12:42 |
| yoctozepto | mnasiadka: so debian broke ping? | 12:44 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: Remove cruft https://review.opendev.org/708861 | 12:46 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: Remove cruft https://review.opendev.org/708861 | 12:47 |
| *** numans has quit IRC | 12:49 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: Remove cruft https://review.opendev.org/708861 | 12:49 |
| jopdorp | ormuco uses murano in production with a kolla deployment | 13:01 |
| mnasiadka | yoctozepto: I haven't used Debian for over 12 years, don't ask me | 13:01 |
| yoctozepto | kplant: then murano has at least one user :D | 13:01 |
| jopdorp | haha | 13:02 |
| yoctozepto | mnasiadka: fair point, let's leave it to hrw, we're already overloaded | 13:03 |
| *** rpittau|bbl is now known as rpittau | 13:07 | |
| kplant | yoctozepto: viva el murano! | 13:14 |
| *** abdysn has joined #openstack-kolla | 13:17 | |
| jopdorp | yocozepto: any idea how long running nova API bootstrap container should take in source mode? | 13:25 |
| jopdorp | it's been stuck there for about an hour now | 13:25 |
| jopdorp | i restarted deployment with -vvv | 13:29 |
| priteau | It might be stuck on bad database entries | 13:35 |
| priteau | jopdorp: Log into the node running it and tail -f /var/log/kolla/nova/nova-manage.log | 13:35 |
| jopdorp | i get this now "WSREP has not yet prepared node for application use" | 13:37 |
| jopdorp | "mariadb : Creating haproxy mysql user" | 13:37 |
| jopdorp | thi si a goo idea priteau> jopdorp: Log into the node running it and tail -f /var/log/kolla/nova/nova-manage.log | 13:37 |
| yoctozepto | check mariadb logs then | 13:53 |
| yoctozepto | it must have b0rked | 13:53 |
| yoctozepto | kplant: viva la vida loca (openstack) | 13:54 |
| *** kplant has quit IRC | 13:57 | |
| yoctozepto | mnasiadka, mgoddard: https://review.opendev.org/708217 approve pretty please | 14:02 |
| patchbot | patch 708217 - kolla-ansible - CI: Refactor VXLAN overlay setup - 5 patch sets | 14:02 |
| *** abdysn has quit IRC | 14:03 | |
| mgoddard | yoctozepto: done | 14:07 |
| yoctozepto | mgoddard, mnasiadka: tyvm | 14:07 |
| yoctozepto | mnasiadka: commented on lb, pep8 | 14:09 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 14:09 |
| emccormick | Does anyone know off the top of their head which version of ceph client is in Stein? | 14:15 |
| emccormick | I thought the repo was in docker/base but I don't see it in there. | 14:15 |
| mgoddard | emccormick: https://ethercalc.openstack.org/kolla-infra-service-matrix | 14:37 |
| emccormick | Awesome. Thanks Goddard! I get to skip a ceph upgrade for another cycle. Wheee | 14:39 |
| *** cah_link has joined #openstack-kolla | 14:50 | |
| *** spiette has quit IRC | 14:51 | |
| *** dave-mccowan has joined #openstack-kolla | 14:51 | |
| *** spiette has joined #openstack-kolla | 14:53 | |
| *** TrevorV has joined #openstack-kolla | 14:54 | |
| yoctozepto | emccormick: you can skip it even longer, client compat is usually quite relaxed | 14:54 |
| emccormick | Yeah, I hate that though. It's an OCD thing ;) | 14:56 |
| *** kplant has joined #openstack-kolla | 14:56 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/kolla master: Use StorageSIG repos for Ceph in CentOS8 https://review.opendev.org/707338 | 15:10 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Refactor VXLAN overlay setup https://review.opendev.org/708217 | 15:16 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 15:25 |
| *** lile has joined #openstack-kolla | 15:37 | |
| *** dswebb has quit IRC | 15:38 | |
| *** cah_link has quit IRC | 15:41 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Switch from shade to openstacksdk https://review.opendev.org/707689 | 15:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Remove activate-virtualenv and deactivate-virtualenv roles https://review.opendev.org/708072 | 15:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 15:47 |
| *** jopdorp has quit IRC | 15:58 | |
| *** TrevorV has quit IRC | 16:04 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 16:08 |
| mnasiadka | yoctozepto: https://review.opendev.org/#/c/707818/ - are we doing this? :) | 16:08 |
| patchbot | patch 707818 - kolla-ansible - CI: Fine tune Galera gmcast.peer_timeout to 15 sec... - 5 patch sets | 16:08 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance https://review.opendev.org/708114 | 16:13 |
| *** mcnugit has joined #openstack-kolla | 16:16 | |
| *** rgogunskiy has quit IRC | 16:17 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/train: Change /run bind mount for neutron/openvswitch https://review.opendev.org/708905 | 16:18 |
| yoctozepto | emccormick: welcome to the club | 16:26 |
| yoctozepto | mnasiadka: yeah, let's try this out | 16:27 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Update docs to say that external Ceph required https://review.opendev.org/708708 | 16:29 |
| *** bengates has quit IRC | 16:32 | |
| mnasiadka | yoctozepto: https://zuul.opendev.org/t/openstack/build/afc9f746a94341bab3d5a924df3493b2/log/primary/logs/kolla/cinder/cinder-volume.txt#1743 - deadlocks are something new? :) | 16:34 |
| mnasiadka | it's stable/train - https://review.opendev.org/#/c/702419/ | 16:34 |
| patchbot | patch 702419 - kolla (stable/train) - CentOS 8: work around mod_ssl certificate issue - 1 patch set | 16:34 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 16:35 |
| mgoddard | yoctozepto: care to kick off the c8 train? https://review.opendev.org/702418 https://review.opendev.org/702419 https://review.opendev.org/702420 | 16:38 |
| patchbot | patch 702418 - kolla (stable/train) - Refactor Apache httpd setup - 1 patch set | 16:38 |
| patchbot | patch 702419 - kolla (stable/train) - CentOS 8: work around mod_ssl certificate issue - 1 patch set | 16:38 |
| patchbot | patch 702420 - kolla (stable/train) - CentOS 8: Add zuul build and publish jobs - 1 patch set | 16:38 |
| mnasiadka | mgoddard: actually - when do we plan to get rid of centos7 in master? | 16:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Update docs to say that external Ceph required https://review.opendev.org/708708 | 16:39 |
| mgoddard | mnasiadka: not sure | 16:39 |
| yoctozepto | mgoddard: ok, queued | 16:39 |
| mgoddard | mnasiadka: it's quite useful for backports | 16:40 |
| mnasiadka | mgoddard: because then I'll need to tidy up OVN images - networking-ovn is no longer needed, it has been merged to neutron code base. | 16:40 |
| mgoddard | and would be nice to develop the migration job in master then backport | 16:40 |
| mgoddard | mnasiadka: you could just support centos8? | 16:40 |
| *** ivve has quit IRC | 16:41 | |
| mgoddard | mnasiadka: if you wrap it in 'if distro_package_manager' we will clean it up at the end with everything else | 16:41 |
| mnasiadka | mgoddard: well, I could propose a change to only remove networking-ovn when python3 | 16:41 |
| mnasiadka | and then it touches centos8 and ubuntu and debian - and we're good | 16:41 |
| mgoddard | I suppose it might be tricky for source images though | 16:42 |
| mnasiadka | I'll check it | 16:43 |
| yoctozepto | just drop c7 soon | 16:47 |
| yoctozepto | we already have c8 in master so we are not breaking centos future now | 16:47 |
| mgoddard | we don't have full coverage with c8 yet | 16:48 |
| yoctozepto | mgoddard: I don't think that prevents us from dropping c7, the coverage is good enough imho | 16:49 |
| yoctozepto | mgoddard: https://review.opendev.org/708082 please merge | 16:49 |
| patchbot | patch 708082 - kolla-ansible - CI: Remove dbus from bindep and playbooks - 3 patch sets | 16:49 |
| mgoddard | it doesn't | 16:49 |
| yoctozepto | I want to rebase revert but it will conflict with this one anyway, so need to base it off this | 16:49 |
| yoctozepto | geez, zuul | 16:50 |
| yoctozepto | so need to rebase both, great | 16:50 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Remove dbus from bindep and playbooks https://review.opendev.org/708082 | 16:50 |
| mgoddard | kayobe is still using c7 | 16:51 |
| yoctozepto | mgoddard: ah :< | 16:51 |
| mgoddard | getting there | 16:51 |
| *** TrevorV has joined #openstack-kolla | 16:51 | |
| mgoddard | also c7 is our only upgrade test | 16:51 |
| yoctozepto | mgoddard: or can you use k-a for train? | 16:52 |
| mgoddard | could. would rather not | 16:52 |
| yoctozepto | ok | 16:52 |
| yoctozepto | understandable | 16:52 |
| *** rpittau is now known as rpittau|afk | 16:53 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: CI: Make provision-net external https://review.opendev.org/708915 | 16:56 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: CI: Make provision-net external https://review.opendev.org/708915 | 16:58 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Revert "Upgrade virtualenv in pre" https://review.opendev.org/707622 | 16:59 |
| yoctozepto | mgoddard, mnasiadka: ^ merge before it rots | 17:00 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances https://review.opendev.org/707604 | 17:01 |
| yoctozepto | mnasiadka: that lock is interesting, it happens from time to time | 17:02 |
| yoctozepto | mnasiadka: if it becomes pain like wsrep, we report it up | 17:02 |
| yoctozepto | mnasiadka: looks random enough though | 17:02 |
| yoctozepto | hmm, ovs in c8 wants to use rdma: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88e/707818/5/check/kolla-ansible-centos8-source-cells/88e3714/primary/logs/kolla/all-ERROR.txt | 17:04 |
| mnasiadka | good for him! | 17:07 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update docs to say that external Ceph required https://review.opendev.org/708708 | 17:08 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3 https://review.opendev.org/635586 | 17:10 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 17:10 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 17:10 |
| yoctozepto | mnasiadka: rejected finetuning due to commit description | 17:10 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 17:10 |
| yoctozepto | mnasiadka: we got it all after deployment succeeded | 17:10 |
| yoctozepto | mnasiadka: or you mean deployment of services | 17:11 |
| yoctozepto | as in bootstrapping | 17:11 |
| yoctozepto | could clarify | 17:11 |
| yoctozepto | mariadb always finished deploying before failure | 17:11 |
| mnasiadka | ok, later - now digging through ovn review, too many files to alter :D | 17:11 |
| yoctozepto | mnasiadka: no problem | 17:11 |
| mnasiadka | mgoddard: any idea how to run it on ovn-controller group, but if it's a compute - check if computes_need_external_bridge? :) https://review.opendev.org/#/c/696841/92/ansible/roles/ovn/tasks/bootstrap.yml@35 | 17:26 |
| patchbot | patch 696841 - kolla-ansible - OVN Support - 92 patch sets | 17:26 |
| mgoddard | mnasiadka: tricky one. I thought of two options | 17:27 |
| mgoddard | 1. use 2 groups - ovn-controller-compute, ovn-controller-network | 17:27 |
| mgoddard | 2. use a variable with a default similar to what you have, then at least it can be tweaked if necessary | 17:28 |
| mgoddard | for 1 you'd need to use host_in_groups rather than group | 17:28 |
| mgoddard | does OVN have a DVR concept? | 17:28 |
| *** evrardjp has quit IRC | 17:34 | |
| *** carli has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-kolla | 17:35 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 17:36 |
| mnasiadka | mgoddard: yeah, in theory by default - but if you don't have external bridge on computes - you get DVR on geneve networks, an centralised routing on DNAT/SNAT/routing from geneve to vlan/flat | 17:37 |
| *** dking_desktop has joined #openstack-kolla | 17:38 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 17:40 |
| mnasiadka | let's see how miserably it fails on centos8 :) | 17:40 |
| *** k_mouza_ has quit IRC | 17:42 | |
| mnasiadka | mgoddard: the insecure stuff - it's just to make it work without ssl (which is default), I'll followup with an ssl enablement change and rework this. | 17:42 |
| mgoddard | cool | 17:42 |
| *** ivve has joined #openstack-kolla | 17:44 | |
| *** mcnugit has quit IRC | 17:47 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 17:51 |
| *** gfidente has quit IRC | 17:52 | |
| openstackgerrit | Merged openstack/kolla stable/train: Refactor Apache httpd setup https://review.opendev.org/702418 | 18:28 |
| *** igordc has joined #openstack-kolla | 18:34 | |
| yoctozepto | mgoddard, mnasiadka: am I right that kayobe-config(-dev) is just config included in kayobe but cut off at release points? | 18:39 |
| yoctozepto | mgoddard, mnasiadka: and is -dev variant simply base filled with aio defaults? | 18:41 |
| *** tonythomas has quit IRC | 18:42 | |
| yoctozepto | (could be nice to document) | 18:42 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 18:59 |
| *** mcnugit has joined #openstack-kolla | 19:07 | |
| *** kplant has quit IRC | 19:07 | |
| mnasiadka | yoctozepto: seems debian-source is 100% fail game | 19:16 |
| yoctozepto | mnasiadka: yes, they broke ping | 19:16 |
| mnasiadka | debian broke ping | 19:16 |
| mnasiadka | how can you break ping? | 19:16 |
| yoctozepto | debian can | 19:17 |
| * yoctozepto cannot | 19:17 | |
| yoctozepto | mnasiadka: are you able to answer my kayobe questions above? :-) | 19:26 |
| mnasiadka | yoctozepto: nope, I’m a user in kayobe world - didn’t have time to investigate repos and CI | 19:27 |
| yoctozepto | mnasiadka: thank you for explanations | 19:28 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Change /run bind mount for neutron/openvswitch https://review.opendev.org/708905 | 19:36 |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: [keystone] adds credential encryption default directory https://review.opendev.org/708955 | 19:38 |
| *** igordc has quit IRC | 19:38 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 19:49 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Check for supported OS distributions in prechecks https://review.opendev.org/699443 | 19:59 |
| osmanlicilegi | mnasiadka: could you rebase https://review.opendev.org/#/c/676389/ when you have some free time | 20:02 |
| patchbot | patch 676389 - kolla-ansible - Use Docker healthchecks for core services - 48 patch sets | 20:02 |
| mnasiadka | osmanlicilegi: will do tomorrow | 20:03 |
| osmanlicilegi | I've just finished reviewing but seems it's on merge conflict | 20:03 |
| openstackgerrit | Merged openstack/kolla-ansible master: Introduce influxdb_datadir_volume https://review.opendev.org/707861 | 20:09 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Remove dbus from bindep and playbooks https://review.opendev.org/708082 | 20:13 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance https://review.opendev.org/708114 | 20:13 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible master: Introduce /etc/timezone to containers https://review.opendev.org/704258 | 20:21 |
| *** mcnugit has quit IRC | 20:27 | |
| yoctozepto | osmanlicilegi: https://review.opendev.org/708318 | 20:34 |
| patchbot | patch 708318 - kolla-ansible - CI: Add linuxbridge jobs - 16 patch sets | 20:34 |
| yoctozepto | osmanlicilegi: btw why "osmanlicilegi"? does it mean something in Turkish? | 20:34 |
| yoctozepto | https://translate.google.com/#view=home&op=translate&sl=tr&tl=en&text=osmanli%20cilegi ? | 20:35 |
| yoctozepto | ottoman honey? well, that's sweet! | 20:35 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 | 20:37 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Generate ARA HTML https://review.opendev.org/707450 | 20:40 |
| yoctozepto | yay, finally | 20:40 |
| yoctozepto | the ara is baaack | 20:40 |
| *** priteau has quit IRC | 20:44 | |
| *** TrevorV has quit IRC | 21:03 | |
| openstackgerrit | Merged openstack/kolla master: Remove skipping B303 for bandit checks https://review.opendev.org/655537 | 21:06 |
| *** dougsz has quit IRC | 21:21 | |
| *** igordc has joined #openstack-kolla | 21:28 | |
| *** jbadiapa has quit IRC | 21:31 | |
| *** lile has quit IRC | 21:47 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Revert "Upgrade virtualenv in pre" https://review.opendev.org/707622 | 22:08 |
| dking_desktop | Would this be the right channel to mention that the instructions at https://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bifrost.html seem outdated? | 22:22 |
| *** vishwanathj has joined #openstack-kolla | 23:35 | |
| *** vishwanathj has quit IRC | 23:39 | |
| *** dave-mccowan has quit IRC | 23:51 | |
| *** dave-mccowan has joined #openstack-kolla | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!