*** macz_ has quit IRC | 00:09 | |
*** k_mouza has joined #openstack-kolla | 00:14 | |
*** k_mouza has quit IRC | 00:19 | |
*** macz_ has joined #openstack-kolla | 02:10 | |
*** macz_ has quit IRC | 02:14 | |
*** k_mouza has joined #openstack-kolla | 02:34 | |
*** k_mouza has quit IRC | 02:39 | |
*** skramaja has joined #openstack-kolla | 03:22 | |
*** macz_ has joined #openstack-kolla | 03:46 | |
*** macz_ has quit IRC | 03:51 | |
openstackgerrit | likui proposed openstack/kolla-ansible master: Indented two spaces to match the other things in this block https://review.opendev.org/c/openstack/kolla-ansible/+/792064 | 03:53 |
---|---|---|
*** skramaja_ has joined #openstack-kolla | 04:17 | |
*** skramaja has quit IRC | 04:17 | |
*** skramaja_ has quit IRC | 04:40 | |
*** skramaja has joined #openstack-kolla | 04:45 | |
*** cah_link has joined #openstack-kolla | 05:17 | |
*** macz_ has joined #openstack-kolla | 05:47 | |
*** macz_ has quit IRC | 05:52 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792071 | 06:10 |
*** zijlboot has joined #openstack-kolla | 06:14 | |
*** zijlboot has quit IRC | 06:15 | |
*** zijlboot has joined #openstack-kolla | 06:16 | |
yoctozepto | morning | 06:24 |
*** shyamb has joined #openstack-kolla | 06:28 | |
mnasiadka | yoctozepto: morning - docker sdk breakage ^^ | 06:33 |
mnasiadka | I’ll put it on the whiteboard in some minutes | 06:33 |
*** k_mouza has joined #openstack-kolla | 06:35 | |
*** vishalmanchanda has joined #openstack-kolla | 06:35 | |
*** k_mouza has quit IRC | 06:39 | |
openstackgerrit | Verification of a change to openstack/kolla-ansible failed: baremetal: Don't start Docker after install on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla-ansible/+/791581 | 06:40 |
*** shyamb has quit IRC | 07:04 | |
hrw | lo | 07:04 |
*** shyamb has joined #openstack-kolla | 07:05 | |
*** e0ne has joined #openstack-kolla | 07:08 | |
*** e0ne has quit IRC | 07:09 | |
*** e0ne has joined #openstack-kolla | 07:10 | |
parallax | morning | 07:10 |
*** shyam89 has joined #openstack-kolla | 07:11 | |
*** shyamb has quit IRC | 07:12 | |
*** kevko_ has joined #openstack-kolla | 07:14 | |
*** andrewbonney has joined #openstack-kolla | 07:17 | |
yoctozepto | mnasiadka: arr :D | 07:18 |
yoctozepto | but it breaks all the branches | 07:18 |
yoctozepto | report a bug :-( | 07:19 |
mnasiadka | yeah, will do | 07:22 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792071 | 07:26 |
yoctozepto | mnasiadka: btw, are the distro-provided bindings fine too? | 07:28 |
yoctozepto | I wonder if we really need the latest as we don't really use that advanced docker apis | 07:28 |
*** bengates has joined #openstack-kolla | 07:28 | |
*** kevko_ has quit IRC | 07:29 | |
*** kevko has joined #openstack-kolla | 07:29 | |
mnasiadka | yoctozepto: there's not really much in 5.0 apart dropping py27 - https://docker-py.readthedocs.io/en/stable/change-log.html# | 07:30 |
mnasiadka | yoctozepto: we could also think about dropping usage of pip and installing distro packages - I think Ubuntu/Debian has them and RDO provides it as well - but let's first constrain it. | 07:31 |
yoctozepto | yeah, that's why I am thinking about relying on distro-provided bindings even; as these newer versions don't really bring features to our use | 07:31 |
yoctozepto | mnasiadka: ++ | 07:32 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792071 | 07:32 |
mnasiadka | ok, added reno | 07:32 |
yoctozepto | you read my mind | 07:32 |
yoctozepto | mnasiadka: fix the whitespace in it | 07:33 |
mnasiadka | ugh | 07:33 |
*** rpittau|afk is now known as rpittau | 07:33 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792071 | 07:33 |
hrw | mnasiadka: we could also install six ;D | 07:41 |
mnasiadka | hrw: well, wonder what are breakages docker py 5.0.0 is hiding :) | 07:42 |
mnasiadka | what other | 07:42 |
priteau | Let's not legitimate a buggy release ;-) | 07:44 |
*** skramaja has quit IRC | 07:44 | |
*** skramaja has joined #openstack-kolla | 07:44 | |
yoctozepto | yeah, it only proves they have some invalid CI/CD in place | 07:45 |
yoctozepto | if it passed this particular issue | 07:45 |
*** shyam89 has quit IRC | 07:47 | |
mnasiadka | usually there's no CI :) | 07:47 |
*** shyam89 has joined #openstack-kolla | 07:48 | |
*** macz_ has joined #openstack-kolla | 07:48 | |
*** macz_ has quit IRC | 07:52 | |
*** gfidente|afk is now known as gfidente | 07:53 | |
yoctozepto | well, it was released Apr 6 | 07:53 |
yoctozepto | so it worked at the time | 07:54 |
yoctozepto | some dep stopped shipping six too | 07:54 |
yoctozepto | and then it broke | 07:54 |
*** ChanServ has quit IRC | 07:54 | |
*** ChanServ has joined #openstack-kolla | 07:54 | |
*** services. sets mode: +o ChanServ | 07:54 | |
mnasiadka | yoctozepto: life ;) | 07:56 |
*** nikparasyr has joined #openstack-kolla | 07:57 | |
*** jbadiapa has joined #openstack-kolla | 07:57 | |
*** shyam89 has quit IRC | 07:58 | |
*** fuhrmannb has joined #openstack-kolla | 07:59 | |
*** dougsz has joined #openstack-kolla | 07:59 | |
*** shyamb has joined #openstack-kolla | 08:02 | |
yoctozepto | life is live | 08:04 |
yoctozepto | na-na-na-na-na | 08:04 |
openstackgerrit | Piotr Parczewski proposed openstack/kolla master: [Security] Fix open redirect in Prometheus https://review.opendev.org/c/openstack/kolla/+/792080 | 08:05 |
*** shyamb has quit IRC | 08:07 | |
*** dougsz has quit IRC | 08:16 | |
kevko | hi \o/ | 08:21 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] Python-Docker from distro https://review.opendev.org/c/openstack/kolla-ansible/+/792082 | 08:22 |
kevko | guys, do you mean we can merge this https://review.opendev.org/c/openstack/kolla-ansible/+/777772 ? as it is configurable | 08:22 |
mnasiadka | maybe we can, but we can't backport in that shape | 08:23 |
kevko | where is a problem | 08:27 |
kevko | yoctozepto: usefull patch with python docker installation ! :) thanks | 08:28 |
mnasiadka | kevko: commented on the patch | 08:30 |
*** k_mouza has joined #openstack-kolla | 08:30 | |
*** k_mouza has quit IRC | 08:34 | |
kevko | mnasiadka: replied | 08:41 |
*** shyamb has joined #openstack-kolla | 08:41 | |
kevko | mnasiadka: in next patchset I will add info about variable and default value | 08:41 |
mnasiadka | yoctozepto: what is interesting, six doesn't fail everytime - https://review.opendev.org/c/openstack/kolla-ansible/+/788687/ - this one is finishing gating :) | 08:42 |
*** shyamb has quit IRC | 08:45 | |
openstackgerrit | Merged openstack/kolla-ansible master: CI: Fix nfv job with kolla dependency https://review.opendev.org/c/openstack/kolla-ansible/+/788687 | 08:58 |
*** jhorstmann has joined #openstack-kolla | 09:00 | |
*** k_mouza has joined #openstack-kolla | 09:04 | |
mnasiadka | yoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/788687 - how far do we need to backport this? | 09:10 |
hrw | mnasiadka: mitaka? | 09:13 |
*** vishalmanchanda has quit IRC | 09:14 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/wallaby: CI: Fix nfv job with kolla dependency https://review.opendev.org/c/openstack/kolla-ansible/+/792107 | 09:16 |
*** mnasiadka has quit IRC | 09:17 | |
mgoddard | morning | 09:21 |
openstackgerrit | Mark Goddard proposed openstack/kayobe-config master: Synchronize kayobe-config https://review.opendev.org/c/openstack/kayobe-config/+/791265 | 09:28 |
yoctozepto | till wallaby | 09:31 |
yoctozepto | kevko: ye | 09:31 |
yoctozepto | yw* | 09:32 |
*** vishalmanchanda has joined #openstack-kolla | 09:38 | |
*** mnasiadka has joined #openstack-kolla | 09:39 | |
mnasiadka | hrw: liberty! ;) | 09:40 |
mgoddard | mnasiadka: you're havana laugh | 09:46 |
hrw | mnasiadka: liberty was the one I started with OpenStack. Mitaka is the one I started contributing | 09:48 |
hrw | I really do not want to remember <Mitaka ;D | 09:48 |
mnasiadka | mgoddard: I don't think Kolla Havana was a thing | 09:49 |
*** macz_ has joined #openstack-kolla | 09:49 | |
hrw | mnasiadka: once wallaby gets released you will backport to get havana working with kolla? | 09:53 |
mnasiadka | hrw: I dream of nothing else and have it on my priority list! ;) | 09:53 |
*** macz_ has quit IRC | 09:54 | |
hrw | cool! | 09:57 |
*** k_mouza has quit IRC | 09:59 | |
*** k_mouza has joined #openstack-kolla | 10:05 | |
openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Fix TCP connections refusing to die after VIP switch https://review.opendev.org/c/openstack/kolla-ansible/+/777772 | 10:10 |
kevko | mnasiadka: ^^ applied your comments ..are u ok with it now ? | 10:10 |
*** muhaha has joined #openstack-kolla | 10:11 | |
kevko | yoctozepto: ^^ | 10:11 |
muhaha | Guy? Maybe offtopic, but whats alternative to AWS KMS, Azure Keyvault in Openstack? Barbican , is it available in Kolla? I found that it can support Hashicorp Vault as backend, but I am not sure if its in scope of Kolla installation | 10:12 |
mnasiadka | muhaha: Barbican yes, Hashicorp Vault not | 10:16 |
mnasiadka | muhaha: I mean Kolla-Ansible won't deploy Vault - you need to do it by yourself | 10:16 |
mnasiadka | kevko: I'll look into it in a couple of minutes | 10:17 |
kevko | mnasiadka: thank you michal | 10:17 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: chrony: remove during upgrade when disabled https://review.opendev.org/c/openstack/kolla-ansible/+/792119 | 10:29 |
muhaha | @mnasiadka thanks | 10:35 |
*** murphyslawbbs has joined #openstack-kolla | 11:01 | |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: docs: Stop setting os_distribution in development all-in-one https://review.opendev.org/c/openstack/kayobe/+/792125 | 11:01 |
openstackgerrit | Pierre Riteau proposed openstack/kayobe-config master: Synchronize kayobe-config https://review.opendev.org/c/openstack/kayobe-config/+/791265 | 11:05 |
mnasiadka | yoctozepto: https://docs.docker.com/config/containers/runmetrics/#running-docker-on-cgroup-v2 - maybe we need to set CgroupNs for libvirt to work? | 11:16 |
openstackgerrit | Merged openstack/kolla-ansible stable/wallaby: CI: Fix nfv job with kolla dependency https://review.opendev.org/c/openstack/kolla-ansible/+/792107 | 11:18 |
yoctozepto | mnasiadka: ooh, that sounds like it | 11:25 |
yoctozepto | especially since it moved from host to private | 11:26 |
yoctozepto | would also explain that weirdness with machined path | 11:26 |
yoctozepto | are you trying it out? | 11:26 |
yoctozepto | I will try it with machined then | 11:28 |
mnasiadka | yoctozepto: currently trying to find what's bloody wrong with ovs 2.15 :) | 11:34 |
hrw | I may not be present on today's meeting - have to go with daughter for planned visit somewhere. | 11:36 |
yoctozepto | mnasiadka: all right, bloody python-docker does not even have the cgroup namespace option support | 11:39 |
mnasiadka | yoctozepto: that sounds fantastic | 11:39 |
hrw | with python-docker in past I had a feeling that it is barely maintained | 11:40 |
hrw | buildx support? nope - just use shell | 11:40 |
mnasiadka | maybe it's time to move to systemd units | 11:41 |
mnasiadka | yoctozepto: anyway you can test with docker CLI :D | 11:42 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye https://review.opendev.org/c/openstack/kolla-ansible/+/790678 | 11:43 |
yoctozepto | well, we can always call api directly | 11:43 |
yoctozepto | using python requests is nice and easy | 11:43 |
yoctozepto | though it could be tricky with command output | 11:43 |
hrw | https://github.com/docker/docker-py/issues/2230 | 11:44 |
openstackgerrit | Merged openstack/kayobe-config master: Synchronize kayobe-config https://review.opendev.org/c/openstack/kayobe-config/+/791265 | 11:45 |
*** macz_ has joined #openstack-kolla | 11:50 | |
*** macz_ has quit IRC | 11:54 | |
*** murphyslawbbs has quit IRC | 12:10 | |
openstackgerrit | Mark Goddard proposed openstack/kayobe-config stable/wallaby: Synchronize kayobe-config https://review.opendev.org/c/openstack/kayobe-config/+/792109 | 12:32 |
kplant | i'm sure you guys are in the loop on this: https://gist.github.com/joepie91/df80d8d36cd9d1bde46ba018af497409 | 12:32 |
Tengu | apparently OFTC might be an alternative. There's a discussion on the openstack-discuss ML already | 12:35 |
Tengu | kplant: http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html | 12:35 |
openstackgerrit | likui proposed openstack/kolla-ansible master: Merge glance sections for nova.conf.j2 https://review.opendev.org/c/openstack/kolla-ansible/+/791913 | 12:36 |
mgoddard | yoctozepto: https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/templates/globals-default.j2#L36 | 12:37 |
mgoddard | yoctozepto: in the context of https://review.opendev.org/c/openstack/kolla-ansible/+/792119 | 12:37 |
mgoddard | actually let's discuss it in the meeting, it's already on the agenda | 12:38 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/791304 | 12:38 |
*** Luzi has joined #openstack-kolla | 12:40 | |
yoctozepto | mgoddard: remove already in wallaby? | 12:42 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kolla-ansible/+/791276 | 12:42 |
mgoddard | yoctozepto: if config says disabled, we should remove | 12:42 |
mgoddard | otherwise you'll be left with a victoria chrony container | 12:43 |
yoctozepto | makes sense | 12:43 |
mgoddard | but if there is an issue with upgrading in CI, users will hit it too | 12:44 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye https://review.opendev.org/c/openstack/kolla-ansible/+/790678 | 12:45 |
mnasiadka | well, after we remove chrony container - there's no ntp sync, so next prechecks will fail I guess | 12:45 |
mgoddard | for good reason! | 12:45 |
mnasiadka | mgoddard: not saying it's not a good reason, just users need to be aware :D | 12:46 |
mgoddard | but if the user does not run prechecks, they won't have time sync | 12:46 |
mnasiadka | and then everything goes haywire ;) | 12:46 |
mgoddard | and if we remove the chrony container at the start of an upgrade, good luck to them | 12:47 |
mgoddard | the lesson here: never change anything :) | 12:48 |
mnasiadka | stay on Liberty or Mitaka | 12:49 |
mnasiadka | :) | 12:49 |
*** ricolin has joined #openstack-kolla | 13:08 | |
openstackgerrit | Merged openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792071 | 13:24 |
*** dasp has quit IRC | 13:24 | |
*** dasp has joined #openstack-kolla | 13:25 | |
*** wuchunyang has joined #openstack-kolla | 13:26 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792110 | 13:27 |
*** ricolin has quit IRC | 13:29 | |
*** macz_ has joined #openstack-kolla | 13:30 | |
*** macz_ has quit IRC | 13:35 | |
*** Luzi has quit IRC | 13:39 | |
*** kevko has quit IRC | 13:49 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792115 | 14:05 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792157 | 14:06 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792158 | 14:07 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792158 | 14:08 |
*** skramaja has quit IRC | 14:09 | |
*** raildo has joined #openstack-kolla | 14:10 | |
*** muhaha has quit IRC | 14:19 | |
yoctozepto | mgoddard, mnasiadka: cgroup namespace HELPED in debian (cgroups v2)! :D so far with machined running | 14:24 |
mnasiadka | yoctozepto: maybe we don’t need machined | 14:25 |
mnasiadka | yoctozepto: but that’s great news | 14:25 |
yoctozepto | mnasiadka: yeah, I will check | 14:26 |
yoctozepto | mnasiadka: great catch mnasiadka | 14:26 |
*** kevko has joined #openstack-kolla | 14:31 | |
*** kevko has quit IRC | 14:33 | |
*** kevko has joined #openstack-kolla | 14:33 | |
*** cah_link has quit IRC | 14:44 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye https://review.opendev.org/c/openstack/kolla-ansible/+/790678 | 14:45 |
openstackgerrit | Merged openstack/kolla master: [Security] Fix open redirect in Prometheus https://review.opendev.org/c/openstack/kolla/+/792080 | 14:46 |
*** kevko has quit IRC | 14:47 | |
*** kevko_ has joined #openstack-kolla | 14:47 | |
yoctozepto | there is a catch though | 14:49 |
yoctozepto | only since docker 20.10 is that option supported | 14:49 |
openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Fix TCP connections refusing to die after VIP switch https://review.opendev.org/c/openstack/kolla-ansible/+/777772 | 14:49 |
yoctozepto | so we would enforce docker upgrade if we just used it unconditionally | 14:50 |
*** cah_link has joined #openstack-kolla | 14:50 | |
yoctozepto | but I guess we can limit ourselves to debian bullseye | 14:50 |
yoctozepto | pity at docker which could not hold its own default | 14:57 |
yoctozepto | I mean; is it that hard not to break? :D | 14:57 |
yoctozepto | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:58 |
yoctozepto | meeting in 2 | 14:58 |
mgoddard | #startmeeting kolla | 15:00 |
openstack | Meeting started Wed May 19 15:00:08 2021 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
*** openstack changes topic to " (Meeting topic: kolla)" | 15:00 | |
openstack | The meeting name has been set to 'kolla' | 15:00 |
mgoddard | #topic rollcall | 15:00 |
*** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:00 | |
yoctozepto | \o/ | 15:00 |
yoctozepto | \_o_/ | 15:00 |
yoctozepto | /_o_\ | 15:00 |
mgoddard | dob | 15:01 |
hrw | /-o-\ | 15:01 |
yoctozepto | no, my arms broke | 15:01 |
headphoneJames | o/ | 15:01 |
wuchunyang | o | 15:02 |
*** rpittau is now known as rpittau|bbl | 15:03 | |
mgoddard | #topic agenda | 15:03 |
*** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
mgoddard | * Roll-call | 15:03 |
mgoddard | * Agenda | 15:03 |
mgoddard | * Announcements | 15:03 |
mgoddard | * Review action items from the last meeting | 15:04 |
mgoddard | * CI status | 15:04 |
mgoddard | * Wallaby release planning | 15:04 |
mgoddard | ** Debian bullseye | 15:04 |
mgoddard | ** chrony | 15:04 |
mgoddard | * Xena cycle planning | 15:04 |
mgoddard | ** master branch life cycle | 15:04 |
mgoddard | * Open discussion | 15:04 |
mgoddard | #topic announcements | 15:04 |
*** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:04 | |
mgoddard | I have none. Anyone else? | 15:04 |
hrw | nope | 15:04 |
openstackgerrit | Merged openstack/kayobe-config stable/wallaby: Synchronize kayobe-config https://review.opendev.org/c/openstack/kayobe-config/+/792109 | 15:05 |
mgoddard | #topic Review action items from the last meeting | 15:05 |
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)" | 15:05 | |
mgoddard | mgoddard email openstack-discuss about quay.io credentials | 15:05 |
mgoddard | mgoddard draft proposal for new release process | 15:05 |
mgoddard | no | 15:05 |
mgoddard | yes | 15:05 |
mgoddard | #action mgoddard email openstack-discuss about quay.io credentials | 15:05 |
mgoddard | #topic CI status | 15:05 |
yoctozepto | lol, that email is hard | 15:05 |
*** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:05 | |
*** macz_ has joined #openstack-kolla | 15:06 | |
yoctozepto | don't get me started on CI | 15:06 |
yoctozepto | of course it's RED | 15:06 |
yoctozepto | but we are patching :-) | 15:06 |
mgoddard | [all distros] [docker SDK] docker py 5.0.0 removed six but imports it and fails prechecks/any functionality | 15:06 |
mgoddard | fix merged to master, backporting in progress | 15:07 |
yoctozepto | on that note | 15:07 |
yoctozepto | we are trying to move to distro-provided sdk | 15:07 |
yoctozepto | it should be good enough as long as it is provided | 15:07 |
yoctozepto | :-) | 15:07 |
hrw | +2 | 15:07 |
mgoddard | well | 15:07 |
mgoddard | do we need it? | 15:07 |
mgoddard | now we have an upper limit on docker | 15:07 |
yoctozepto | nicer than pip-installing | 15:08 |
mgoddard | what happens if we need a feature from a newer version? | 15:08 |
yoctozepto | the crowds will love you - right, kevko? | 15:08 |
mgoddard | or if RDO drops | 15:08 |
yoctozepto | then we drop centos support | 15:08 |
*** murphyslawbbs has joined #openstack-kolla | 15:08 | |
yoctozepto | well, the newer versions seem be lacking behind in feature support anyhow | 15:09 |
yoctozepto | that's why I mentioned we are not really losing anything but using an older version | 15:09 |
mgoddard | ok, well let's not rush into it | 15:09 |
yoctozepto | for cgroups namespace I am directly modifying the API query contents | 15:09 |
yoctozepto | ok, let's move on | 15:10 |
mgoddard | ok, let's move on. Lots to get through | 15:10 |
yoctozepto | more interesting stuff ahead | 15:11 |
mgoddard | #topic Debian bullseye | 15:11 |
*** openstack changes topic to "Debian bullseye (Meeting topic: kolla)" | 15:11 | |
mgoddard | who wants to give a status update? | 15:11 |
yoctozepto | the bulls' only eye has arrived at the green station: https://review.opendev.org/c/openstack/kolla-ansible/+/790678 | 15:11 |
hrw | yay! | 15:11 |
yoctozepto | now checking without machined | 15:11 |
yoctozepto | we'll see | 15:12 |
yoctozepto | anyhow, we have a working solution *with* cgroups v2 | 15:12 |
yoctozepto | which is awesome | 15:12 |
yoctozepto | the other issue is with the upgrade | 15:12 |
yoctozepto | ovs 2.15 | 15:12 |
yoctozepto | just like in UCA before we pinned it | 15:12 |
yoctozepto | mnasiadka debugging this | 15:12 |
yoctozepto | no idea about the current progress, mnasiadka silent recently | 15:13 |
yoctozepto | perhaps ovs broke his internets | 15:13 |
*** nikparasyr has left #openstack-kolla | 15:13 | |
yoctozepto | this is worse on debian because we can't pin any "older version" | 15:13 |
yoctozepto | as this is from base bullseye | 15:13 |
yoctozepto | also, with current knowledge, expect breakage when rdo moves to ovs 2.15 | 15:14 |
yoctozepto | ;d | 15:14 |
mgoddard | could we use a buster repo? | 15:14 |
yoctozepto | I don't know, hrw, wdyt? | 15:14 |
yoctozepto | the other side to that isssue is that | 15:15 |
yoctozepto | it's not 100% | 15:15 |
yoctozepto | it may succeed | 15:15 |
yoctozepto | and it always succeeds in multinode | 15:15 |
hrw | mgoddard: shouldn't | 15:15 |
yoctozepto | perhaps ovs likes to have friends to get to work ;d | 15:15 |
yoctozepto | and it is 80% sad when alon | 15:15 |
yoctozepto | (the estimation is made up) | 15:16 |
yoctozepto | alone* | 15:16 |
mgoddard | have we spoken to debian openstack team about it? | 15:16 |
mgoddard | or ubuntu even | 15:17 |
yoctozepto | me not | 15:17 |
yoctozepto | hrw, mnasiadka? | 15:17 |
mgoddard | probably should | 15:18 |
hrw | not me | 15:18 |
hrw | I had to dig in non kolla stuff | 15:18 |
mgoddard | or ask on openstack-discuss | 15:18 |
*** cah_link has quit IRC | 15:18 | |
yoctozepto | well, one can argue nobody runs ovs on singlenode because it does not make much sense ;d | 15:19 |
*** cah_link1 has joined #openstack-kolla | 15:19 | |
yoctozepto | I asked zigo on #debian-openstack now | 15:20 |
yoctozepto | (on oftc) | 15:20 |
mgoddard | ok | 15:20 |
mgoddard | can we go back to debian & libvirt | 15:21 |
*** cah_link1 is now known as cah_link | 15:21 | |
mgoddard | https://docs.docker.com/engine/api/version-history/ suggests the CgroupnsMode parameter is API v1.41 | 15:21 |
mgoddard | anyone know how to map that to a docker version? | 15:22 |
mgoddard | 20.10.0 | 15:23 |
yoctozepto | 20.10 | 15:23 |
mgoddard | https://docs.docker.com/engine/release-notes/ | 15:23 |
yoctozepto | i mentioned this above | 15:23 |
yoctozepto | before the meeting | 15:23 |
yoctozepto | :-) | 15:23 |
yoctozepto | they decided to change the default | 15:23 |
yoctozepto | for fun | 15:23 |
mgoddard | I was elsewhere | 15:23 |
yoctozepto | and added a knob to change it back | 15:23 |
yoctozepto | so much for backward compat | 15:23 |
yoctozepto | well, at least they default to "more secure" | 15:24 |
mgoddard | people in glass houses etc. | 15:24 |
yoctozepto | I know | 15:24 |
yoctozepto | I like to rant | 15:24 |
yoctozepto | since it's going to happen on bullseye | 15:25 |
yoctozepto | we can condition it on being on bullseye | 15:25 |
yoctozepto | and then we relax as more platforms move to cgroups v2 | 15:25 |
mgoddard | just so I'm clear | 15:25 |
mgoddard | this default changed in 20.10 | 15:25 |
mgoddard | but only affects cgroups v2 | 15:25 |
yoctozepto | indeed | 15:25 |
mgoddard | which so far only debian bullseye uses out of our supported platforms | 15:26 |
yoctozepto | exactly | 15:26 |
mgoddard | k | 15:26 |
yoctozepto | and 20.10 is the only to support cgroups v2 out of the box | 15:26 |
yoctozepto | so older ones are supposed to fail / be unsupported | 15:26 |
mgoddard | so we need to either set 20.10 / 1.41 as our minimum supported version, or have some check for cgroups v2 in the module | 15:27 |
yoctozepto | I say we use this know only on bullseye | 15:28 |
yoctozepto | that is easiest ;d | 15:28 |
yoctozepto | knob* | 15:28 |
hrw | 20.10 is available for each of our host distros | 15:28 |
hrw | so we can depend 20.10 for wallaby+ | 15:28 |
mgoddard | current min docker version is 1.10 :) | 15:29 |
yoctozepto | I would not discriminate people running non-upstream docker ;d | 15:29 |
mgoddard | well, just because it's available, doesn't mean its in use | 15:29 |
yoctozepto | well, we can bump to 18.09 | 15:29 |
yoctozepto | as that seems to work | 15:29 |
mgoddard | does it? | 15:29 |
yoctozepto | 1.10 is scary | 15:29 |
mgoddard | +1 | 15:29 |
yoctozepto | yes, though checked for sure on train/ussuri | 15:30 |
hrw | https://www.docker.com/blog/docker-1-10/ - 4th Feb 2016 | 15:30 |
hrw | time to upgrade indeed | 15:30 |
mgoddard | but I thought the new parameter was added in 20.10? | 15:30 |
yoctozepto | yes, that's why we limit it to bullseye which requires both 20.10 and the knob | 15:30 |
*** cah_link has quit IRC | 15:30 | |
yoctozepto | and we have it dealt with | 15:30 |
mgoddard | ok | 15:31 |
yoctozepto | btw, it's passing without machined | 15:31 |
mgoddard | can we rely on distros not to switch to cgroupsv2? | 15:31 |
yoctozepto | I will check libvirt logs if no oddiness happened and refactor | 15:32 |
yoctozepto | oh, I am pretty sure they will not | 15:32 |
yoctozepto | neither would risk pissing off enterprise users | 15:32 |
mgoddard | ok | 15:33 |
mgoddard | added some notes to the patch | 15:33 |
hrw | ubuntu 22.04 will be cg2 | 15:33 |
hrw | similar with centos 9 | 15:33 |
yoctozepto | yes | 15:33 |
yoctozepto | I meant in their current versions | 15:33 |
mgoddard | we'll have a release that supports a migration | 15:33 |
yoctozepto | I hope I was not misread | 15:34 |
mgoddard | I guess we can handle that when we get to it | 15:34 |
yoctozepto | yeas | 15:34 |
hrw | cs9 will land in Xena or Yeti (I forgot dates) | 15:34 |
mgoddard | Let's move on | 15:34 |
mgoddard | #topic chrony | 15:34 |
*** openstack changes topic to "chrony (Meeting topic: kolla)" | 15:34 | |
yoctozepto | noo | 15:34 |
yoctozepto | one more thing | 15:34 |
mgoddard | #undo | 15:34 |
openstack | Removing item from minutes: #topic chrony | 15:34 |
yoctozepto | so Wallaby and Debian | 15:34 |
yoctozepto | is this the release where we support both Buster and Bullseye, right? | 15:35 |
yoctozepto | (on host) | 15:35 |
yoctozepto | (as the images are simply bullseye) | 15:35 |
hrw | yeah | 15:35 |
hrw | both can have same docker version but buster is cg1 | 15:35 |
yoctozepto | ok; should we test both in CI then? I would | 15:35 |
hrw | good point | 15:35 |
* yoctozepto is moving to debian-based setup and would love good CI coverage | 15:36 | |
hrw | yay | 15:36 |
mgoddard | here's what we said for ubuntu in victoria | 15:36 |
mgoddard | The Victoria release adds support for Ubuntu Focal 20.04 as a host operating system. Ubuntu users upgrading from Ussuri should first upgrade OpenStack containers to Victoria, which uses the Ubuntu Focal 20.04 base container image. Hosts should then be upgraded to Ubuntu Focal 20.04. | 15:36 |
mgoddard | (from https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html) | 15:36 |
mgoddard | I don't know if anyone ever tested it :) | 15:37 |
hrw | ;P | 15:37 |
mgoddard | so if we assume the same approach for debian | 15:37 |
mgoddard | we provide bullseye based containers in wallaby | 15:37 |
yoctozepto | yes, and it looks worky | 15:38 |
mgoddard | and support both buster and bullseye hosts | 15:38 |
*** dave-mccowan has quit IRC | 15:38 | |
yoctozepto | it looked worky with ubuntu too | 15:38 |
yoctozepto | but we did not test it in CI | 15:38 |
yoctozepto | I need to check it | 15:38 |
mgoddard | buster host, victoria/buster containers -> buster host, wallaby/bullseye containers -> bullseye host, wallaby/bullseye containers | 15:39 |
hrw | yeah | 15:39 |
mgoddard | so our upgrade jobs should use buster in wallaby | 15:39 |
mgoddard | and our host OS checks should allow both | 15:39 |
yoctozepto | yeah, we have done it for ubuntu | 15:40 |
yoctozepto | using bionic in upgrade | 15:40 |
yoctozepto | and focal in others | 15:40 |
yoctozepto | so let's do the same here | 15:40 |
yoctozepto | buster in upgrade | 15:40 |
yoctozepto | bullseye in others | 15:40 |
mgoddard | I'll add add notes to the patch | 15:41 |
yoctozepto | (there is only one but I might throw more in Xena) | 15:41 |
yoctozepto | thanks | 15:41 |
mgoddard | can I chrony yet? | 15:42 |
mgoddard | #topic chrony | 15:43 |
*** openstack changes topic to "chrony (Meeting topic: kolla)" | 15:43 | |
mgoddard | #link https://review.opendev.org/c/openstack/kolla-ansible/+/792119 | 15:43 |
mgoddard | wallaby deprecates chrony, and disables it by default | 15:43 |
mgoddard | therefore we should clean up the container, if disabled | 15:43 |
mgoddard | but, how do we do this cleanly without losing time sync? | 15:44 |
yoctozepto | good q | 15:44 |
*** kevko_ has quit IRC | 15:44 | |
yoctozepto | well, if there was chrony container to remove | 15:44 |
yoctozepto | and it worked correctly | 15:44 |
hrw | how do we handle it on fresh installs? | 15:44 |
yoctozepto | then we are very likely breaking it | 15:45 |
yoctozepto | can we do it like this | 15:45 |
yoctozepto | if we do upgrade | 15:45 |
yoctozepto | and chrony is disabled | 15:45 |
yoctozepto | but it was enabled (i.e., the playbook sees containers to go down) | 15:45 |
yoctozepto | we pause the playbook | 15:46 |
yoctozepto | and wait for user to acknowledge this | 15:46 |
yoctozepto | we can have a variable to skip this acknowledgment | 15:46 |
yoctozepto | (to support automated users who read renos) | 15:46 |
yoctozepto | and also we will not pause if no containers exist | 15:46 |
yoctozepto | this way we target the right people | 15:46 |
mgoddard | or people who run it twice :) | 15:47 |
yoctozepto | twice? I excluded those | 15:47 |
mgoddard | we have a time sync precheck | 15:48 |
yoctozepto | it's used in different situations | 15:48 |
mgoddard | could we use that? | 15:48 |
yoctozepto | and it would succeed right after | 15:48 |
yoctozepto | we just need to let users *know for sure* | 15:48 |
mgoddard | how long would it take to not succeed? | 15:48 |
yoctozepto | also, I meant this pause ~> https://docs.ansible.com/ansible/latest/collections/ansible/builtin/pause_module.html | 15:48 |
yoctozepto | mgoddard: I think it depends on kernel observing the clock stability | 15:49 |
yoctozepto | I noticed it being set as unsync after 24h | 15:49 |
yoctozepto | no, we will not add a wait for this ;-) | 15:49 |
yoctozepto | well, the fun fact is | 15:50 |
yoctozepto | the host would have worky ntp | 15:50 |
yoctozepto | if not for kolla-ansible which broke it on purpose to get chrony on board :-) | 15:50 |
mgoddard | how about this | 15:51 |
yoctozepto | yes | 15:51 |
mgoddard | check systemd for known ntp daemons | 15:51 |
mgoddard | add a flag to override, aka acknowledge the change | 15:51 |
mgoddard | provide a command/playbook to cleanup chrony before upgrade | 15:52 |
mgoddard | so ideal workflow would be | 15:52 |
mgoddard | kolla-ansible cleanup_chrony | 15:52 |
yoctozepto | 1) kill_my_chrony | 15:52 |
mgoddard | kolla-ansible prechecks | 15:52 |
mgoddard | kolla-ansible upgrade | 15:53 |
mgoddard | but, for those who ignore renos | 15:53 |
mgoddard | kolla-ansible upgrade | 15:53 |
mgoddard | will check for ntp daemons before cleaning up chrony | 15:53 |
yoctozepto | well, we can always teach people a lesson to read renos | 15:53 |
mgoddard | my clients tend not to like it if I teach them a lesson... | 15:54 |
yoctozepto | but are not you the one doing their upgrades? | 15:54 |
yoctozepto | (or someone else from stackhpc) | 15:54 |
mgoddard | not always | 15:54 |
yoctozepto | well, I think upgrades are the pinnacle of openstack support | 15:55 |
yoctozepto | so they should rethink their attitude | 15:55 |
yoctozepto | but I get you | 15:55 |
mgoddard | I will pass on your message :D | 15:55 |
mgoddard | anyway | 15:55 |
mgoddard | needs more thought, but we have some ideas | 15:55 |
mgoddard | #topic master branch life cycle | 15:56 |
*** openstack changes topic to "master branch life cycle (Meeting topic: kolla)" | 15:56 | |
mgoddard | #link https://etherpad.opendev.org/p/kolla-release-process-draft | 15:56 |
mgoddard | did anyone read it? | 15:56 |
yoctozepto | I didn't have time to think about time frame | 15:56 |
yoctozepto | but I read it | 15:56 |
hrw | I did | 15:56 |
hrw | commented even | 15:57 |
yoctozepto | my biggest concern is | 15:59 |
yoctozepto | in this simplistic view | 15:59 |
yoctozepto | we lose the ability to e.g. test bifrost master | 15:59 |
hrw | nope | 16:00 |
yoctozepto | or perhaps not | 16:00 |
yoctozepto | because I think I switch the reference forcibly | 16:00 |
mgoddard | and bifrost master now has a job that uses wallaby? | 16:00 |
hrw | R+9 is when we use master source instead of stable/previous | 16:00 |
yoctozepto | mgoddard: no, just realised the code is replacing the reference with what is in the change | 16:00 |
mgoddard | R+9 is next week | 16:01 |
yoctozepto | so I was just confusing myself and you | 16:01 |
yoctozepto | yeah, the timeframe is to be discussed really | 16:01 |
yoctozepto | and for the meeting as well | 16:01 |
openstackgerrit | Merged openstack/kolla-ansible stable/wallaby: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792110 | 16:01 |
yoctozepto | as we missed its timeframe | 16:01 |
yoctozepto | thanks mgoddard | 16:01 |
mgoddard | bifrost job may use master, but we will not be testing that, so who knows if it works? | 16:02 |
yoctozepto | mgoddard: I would say differently: we may break the job on bifrost queue now and not know it | 16:02 |
*** wuchunyang has quit IRC | 16:03 | |
mgoddard | or they may require a change on our side, but we cannot test it | 16:03 |
yoctozepto | we can test it "once" | 16:04 |
yoctozepto | but then it reverts back to stable for subsequent runs | 16:04 |
*** bengates has quit IRC | 16:04 | |
mgoddard | I'll put the draft onto openstack-discuss | 16:04 |
mgoddard | and also announce end of feature freeze, which should have happened some time ago | 16:05 |
mgoddard | thanks all | 16:05 |
mgoddard | #endmeeting | 16:05 |
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:05 | |
openstack | Meeting ended Wed May 19 16:05:23 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:05 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.html | 16:05 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.txt | 16:05 |
openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.log.html | 16:05 |
hrw | thx | 16:05 |
*** bengates has joined #openstack-kolla | 16:05 | |
yoctozepto | thanks mgoddard | 16:05 |
* hrw out | 16:05 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/wallaby: [Security] Fix open redirect in Prometheus https://review.opendev.org/c/openstack/kolla/+/792162 | 16:08 |
*** bengates has quit IRC | 16:09 | |
*** zijlboot has quit IRC | 16:12 | |
*** zijlboot has joined #openstack-kolla | 16:12 | |
*** k_mouza has quit IRC | 16:14 | |
*** zijlboot_ has joined #openstack-kolla | 16:16 | |
*** zijlboot_ has quit IRC | 16:18 | |
*** zijlboot_ has joined #openstack-kolla | 16:18 | |
*** zijlboot_ has quit IRC | 16:18 | |
*** zijlboot_ has joined #openstack-kolla | 16:18 | |
*** zijlboot has quit IRC | 16:19 | |
*** zijlboot_ has quit IRC | 16:25 | |
*** rpittau|bbl is now known as rpittau | 16:45 | |
*** zijlboot has joined #openstack-kolla | 16:53 | |
*** zijlboot has quit IRC | 16:57 | |
*** ricolin_ has joined #openstack-kolla | 17:03 | |
*** jonaspaulo has joined #openstack-kolla | 17:06 | |
*** fuhrmannb has quit IRC | 17:08 | |
openstackgerrit | Merged openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792158 | 17:10 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: chrony: remove during upgrade when disabled https://review.opendev.org/c/openstack/kolla-ansible/+/792119 | 17:20 |
openstackgerrit | Rafael Weingartner proposed openstack/kolla-ansible master: Make setup module arguments configurable https://review.opendev.org/c/openstack/kolla-ansible/+/783392 | 17:27 |
*** rpittau is now known as rpittau|afk | 17:37 | |
*** waxfire has joined #openstack-kolla | 17:41 | |
*** waxfire has left #openstack-kolla | 17:42 | |
*** murphyslawbbs has quit IRC | 17:46 | |
*** k_mouza has joined #openstack-kolla | 18:00 | |
*** k_mouza has quit IRC | 18:05 | |
*** andrewbonney has quit IRC | 18:05 | |
*** vishalmanchanda has quit IRC | 18:22 | |
*** jbadiapa has quit IRC | 18:30 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: chrony: remove during upgrade when disabled https://review.opendev.org/c/openstack/kolla-ansible/+/792119 | 18:43 |
openstackgerrit | Rafael Weingartner proposed openstack/kolla-ansible master: Make setup module arguments configurable https://review.opendev.org/c/openstack/kolla-ansible/+/783392 | 18:47 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: chrony: cleanup during overcloud host upgrade https://review.opendev.org/c/openstack/kayobe/+/792243 | 18:48 |
*** dking has joined #openstack-kolla | 18:56 | |
*** k3nny0ne has joined #openstack-kolla | 18:58 | |
*** k3nny0ne has quit IRC | 18:59 | |
*** k3nny0ne has joined #openstack-kolla | 19:01 | |
*** mushrushu has quit IRC | 19:25 | |
*** mushrushu has joined #openstack-kolla | 19:26 | |
*** samcat116 has joined #openstack-kolla | 19:46 | |
*** e0ne has quit IRC | 20:01 | |
openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: baremetal: Install Docker SDK less than 5.0.0 https://review.opendev.org/c/openstack/kolla-ansible/+/792157 | 20:08 |
*** zul has quit IRC | 20:09 | |
*** e0ne has joined #openstack-kolla | 20:14 | |
*** k_mouza has joined #openstack-kolla | 20:15 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!