Wednesday, 2021-05-19

« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »
*** macz_ has quit IRC00:09
*** k_mouza has joined #openstack-kolla00:14
*** k_mouza has quit IRC00:19
*** macz_ has joined #openstack-kolla02:10
*** macz_ has quit IRC02:14
*** k_mouza has joined #openstack-kolla02:34
*** k_mouza has quit IRC02:39
*** skramaja has joined #openstack-kolla03:22
*** macz_ has joined #openstack-kolla03:46
*** macz_ has quit IRC03:51
openstackgerritlikui proposed openstack/kolla-ansible master: Indented two spaces to match the other things in this block  https://review.opendev.org/c/openstack/kolla-ansible/+/79206403:53
*** skramaja_ has joined #openstack-kolla04:17
*** skramaja has quit IRC04:17
*** skramaja_ has quit IRC04:40
*** skramaja has joined #openstack-kolla04:45
*** cah_link has joined #openstack-kolla05:17
*** macz_ has joined #openstack-kolla05:47
*** macz_ has quit IRC05:52
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79207106:10
*** zijlboot has joined #openstack-kolla06:14
*** zijlboot has quit IRC06:15
*** zijlboot has joined #openstack-kolla06:16
yoctozeptomorning06:24
*** shyamb has joined #openstack-kolla06:28
mnasiadkayoctozepto: morning - docker sdk breakage ^^06:33
mnasiadkaI’ll put it on the whiteboard in some minutes06:33
*** k_mouza has joined #openstack-kolla06:35
*** vishalmanchanda has joined #openstack-kolla06:35
*** k_mouza has quit IRC06:39
openstackgerritVerification of a change to openstack/kolla-ansible failed: baremetal: Don't start Docker after install on Debian/Ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/79158106:40
*** shyamb has quit IRC07:04
hrwlo07:04
*** shyamb has joined #openstack-kolla07:05
*** e0ne has joined #openstack-kolla07:08
*** e0ne has quit IRC07:09
*** e0ne has joined #openstack-kolla07:10
parallaxmorning07:10
*** shyam89 has joined #openstack-kolla07:11
*** shyamb has quit IRC07:12
*** kevko_ has joined #openstack-kolla07:14
*** andrewbonney has joined #openstack-kolla07:17
yoctozeptomnasiadka: arr :D07:18
yoctozeptobut it breaks all the branches07:18
yoctozeptoreport a bug :-(07:19
mnasiadkayeah, will do07:22
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79207107:26
yoctozeptomnasiadka: btw, are the distro-provided bindings fine too?07:28
yoctozeptoI wonder if we really need the latest as we don't really use that advanced docker apis07:28
*** bengates has joined #openstack-kolla07:28
*** kevko_ has quit IRC07:29
*** kevko has joined #openstack-kolla07:29
mnasiadkayoctozepto: there's not really much in 5.0 apart dropping py27 -  https://docker-py.readthedocs.io/en/stable/change-log.html#07:30
mnasiadkayoctozepto: we could also think about dropping usage of pip and installing distro packages - I think Ubuntu/Debian has them and RDO provides it as well - but let's first constrain it.07:31
yoctozeptoyeah, that's why I am thinking about relying on distro-provided bindings even; as these newer versions don't really bring features to our use07:31
yoctozeptomnasiadka: ++07:32
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79207107:32
mnasiadkaok, added reno07:32
yoctozeptoyou read my mind07:32
yoctozeptomnasiadka: fix the whitespace in it07:33
mnasiadkaugh07:33
*** rpittau|afk is now known as rpittau07:33
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79207107:33
hrwmnasiadka: we could also install six ;D07:41
mnasiadkahrw: well, wonder what are breakages docker py 5.0.0 is hiding :)07:42
mnasiadkawhat other07:42
priteauLet's not legitimate a buggy release ;-)07:44
*** skramaja has quit IRC07:44
*** skramaja has joined #openstack-kolla07:44
yoctozeptoyeah, it only proves they have some invalid CI/CD in place07:45
yoctozeptoif it passed this particular issue07:45
*** shyam89 has quit IRC07:47
mnasiadkausually there's no CI :)07:47
*** shyam89 has joined #openstack-kolla07:48
*** macz_ has joined #openstack-kolla07:48
*** macz_ has quit IRC07:52
*** gfidente|afk is now known as gfidente07:53
yoctozeptowell, it was released Apr 607:53
yoctozeptoso it worked at the time07:54
yoctozeptosome dep stopped shipping six too07:54
yoctozeptoand then it broke07:54
*** ChanServ has quit IRC07:54
*** ChanServ has joined #openstack-kolla07:54
*** services. sets mode: +o ChanServ07:54
mnasiadkayoctozepto: life ;)07:56
*** nikparasyr has joined #openstack-kolla07:57
*** jbadiapa has joined #openstack-kolla07:57
*** shyam89 has quit IRC07:58
*** fuhrmannb has joined #openstack-kolla07:59
*** dougsz has joined #openstack-kolla07:59
*** shyamb has joined #openstack-kolla08:02
yoctozeptolife is live08:04
yoctozeptona-na-na-na-na08:04
openstackgerritPiotr Parczewski proposed openstack/kolla master: [Security] Fix open redirect in Prometheus  https://review.opendev.org/c/openstack/kolla/+/79208008:05
*** shyamb has quit IRC08:07
*** dougsz has quit IRC08:16
kevkohi \o/08:21
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [WIP] Python-Docker from distro  https://review.opendev.org/c/openstack/kolla-ansible/+/79208208:22
kevkoguys, do you mean we can merge this https://review.opendev.org/c/openstack/kolla-ansible/+/777772 ? as it is configurable08:22
mnasiadkamaybe we can, but we can't backport in that shape08:23
kevkowhere is a problem08:27
kevkoyoctozepto: usefull patch with python docker installation ! :) thanks08:28
mnasiadkakevko: commented on the patch08:30
*** k_mouza has joined #openstack-kolla08:30
*** k_mouza has quit IRC08:34
kevkomnasiadka: replied08:41
*** shyamb has joined #openstack-kolla08:41
kevkomnasiadka: in next patchset I will add info about variable and default value08:41
mnasiadkayoctozepto: what is interesting, six doesn't fail everytime - https://review.opendev.org/c/openstack/kolla-ansible/+/788687/ - this one is finishing gating :)08:42
*** shyamb has quit IRC08:45
openstackgerritMerged openstack/kolla-ansible master: CI: Fix nfv job with kolla dependency  https://review.opendev.org/c/openstack/kolla-ansible/+/78868708:58
*** jhorstmann has joined #openstack-kolla09:00
*** k_mouza has joined #openstack-kolla09:04
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/788687 - how far do we need to backport this?09:10
hrwmnasiadka: mitaka?09:13
*** vishalmanchanda has quit IRC09:14
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/wallaby: CI: Fix nfv job with kolla dependency  https://review.opendev.org/c/openstack/kolla-ansible/+/79210709:16
*** mnasiadka has quit IRC09:17
mgoddardmorning09:21
openstackgerritMark Goddard proposed openstack/kayobe-config master: Synchronize kayobe-config  https://review.opendev.org/c/openstack/kayobe-config/+/79126509:28
yoctozeptotill wallaby09:31
yoctozeptokevko: ye09:31
yoctozeptoyw*09:32
*** vishalmanchanda has joined #openstack-kolla09:38
*** mnasiadka has joined #openstack-kolla09:39
mnasiadkahrw: liberty! ;)09:40
mgoddardmnasiadka: you're havana laugh09:46
hrwmnasiadka: liberty was the one I started with OpenStack. Mitaka is the one I started contributing09:48
hrwI really do not want to remember <Mitaka ;D09:48
mnasiadkamgoddard: I don't think Kolla Havana was a thing09:49
*** macz_ has joined #openstack-kolla09:49
hrwmnasiadka: once wallaby gets released you will backport to get havana working with kolla?09:53
mnasiadkahrw: I dream of nothing else and have it on my priority list! ;)09:53
*** macz_ has quit IRC09:54
hrwcool!09:57
*** k_mouza has quit IRC09:59
*** k_mouza has joined #openstack-kolla10:05
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Fix TCP connections refusing to die after VIP switch  https://review.opendev.org/c/openstack/kolla-ansible/+/77777210:10
kevkomnasiadka: ^^ applied your comments ..are u ok with it now ?10:10
*** muhaha has joined #openstack-kolla10:11
kevkoyoctozepto: ^^10:11
muhahaGuy? Maybe offtopic, but whats alternative to AWS KMS, Azure Keyvault in Openstack? Barbican , is it available in Kolla? I found that it can support Hashicorp Vault as backend, but I am not sure if its in scope of Kolla installation10:12
mnasiadkamuhaha: Barbican yes, Hashicorp Vault not10:16
mnasiadkamuhaha: I mean Kolla-Ansible won't deploy Vault - you need to do it by yourself10:16
mnasiadkakevko: I'll look into it in a couple of minutes10:17
kevkomnasiadka: thank you michal10:17
openstackgerritMark Goddard proposed openstack/kolla-ansible master: chrony: remove during upgrade when disabled  https://review.opendev.org/c/openstack/kolla-ansible/+/79211910:29
muhaha@mnasiadka thanks10:35
*** murphyslawbbs has joined #openstack-kolla11:01
openstackgerritMark Goddard proposed openstack/kayobe master: docs: Stop setting os_distribution in development all-in-one  https://review.opendev.org/c/openstack/kayobe/+/79212511:01
openstackgerritPierre Riteau proposed openstack/kayobe-config master: Synchronize kayobe-config  https://review.opendev.org/c/openstack/kayobe-config/+/79126511:05
mnasiadkayoctozepto: https://docs.docker.com/config/containers/runmetrics/#running-docker-on-cgroup-v2 - maybe we need to set CgroupNs for libvirt to work?11:16
openstackgerritMerged openstack/kolla-ansible stable/wallaby: CI: Fix nfv job with kolla dependency  https://review.opendev.org/c/openstack/kolla-ansible/+/79210711:18
yoctozeptomnasiadka: ooh, that sounds like it11:25
yoctozeptoespecially since it moved from host to private11:26
yoctozeptowould also explain that weirdness with machined path11:26
yoctozeptoare you trying it out?11:26
yoctozeptoI will try it with machined then11:28
mnasiadkayoctozepto: currently trying to find what's bloody wrong with ovs 2.15 :)11:34
hrwI may not be present on today's meeting - have to go with daughter for planned visit somewhere.11:36
yoctozeptomnasiadka: all right, bloody python-docker does not even have the cgroup namespace option support11:39
mnasiadkayoctozepto: that sounds fantastic11:39
hrwwith python-docker in past I had a feeling that it is barely maintained11:40
hrwbuildx support? nope - just use shell11:40
mnasiadkamaybe it's time to move to systemd units11:41
mnasiadkayoctozepto: anyway you can test with docker CLI :D11:42
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye  https://review.opendev.org/c/openstack/kolla-ansible/+/79067811:43
yoctozeptowell, we can always call api directly11:43
yoctozeptousing python requests is nice and easy11:43
yoctozeptothough it could be tricky with command output11:43
hrwhttps://github.com/docker/docker-py/issues/223011:44
openstackgerritMerged openstack/kayobe-config master: Synchronize kayobe-config  https://review.opendev.org/c/openstack/kayobe-config/+/79126511:45
*** macz_ has joined #openstack-kolla11:50
*** macz_ has quit IRC11:54
*** murphyslawbbs has quit IRC12:10
openstackgerritMark Goddard proposed openstack/kayobe-config stable/wallaby: Synchronize kayobe-config  https://review.opendev.org/c/openstack/kayobe-config/+/79210912:32
kplanti'm sure you guys are in the loop on this: https://gist.github.com/joepie91/df80d8d36cd9d1bde46ba018af49740912:32
Tenguapparently OFTC might be an alternative. There's a discussion on the openstack-discuss ML already12:35
Tengukplant: http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html12:35
openstackgerritlikui proposed openstack/kolla-ansible master: Merge glance sections for nova.conf.j2  https://review.opendev.org/c/openstack/kolla-ansible/+/79191312:36
mgoddardyoctozepto: https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/templates/globals-default.j2#L3612:37
mgoddardyoctozepto: in the context of https://review.opendev.org/c/openstack/kolla-ansible/+/79211912:37
mgoddardactually let's discuss it in the meeting, it's already on the agenda12:38
openstackgerritMark Goddard proposed openstack/kayobe master: Use ansible_facts to reference facts  https://review.opendev.org/c/openstack/kayobe/+/79130412:38
*** Luzi has joined #openstack-kolla12:40
yoctozeptomgoddard: remove already in wallaby?12:42
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use ansible_facts to reference facts  https://review.opendev.org/c/openstack/kolla-ansible/+/79127612:42
mgoddardyoctozepto: if config says disabled, we should remove12:42
mgoddardotherwise you'll be left with a victoria chrony container12:43
yoctozeptomakes sense12:43
mgoddardbut if there is an issue with upgrading in CI, users will hit it too12:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye  https://review.opendev.org/c/openstack/kolla-ansible/+/79067812:45
mnasiadkawell, after we remove chrony container - there's no ntp sync, so next prechecks will fail I guess12:45
mgoddardfor good reason!12:45
mnasiadkamgoddard: not saying it's not a good reason, just users need to be aware :D12:46
mgoddardbut if the user does not run prechecks, they won't have time sync12:46
mnasiadkaand then everything goes haywire ;)12:46
mgoddardand if we remove the chrony container at the start of an upgrade, good luck to them12:47
mgoddardthe lesson here: never change anything :)12:48
mnasiadkastay on Liberty or Mitaka12:49
mnasiadka:)12:49
*** ricolin has joined #openstack-kolla13:08
openstackgerritMerged openstack/kolla-ansible master: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79207113:24
*** dasp has quit IRC13:24
*** dasp has joined #openstack-kolla13:25
*** wuchunyang has joined #openstack-kolla13:26
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/wallaby: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79211013:27
*** ricolin has quit IRC13:29
*** macz_ has joined #openstack-kolla13:30
*** macz_ has quit IRC13:35
*** Luzi has quit IRC13:39
*** kevko has quit IRC13:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/victoria: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79211514:05
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79215714:06
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79215814:07
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79215814:08
*** skramaja has quit IRC14:09
*** raildo has joined #openstack-kolla14:10
*** muhaha has quit IRC14:19
yoctozeptomgoddard, mnasiadka: cgroup namespace HELPED in debian (cgroups v2)! :D so far with machined running14:24
mnasiadkayoctozepto: maybe we don’t need machined14:25
mnasiadkayoctozepto: but that’s great news14:25
yoctozeptomnasiadka: yeah, I will check14:26
yoctozeptomnasiadka: great catch mnasiadka14:26
*** kevko has joined #openstack-kolla14:31
*** kevko has quit IRC14:33
*** kevko has joined #openstack-kolla14:33
*** cah_link has quit IRC14:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] [CI] Play with Debian bullseye  https://review.opendev.org/c/openstack/kolla-ansible/+/79067814:45
openstackgerritMerged openstack/kolla master: [Security] Fix open redirect in Prometheus  https://review.opendev.org/c/openstack/kolla/+/79208014:46
*** kevko has quit IRC14:47
*** kevko_ has joined #openstack-kolla14:47
yoctozeptothere is a catch though14:49
yoctozeptoonly since docker 20.10 is that option supported14:49
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Fix TCP connections refusing to die after VIP switch  https://review.opendev.org/c/openstack/kolla-ansible/+/77777214:49
yoctozeptoso we would enforce docker upgrade if we just used it unconditionally14:50
*** cah_link has joined #openstack-kolla14:50
yoctozeptobut I guess we can limit ourselves to debian bullseye14:50
yoctozeptopity at docker which could not hold its own default14:57
yoctozeptoI mean; is it that hard not to break? :D14:57
yoctozeptomgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:58
yoctozeptomeeting in 214:58
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed May 19 15:00:08 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
yoctozepto\o/15:00
yoctozepto\_o_/15:00
yoctozepto /_o_\15:00
mgoddarddob15:01
hrw /-o-\15:01
yoctozeptono, my arms broke15:01
headphoneJameso/15:01
wuchunyango15:02
*** rpittau is now known as rpittau|bbl15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Agenda15:03
mgoddard* Announcements15:03
mgoddard* Review action items from the last meeting15:04
mgoddard* CI status15:04
mgoddard* Wallaby release planning15:04
mgoddard  ** Debian bullseye15:04
mgoddard  ** chrony15:04
mgoddard* Xena cycle planning15:04
mgoddard  ** master branch life cycle15:04
mgoddard* Open discussion15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddardI have none. Anyone else?15:04
hrwnope15:04
openstackgerritMerged openstack/kayobe-config stable/wallaby: Synchronize kayobe-config  https://review.opendev.org/c/openstack/kayobe-config/+/79210915:05
mgoddard#topic Review action items from the last meeting15:05
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)"15:05
mgoddardmgoddard email openstack-discuss about quay.io credentials15:05
mgoddardmgoddard draft proposal for new release process15:05
mgoddardno15:05
mgoddardyes15:05
mgoddard#action mgoddard email openstack-discuss about quay.io credentials15:05
mgoddard#topic CI status15:05
yoctozeptolol, that email is hard15:05
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:05
*** macz_ has joined #openstack-kolla15:06
yoctozeptodon't get me started on CI15:06
yoctozeptoof course it's RED15:06
yoctozeptobut we are patching :-)15:06
mgoddard[all distros] [docker SDK] docker py 5.0.0 removed six but imports it and fails prechecks/any functionality15:06
mgoddardfix merged to master, backporting in progress15:07
yoctozeptoon that note15:07
yoctozeptowe are trying to move to distro-provided sdk15:07
yoctozeptoit should be good enough as long as it is provided15:07
yoctozepto:-)15:07
hrw+215:07
mgoddardwell15:07
mgoddarddo we need it?15:07
mgoddardnow we have an upper limit on docker15:07
yoctozeptonicer than pip-installing15:08
mgoddardwhat happens if we need a feature from a newer version?15:08
yoctozeptothe crowds will love you - right, kevko?15:08
mgoddardor if RDO drops15:08
yoctozeptothen we drop centos support15:08
*** murphyslawbbs has joined #openstack-kolla15:08
yoctozeptowell, the newer versions seem be lacking behind in feature support anyhow15:09
yoctozeptothat's why I mentioned we are not really losing anything but using an older version15:09
mgoddardok, well let's not rush into it15:09
yoctozeptofor cgroups namespace I am directly modifying the API query contents15:09
yoctozeptook, let's move on15:10
mgoddardok, let's move on. Lots to get through15:10
yoctozeptomore interesting stuff ahead15:11
mgoddard#topic Debian bullseye15:11
*** openstack changes topic to "Debian bullseye (Meeting topic: kolla)"15:11
mgoddardwho wants to give a status update?15:11
yoctozeptothe bulls' only eye has arrived at the green station: https://review.opendev.org/c/openstack/kolla-ansible/+/79067815:11
hrwyay!15:11
yoctozeptonow checking without machined15:11
yoctozeptowe'll see15:12
yoctozeptoanyhow, we have a working solution *with* cgroups v215:12
yoctozeptowhich is awesome15:12
yoctozeptothe other issue is with the upgrade15:12
yoctozeptoovs 2.1515:12
yoctozeptojust like in UCA before we pinned it15:12
yoctozeptomnasiadka debugging this15:12
yoctozeptono idea about the current progress, mnasiadka silent recently15:13
yoctozeptoperhaps ovs broke his internets15:13
*** nikparasyr has left #openstack-kolla15:13
yoctozeptothis is worse on debian because we can't pin any "older version"15:13
yoctozeptoas this is from base bullseye15:13
yoctozeptoalso, with current knowledge, expect breakage when rdo moves to ovs 2.1515:14
yoctozepto;d15:14
mgoddardcould we use a buster repo?15:14
yoctozeptoI don't know, hrw, wdyt?15:14
yoctozeptothe other side to that isssue is that15:15
yoctozeptoit's not 100%15:15
yoctozeptoit may succeed15:15
yoctozeptoand it always succeeds in multinode15:15
hrwmgoddard: shouldn't15:15
yoctozeptoperhaps ovs likes to have friends to get to work ;d15:15
yoctozeptoand it is 80% sad when alon15:15
yoctozepto(the estimation is made up)15:16
yoctozeptoalone*15:16
mgoddardhave we spoken to debian openstack team about it?15:16
mgoddardor ubuntu even15:17
yoctozeptome not15:17
yoctozeptohrw, mnasiadka?15:17
mgoddardprobably should15:18
hrwnot me15:18
hrwI had to dig in non kolla stuff15:18
mgoddardor ask on openstack-discuss15:18
*** cah_link has quit IRC15:18
yoctozeptowell, one can argue nobody runs ovs on singlenode because it does not make much sense ;d15:19
*** cah_link1 has joined #openstack-kolla15:19
yoctozeptoI asked zigo on #debian-openstack now15:20
yoctozepto(on oftc)15:20
mgoddardok15:20
mgoddardcan we go back to debian & libvirt15:21
*** cah_link1 is now known as cah_link15:21
mgoddardhttps://docs.docker.com/engine/api/version-history/ suggests the CgroupnsMode parameter is API v1.4115:21
mgoddardanyone know how to map that to a docker version?15:22
mgoddard20.10.015:23
yoctozepto20.1015:23
mgoddardhttps://docs.docker.com/engine/release-notes/15:23
yoctozeptoi mentioned this above15:23
yoctozeptobefore the meeting15:23
yoctozepto:-)15:23
yoctozeptothey decided to change the default15:23
yoctozeptofor fun15:23
mgoddardI was elsewhere15:23
yoctozeptoand added a knob to change it back15:23
yoctozeptoso much for backward compat15:23
yoctozeptowell, at least they default to "more secure"15:24
mgoddardpeople in glass houses etc.15:24
yoctozeptoI know15:24
yoctozeptoI like to rant15:24
yoctozeptosince it's going to happen on bullseye15:25
yoctozeptowe can condition it on being on bullseye15:25
yoctozeptoand then we relax as more platforms move to cgroups v215:25
mgoddardjust so I'm clear15:25
mgoddardthis default changed in 20.1015:25
mgoddardbut only affects cgroups v215:25
yoctozeptoindeed15:25
mgoddardwhich so far only debian bullseye uses out of our supported platforms15:26
yoctozeptoexactly15:26
mgoddardk15:26
yoctozeptoand 20.10 is the only to support cgroups v2 out of the box15:26
yoctozeptoso older ones are supposed to fail / be unsupported15:26
mgoddardso we need to either set 20.10 / 1.41 as our minimum supported version, or have some check for cgroups v2 in the module15:27
yoctozeptoI say we use this know only on bullseye15:28
yoctozeptothat is easiest ;d15:28
yoctozeptoknob*15:28
hrw20.10 is available for each of our host distros15:28
hrwso we can depend 20.10 for wallaby+15:28
mgoddardcurrent min docker version is 1.10 :)15:29
yoctozeptoI would not discriminate people running non-upstream docker ;d15:29
mgoddardwell, just because it's available, doesn't mean its in use15:29
yoctozeptowell, we can bump to 18.0915:29
yoctozeptoas that seems to work15:29
mgoddarddoes it?15:29
yoctozepto1.10 is scary15:29
mgoddard+115:29
yoctozeptoyes, though checked for sure on train/ussuri15:30
hrwhttps://www.docker.com/blog/docker-1-10/ - 4th Feb 201615:30
hrwtime to upgrade indeed15:30
mgoddardbut I thought the new parameter was added in 20.10?15:30
yoctozeptoyes, that's why we limit it to bullseye which requires both 20.10 and the knob15:30
*** cah_link has quit IRC15:30
yoctozeptoand we have it dealt with15:30
mgoddardok15:31
yoctozeptobtw, it's passing without machined15:31
mgoddardcan we rely on distros not to switch to cgroupsv2?15:31
yoctozeptoI will check libvirt logs if no oddiness happened and refactor15:32
yoctozeptooh, I am pretty sure they will not15:32
yoctozeptoneither would risk pissing off enterprise users15:32
mgoddardok15:33
mgoddardadded some notes to the patch15:33
hrwubuntu 22.04 will be cg215:33
hrwsimilar with centos 915:33
yoctozeptoyes15:33
yoctozeptoI meant in their current versions15:33
mgoddardwe'll have a release that supports a migration15:33
yoctozeptoI hope I was not misread15:34
mgoddardI guess we can handle that when we get to it15:34
yoctozeptoyeas15:34
hrwcs9 will land in Xena or Yeti (I forgot dates)15:34
mgoddardLet's move on15:34
mgoddard#topic chrony15:34
*** openstack changes topic to "chrony (Meeting topic: kolla)"15:34
yoctozeptonoo15:34
yoctozeptoone more thing15:34
mgoddard#undo15:34
openstackRemoving item from minutes: #topic chrony15:34
yoctozeptoso Wallaby and Debian15:34
yoctozeptois this the release where we support both Buster and Bullseye, right?15:35
yoctozepto(on host)15:35
yoctozepto(as the images are simply bullseye)15:35
hrwyeah15:35
hrwboth can have same docker version but buster is cg115:35
yoctozeptook; should we test both in CI then? I would15:35
hrwgood point15:35
* yoctozepto is moving to debian-based setup and would love good CI coverage15:36
hrwyay15:36
mgoddardhere's what we said for ubuntu in victoria15:36
mgoddardThe Victoria release adds support for Ubuntu Focal 20.04 as a host operating system. Ubuntu users upgrading from Ussuri should first upgrade OpenStack containers to Victoria, which uses the Ubuntu Focal 20.04 base container image. Hosts should then be upgraded to Ubuntu Focal 20.04.15:36
mgoddard(from https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html)15:36
mgoddardI don't know if anyone ever tested it :)15:37
hrw;P15:37
mgoddardso if we assume the same approach for debian15:37
mgoddardwe provide bullseye based containers in wallaby15:37
yoctozeptoyes, and it looks worky15:38
mgoddardand support both buster and bullseye hosts15:38
*** dave-mccowan has quit IRC15:38
yoctozeptoit looked worky with ubuntu too15:38
yoctozeptobut we did not test it in CI15:38
yoctozeptoI need to check it15:38
mgoddardbuster host, victoria/buster containers -> buster host, wallaby/bullseye containers -> bullseye host, wallaby/bullseye containers15:39
hrwyeah15:39
mgoddardso our upgrade jobs should use buster in wallaby15:39
mgoddardand our host OS checks should allow both15:39
yoctozeptoyeah, we have done it for ubuntu15:40
yoctozeptousing bionic in upgrade15:40
yoctozeptoand focal in others15:40
yoctozeptoso let's do the same here15:40
yoctozeptobuster in upgrade15:40
yoctozeptobullseye in others15:40
mgoddardI'll add add notes to the patch15:41
yoctozepto(there is only one but I might throw more in Xena)15:41
yoctozeptothanks15:41
mgoddardcan I chrony yet?15:42
mgoddard#topic chrony15:43
*** openstack changes topic to "chrony (Meeting topic: kolla)"15:43
mgoddard#link https://review.opendev.org/c/openstack/kolla-ansible/+/79211915:43
mgoddardwallaby deprecates chrony, and disables it by default15:43
mgoddardtherefore we should clean up the container, if disabled15:43
mgoddardbut, how do we do this cleanly without losing time sync?15:44
yoctozeptogood q15:44
*** kevko_ has quit IRC15:44
yoctozeptowell, if there was chrony container to remove15:44
yoctozeptoand it worked correctly15:44
hrwhow do we handle it on fresh installs?15:44
yoctozeptothen we are very likely breaking it15:45
yoctozeptocan we do it like this15:45
yoctozeptoif we do upgrade15:45
yoctozeptoand chrony is disabled15:45
yoctozeptobut it was enabled (i.e., the playbook sees containers to go down)15:45
yoctozeptowe pause the playbook15:46
yoctozeptoand wait for user to acknowledge this15:46
yoctozeptowe can have a variable to skip this acknowledgment15:46
yoctozepto(to support automated users who read renos)15:46
yoctozeptoand also we will not pause if no containers exist15:46
yoctozeptothis way we target the right people15:46
mgoddardor people who run it twice :)15:47
yoctozeptotwice? I excluded those15:47
mgoddardwe have a time sync precheck15:48
yoctozeptoit's used in different situations15:48
mgoddardcould we use that?15:48
yoctozeptoand it would succeed right after15:48
yoctozeptowe just need to let users *know for sure*15:48
mgoddardhow long would it take to not succeed?15:48
yoctozeptoalso, I meant this pause ~> https://docs.ansible.com/ansible/latest/collections/ansible/builtin/pause_module.html15:48
yoctozeptomgoddard: I think it depends on kernel observing the clock stability15:49
yoctozeptoI noticed it being set as unsync after 24h15:49
yoctozeptono, we will not add a wait for this ;-)15:49
yoctozeptowell, the fun fact is15:50
yoctozeptothe host would have worky ntp15:50
yoctozeptoif not for kolla-ansible which broke it on purpose to get chrony on board :-)15:50
mgoddardhow about this15:51
yoctozeptoyes15:51
mgoddardcheck systemd for known ntp daemons15:51
mgoddardadd a flag to override, aka acknowledge the change15:51
mgoddardprovide a command/playbook to cleanup chrony before upgrade15:52
mgoddardso ideal workflow would be15:52
mgoddardkolla-ansible cleanup_chrony15:52
yoctozepto1) kill_my_chrony15:52
mgoddardkolla-ansible prechecks15:52
mgoddardkolla-ansible upgrade15:53
mgoddardbut, for those who ignore renos15:53
mgoddardkolla-ansible upgrade15:53
mgoddardwill check for ntp daemons before cleaning up chrony15:53
yoctozeptowell, we can always teach people a lesson to read renos15:53
mgoddardmy clients tend not to like it if I teach them a lesson...15:54
yoctozeptobut are not you the one doing their upgrades?15:54
yoctozepto(or someone else from stackhpc)15:54
mgoddardnot always15:54
yoctozeptowell, I think upgrades are the pinnacle of openstack support15:55
yoctozeptoso they should rethink their attitude15:55
yoctozeptobut I get you15:55
mgoddardI will pass on your message :D15:55
mgoddardanyway15:55
mgoddardneeds more thought, but we have some ideas15:55
mgoddard#topic master branch life cycle15:56
*** openstack changes topic to "master branch life cycle (Meeting topic: kolla)"15:56
mgoddard#link https://etherpad.opendev.org/p/kolla-release-process-draft15:56
mgoddarddid anyone read it?15:56
yoctozeptoI didn't have time to think about time frame15:56
yoctozeptobut I read it15:56
hrwI did15:56
hrwcommented even15:57
yoctozeptomy biggest concern is15:59
yoctozeptoin this simplistic view15:59
yoctozeptowe lose the ability to e.g. test bifrost master15:59
hrwnope16:00
yoctozeptoor perhaps not16:00
yoctozeptobecause I think I switch the reference forcibly16:00
mgoddardand bifrost master now has a job that uses wallaby?16:00
hrwR+9 is when we use master source instead of stable/previous16:00
yoctozeptomgoddard: no, just realised the code is replacing the reference with what is in the change16:00
mgoddardR+9 is next week16:01
yoctozeptoso I was just confusing myself and you16:01
yoctozeptoyeah, the timeframe is to be discussed really16:01
yoctozeptoand for the meeting as well16:01
openstackgerritMerged openstack/kolla-ansible stable/wallaby: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79211016:01
yoctozeptoas we missed its timeframe16:01
yoctozeptothanks mgoddard16:01
mgoddardbifrost job may use master, but we will not be testing that, so who knows if it works?16:02
yoctozeptomgoddard: I would say differently: we may break the job on bifrost queue now and not know it16:02
*** wuchunyang has quit IRC16:03
mgoddardor they may require a change on our side, but we cannot test it16:03
yoctozeptowe can test it "once"16:04
yoctozeptobut then it reverts back to stable for subsequent runs16:04
*** bengates has quit IRC16:04
mgoddardI'll put the draft onto openstack-discuss16:04
mgoddardand also announce end of feature freeze, which should have happened some time ago16:05
mgoddardthanks all16:05
mgoddard#endmeeting16:05
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:05
openstackMeeting ended Wed May 19 16:05:23 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:05
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.html16:05
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.txt16:05
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-05-19-15.00.log.html16:05
hrwthx16:05
*** bengates has joined #openstack-kolla16:05
yoctozeptothanks mgoddard16:05
* hrw out16:05
openstackgerritMark Goddard proposed openstack/kolla stable/wallaby: [Security] Fix open redirect in Prometheus  https://review.opendev.org/c/openstack/kolla/+/79216216:08
*** bengates has quit IRC16:09
*** zijlboot has quit IRC16:12
*** zijlboot has joined #openstack-kolla16:12
*** k_mouza has quit IRC16:14
*** zijlboot_ has joined #openstack-kolla16:16
*** zijlboot_ has quit IRC16:18
*** zijlboot_ has joined #openstack-kolla16:18
*** zijlboot_ has quit IRC16:18
*** zijlboot_ has joined #openstack-kolla16:18
*** zijlboot has quit IRC16:19
*** zijlboot_ has quit IRC16:25
*** rpittau|bbl is now known as rpittau16:45
*** zijlboot has joined #openstack-kolla16:53
*** zijlboot has quit IRC16:57
*** ricolin_ has joined #openstack-kolla17:03
*** jonaspaulo has joined #openstack-kolla17:06
*** fuhrmannb has quit IRC17:08
openstackgerritMerged openstack/kolla-ansible stable/train: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79215817:10
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: chrony: remove during upgrade when disabled  https://review.opendev.org/c/openstack/kolla-ansible/+/79211917:20
openstackgerritRafael Weingartner proposed openstack/kolla-ansible master: Make setup module arguments configurable  https://review.opendev.org/c/openstack/kolla-ansible/+/78339217:27
*** rpittau is now known as rpittau|afk17:37
*** waxfire has joined #openstack-kolla17:41
*** waxfire has left #openstack-kolla17:42
*** murphyslawbbs has quit IRC17:46
*** k_mouza has joined #openstack-kolla18:00
*** k_mouza has quit IRC18:05
*** andrewbonney has quit IRC18:05
*** vishalmanchanda has quit IRC18:22
*** jbadiapa has quit IRC18:30
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: chrony: remove during upgrade when disabled  https://review.opendev.org/c/openstack/kolla-ansible/+/79211918:43
openstackgerritRafael Weingartner proposed openstack/kolla-ansible master: Make setup module arguments configurable  https://review.opendev.org/c/openstack/kolla-ansible/+/78339218:47
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: chrony: cleanup during overcloud host upgrade  https://review.opendev.org/c/openstack/kayobe/+/79224318:48
*** dking has joined #openstack-kolla18:56
*** k3nny0ne has joined #openstack-kolla18:58
*** k3nny0ne has quit IRC18:59
*** k3nny0ne has joined #openstack-kolla19:01
*** mushrushu has quit IRC19:25
*** mushrushu has joined #openstack-kolla19:26
*** samcat116 has joined #openstack-kolla19:46
*** e0ne has quit IRC20:01
openstackgerritMerged openstack/kolla-ansible stable/ussuri: baremetal: Install Docker SDK less than 5.0.0  https://review.opendev.org/c/openstack/kolla-ansible/+/79215720:08
*** zul has quit IRC20:09
*** e0ne has joined #openstack-kolla20:14
*** k_mouza has joined #openstack-kolla20:15
« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!