Wednesday, 2023-01-25

gmannmnasiadka: frickler: mgoddard: can we merge this if all good in this https://review.opendev.org/c/openstack/kolla-ansible/+/870879/402:18
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: CI: Add Kayobe RL9 jobs  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87168107:08
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161007:39
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161007:42
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161007:43
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: CI: Add Kayobe RL9 jobs  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87168107:46
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161007:58
kevkomnasiadka: I don't think this will work 08:01
mnasiadkakevko: works on a local Jammy install08:01
kevkomnasiadka: really ? asc format ? 08:01
mnasiadkayes, it's dumb, but it works08:02
kevkomnasiadka: local yammy08:04
kevkoFetched 133 MB in 25s (5,352 kB/s)                                                                                                                                                                                08:04
kevkoReading package lists... Done08:04
kevkoBuilding dependency tree... Done08:04
kevkoReading state information... Done08:04
kevko60 packages can be upgraded. Run 'apt list --upgradable' to see them.08:04
kevkoN: Skipping acquire of configured file 'stable/binary-i386/Packages' as repository 'https://download.docker.com/linux/ubuntu jammy InRelease' doesn't support architecture 'i386'08:04
kevkoroot@pixla:/home/michalarbet/ultimum/git/upstream/ansible-collection-kolla# cat /etc/apt/sources.list.d/docker.list08:05
kevkodeb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu jammy stable08:05
kevkomnasiadka: let's see ...i have another revision if it will not work 08:06
kevkomnasiadka: from zuul ->  Destination /etc/apt/keyrings is not writable",08:10
mnasiadkaguess it needs become08:10
kevkomnasiadka: yeah08:11
opendevreviewMichal Arbet proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161008:18
hrwmorning08:21
hrwkevko: are you moving functionality present in kolla to a-c-kolla?08:22
hrwyeah, kind of same08:22
kevkohrw: no, I was trying wallaby (where ansible/roles/baremetal is present - and this task is also present there) ..and noticed that this should be fixed as it is deprecated, but when i was trying to find that task in master i just realized that this *was* moved to ansible-collection (role baremetal)08:23
kevkoso i was going to fix a-c-kolla :) 08:24
hrwkevko: template/repos.yaml rewrite was called similar way08:24
hrwkevko: I would rather use template and create docker.source file08:25
kevkohrw: yeah, i've stolen your git commit message :D :D :D 08:25
hrwwith this multiline format of repo08:25
hrwanyway, your way works too ;d08:25
mnasiadkakevko: I think you went from asc to gpg once again :)08:28
mnasiadkaanyway around a-c-kolla - these three are waiting: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/871645 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/871646 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87168108:29
hrwboth asc and gpg works08:31
mnasiadkagpg needs dearmor on docker key, asc doesn't ;)08:34
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Add docker_systemd_reload functionality  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87129308:43
mnasiadkawell, actually seems .gpg works as well09:15
kevkomnasiadka: hmm, it is opposite no ? 09:16
mnasiadkawell, on my local jammy instance, .gpg did not work, .asc did, but if this works in CI, maybe I did something wrong ;)09:16
kevkoasc is -> "A ASCII-armored representation of key " - so you need dearmor , .gpg is GNU Privacy Guard public keyring file, binary format. 09:18
kevkoand you need to have binary format i think09:18
kevkomnasiadka: well, maybe apt is recongnizing filetype depends on ".gpg" or ".asc" and then use it ..09:20
kevkoi thought that there is needed to have binary format ..that's the reason why i used --dearmor ..if .asc did the trick ..it's ok probably09:22
mnasiadkawell, the current version of the patch works in CI, so I don't see why it shouldn't work anywhere else09:22
hrwapt goes by extension, yes09:38
hrwthat's why docker/base/Dockerfile.j2 has key for it in table of keys09:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/yoga: Add CentOS Stream 9 / Rocky Linux 9 host support  https://review.opendev.org/c/openstack/kolla-ansible/+/86492009:41
kevkohrw: mnasiadka: it's working ..i am just saying there is ASCI armored file downloaded from download.docker.com/linux/{distro}/gpg ..and its name is .gpg ..which is not correct ..so i don't understand how it works :D 09:41
hrwah, ok09:41
mnasiadkalol09:42
mnasiadkaso should it be named .asc? :D09:42
kevkobut then it is not working i think 09:42
kevkolet me try :D 09:42
kevkoi think it  should :D 09:45
kevkohttps://asecuritysite.com/rsa/pgp1 < format 09:45
kevkodownloaded docker key -> https://paste.openstack.org/show/bOTBxeu6MvEosl6fkemq09:46
opendevreviewMichal Arbet proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161009:49
kevkoi am detailistic :) 09:49
opendevreviewMerged openstack/ansible-collection-kolla master: CI: Rename source jobs  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87164509:55
opendevreviewMerged openstack/ansible-collection-kolla master: CI: Add Kayobe RL9 jobs  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87168109:56
opendevreviewVerification of a change to openstack/ansible-collection-kolla master failed: CI: add aarch64 kolla-ansible job  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87164609:56
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: CI: add aarch64 kolla-ansible job  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87164610:02
opendevreviewMichal Arbet proposed openstack/ansible-collection-kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161010:02
kevkohrw: what about +2 and +2 from you for https://review.opendev.org/c/openstack/kolla-ansible/+/865434 simple patch ? :P 10:03
kevko+w10:04
hrwdone10:04
mnasiadkaok, kevko - can you followup with some docs? :)10:05
kevkomnasiadka: for which patch  10:12
kevkomnasiadka: do you mean rabbitmq options ? :D ..i can ..i am not goog in docs :D and i hate it ..but yeah probably I can :D 10:13
mnasiadkayeah, for rmq :)10:19
kevkomnasiadka: i will take a look10:43
kevkomnasiadka: btw, if you have a time ..i have another patchset which is resolving long-term bug in kolla_docker module :P 10:44
kevkohttps://review.opendev.org/c/openstack/kolla-ansible/+/87079810:44
mnasiadkakevko: will have a look, but I'm thinking if we could have any tests that would make sure it's correct now ;)10:49
kevkomnasiadka: we probably should have something for testing ..i've tested on my built environment (btw i found this bug because I realized that heat-engines are killed because even if graceful_timeout is configured in k-a it' ignored because of default timeout in a module ...)10:51
kevkomnasiadka: and this is global problem in k-a ..there are several containers killed after 10sec and they are not waiting for 60sec ..only 10 sec ...this is big problem i think10:52
mnasiadkawell, we have a bunch of kolla_docker tests11:02
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990611:19
opendevreviewMerged openstack/kolla-ansible master: Add ability to configure rabbitmq  https://review.opendev.org/c/openstack/kolla-ansible/+/86543411:20
opendevreviewMerged openstack/ansible-collection-kolla master: CI: add aarch64 kolla-ansible job  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87164611:20
opendevreviewMerged openstack/kolla-ansible master: CI: show coverage report in job  https://review.opendev.org/c/openstack/kolla-ansible/+/87147111:20
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990612:50
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko - meeting in 913:51
kevkook, then i am going for cigarette :P 13:52
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990613:54
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Jan 25 14:00:00 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadka#topic rollcall14:00
mnasiadkao/14:00
mhiner\o14:01
scottsolo/14:01
matfechnero/14:01
kevko\o/14:01
mmalchuk\o14:01
fricklero/14:02
hrw /o\14:02
ebbexo/14:02
bbezak\o14:02
mnasiadka#topic agenda14:03
mnasiadka* CI status14:03
mnasiadka* Release tasks14:03
mnasiadka* Regular stable releases (first meeting in a month)14:03
mnasiadka* Current cycle planning14:03
mnasiadka* Additional agenda (from whiteboard)14:03
mnasiadka* Open discussion14:03
mnasiadka#topic CI Status14:03
mnasiadkaSo, I've fixed some a-c-kolla missing CI jobs after renames14:04
mnasiadkaI'm a bit concerned about Rocky Ironic CI jobs issues, both in K-A and Kayobe14:05
mnasiadkaI'll write that down on the whiteboard, because they are failing not every time, but often14:06
mnasiadka#topic Release tasks14:06
mnasiadkaI don't think we have any14:06
mnasiadka#topic Current cycle planning14:06
mnasiadkaSo, I think it's time to merge the systemd patch in kolla-ansible14:06
mnasiadkaI tried it out, seems to be working ok, and I have a feeling if we don't merge it - we'll postpone it again and again14:06
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/81672414:07
mnasiadkafrickler, kevko, hrw, bbezak - wdyt?14:07
fricklertrial by fire, sounds reasonable14:07
kevkolet's do it 14:07
opendevreviewMerged openstack/kolla-ansible master: Adding optional delay between l3 agent restarts  https://review.opendev.org/c/openstack/kolla-ansible/+/86291614:08
mnasiadkaWe still have quite a couple of months before release, so it's time I guess14:08
bbezakmaster is for testing isn't?14:08
bbezak:)14:08
mnasiadkaAnd we would make mhiner happy14:08
mhinervery much :D14:08
hrwalways can revert14:08
mnasiadkayup14:08
mnasiadkaAdded my +2, another trial by fire participant needed.14:09
kevkoi gave also - merging :) 14:10
mnasiadkaGreat, one thing over the line, let's see if it breaks :)14:11
mnasiadkamhiner: I see there are some other patches that you could refresh/rebase now, and we could tackle them one by one14:11
mnasiadkaThe other long awaited patch is Let's Encrypt, but I didn't have time yet to do some proper testing of that.14:12
kevkoanother candidate for trial by a fire ? :D 14:13
mhineryes, they deal with preparation for podman introduction14:13
mnasiadkanot yet, for systemd patch I did have a look on a local env :)14:13
mnasiadkafor let's encrypt it needs to wait a bit, unless other cores are happy to give a meaningful review to it14:14
kevkoi really want to test letsencypt on local env (but currently i am reworking my pipelines to build en ENV ...but i am almost done ... so i will check it - i hope)14:14
mnasiadkagood14:15
mnasiadkaok, let's go further14:15
mnasiadka#topic Open discussion14:16
mnasiadkaI have a small question14:16
mnasiadkais there a reason kolla-ansible-debian-aarch64 is not voting?14:16
hrwno idea14:16
kevkomaybe because smallest group of people who is using k-a on aarch64 ? 14:17
mnasiadkawell, if it's reliable, and I think it is - we should mark it as voting14:17
mnasiadkaand we should at least try to not break it :)14:17
mnasiadka(with non-voting I have a feeling we don't really care)14:18
mnasiadkaDoes anybody care enough to mark it as voting?14:18
hrwlet me14:18
mnasiadkagood14:19
kevkodon't care 14:19
mmalchukaccording Support Matrix14:19
mmalchukwe support Debian14:19
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make debian/aarch64 voting  https://review.opendev.org/c/openstack/kolla-ansible/+/87173214:20
mmalchukso it should vote14:20
hrwhttps://review.opendev.org/c/openstack/kolla/+/871348 needs brave person for trivial change14:20
mnasiadkadebian x86 is voting, so aarch64 should as well14:20
mnasiadkaas long as we have any CI infrastructure to test it :)14:20
hrwhttps://review.opendev.org/c/openstack/kolla-ansible/+/869168 is aarch64 doc update which needs review/work14:20
hrwmnasiadka: opendev got new aarch64 nodes iirc14:21
frickleryes, working since today14:21
fricklerthough it may actually be only one physical node I think14:21
mnasiadkahrw: doc update seems ok, let's merge it14:22
mnasiadkahrw: from another front - I think you've tried with multiarch images once?14:22
hrwmnasiadka: nope14:22
frickleron another note: I'll be holidaying the next two weeks and plan to be mostly offline. so don't ping me, I'll ping you ;)14:22
hrwfrickler: have fun14:22
fricklersomeone else will have to take care of stable releases next week, too14:22
hrwwho goes to fosdem?14:23
kevkowhat is fosdem ? 14:24
hrwfosdem is two weeks conference done in 2 days14:25
mnasiadkagoogle died? :) https://fosdem.org14:25
mnasiadkaok, any other topics?14:26
* hrw not14:26
* kevko not 14:27
fricklermaybe quickly mention the recent nova+cinder vuln14:27
fricklerin case anyone missed it14:27
kevkolink ? 14:28
mmalchukwhich one?14:28
fricklerhttps://launchpad.net/bugs/199618814:29
mnasiadka#link https://security.openstack.org/ossa/OSSA-2023-002.html14:29
fricklerand that one, yes14:29
mnasiadkathe patch backports in relevant projects are underway14:30
mmalchukwow14:30
fricklernot affecting kolla directly, but important to fix IMO for anyone having 3rd party accessing their cloud14:31
kevkowoow14:31
mmalchukthanks for info)14:31
spatelSorry to hijack thread but i have quick question, I am upgrading kolla from Xena to Yoga and i want to upgrade one components at a time (Ex: first keystone then nova and then neutron one by one. what are the best ways?)14:35
fricklerguess that's it?14:35
kevkospatel: use -t keystone etc14:35
spatelHmm! nice.. let me give it a try Thanks :)14:37
mnasiadkaOk, I think that's enough :)14:38
mnasiadkaThanks for coming!14:38
mnasiadka#endmeeting14:38
opendevmeetMeeting ended Wed Jan 25 14:38:34 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:38
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-01-25-14.00.html14:38
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-01-25-14.00.txt14:38
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-01-25-14.00.log.html14:38
mnasiadkaspatel: if you use OVN, then you need to upgrade all ovn components in one run14:38
mmalchukthx mnasiadka 14:39
fricklerthx for chairing mnasiadka 14:39
spatelI am using OVN 14:39
spatelso i need to do -i multinode upgrade? 14:39
mnasiadkayes, just make sure ALL hosts with ovn components are upgraded in one run, because old ovn-controller can't connect to newer OVN-NB/SB databases14:40
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892914:41
mnasiadkahrw, frickler, bbezak: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/871610 if you have time :)14:48
hrwdone14:49
spatelmnasiadka oh!! 14:50
spatelThank you! for heads up14:50
mnasiadkaspatel: https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html14:52
spatelHmm! that is interesting, what if someone has 300 compute nodes in that case we have to do all in same time? 14:55
kevkospatel: --limit host   :) ? 14:55
spatelBut if we upgrade controller nodes then all compute nodes won't talk to controller until we upgrade them correct?14:56
spatelAs per mnasiadka mentioned that it should be all in ovn case14:56
kevkospatel: well, it depends on what component you are upgrading 14:57
spatelOther components doesn't care about backward compatibility correct? only OVN care about running same version across the board 14:58
spatelIn current environment i am doing single node per day instead doing upgrade all same day. so i pick few compute nodes today and then other next day and so no.. 14:59
kevkospatel: it depends ..you have to have some experiences ..for example rabbitmq is also needed to upgrade at once ..because of erlang/rabbitmq14:59
spatelBut rabbit is just 3 node so easy enough. 14:59
kevkospatel: for example neutron control plane +1 ver and agents -1 ver is compatible ..also nova-control plane and nova-compute plane14:59
spatelWe have 325 nodes in cluster and i can't upgrade everything in single day. that is my issue 15:00
kevkowell, i am not expert in ovn (we are still using ovs ) ... but maybe you should split your cloud into smaller ones ? 15:01
kevko(if there is really no option to upgrade once by once )15:01
spatelkevko haha i have 5 cloud and all has around 200 to 300 nodes. how much i should split :)15:02
mnasiadkaspatel: yes, correct - and ovn-controller will drop all rules - so networking will stop ;-)15:18
mnasiadkaspatel: so if you want to split, you need to do ovn-controllers first, and then the SB/NB databases (+ northd)15:18
spatelmnasiadka you are saying first upgrade all compute nodes running ovn-controller and then go to nb/sb upgrade correct? 15:19
mnasiadkaspatel: if you want to split them - yes, but I think safest is just to run with -t openvswitch,ovn on all hosts (if you want to split by service)15:20
spateldoes it going to pause traffic for sometime? 15:23
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672415:24
mnasiadkaspatel: no, it should be fine15:25
spatelhmm okiee 15:26
opendevreviewVerification of a change to openstack/ansible-collection-kolla master failed: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87161015:43
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add util-linux to centos base packages  https://review.opendev.org/c/openstack/kolla/+/87173815:44
mnasiadkahrw: I got tired of the message that /usr/sbin/nologin is missing ^^15:46
hrwmnasiadka: did you also changed order of RUN calls?15:46
hrwmnasiadka: first install, then create users15:46
hrwno, you did not15:47
mnasiadkaargh right15:47
hrwso message is still there15:47
mnasiadkalet me fix that15:47
hrwI was looking at it one day and forgot to send. 15:47
hrwdid reset since then15:47
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add util-linux to centos base packages  https://review.opendev.org/c/openstack/kolla/+/87173815:48
mnasiadkaargh, wrong lin15:49
mnasiadkae15:49
mnasiadka:)15:49
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add util-linux to centos base packages  https://review.opendev.org/c/openstack/kolla/+/87173815:50
hrw+116:02
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add util-linux to centos base packages  https://review.opendev.org/c/openstack/kolla/+/87173816:22
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990616:32
spatelmnasiadka should we document this important notice in upgrade doc? related OVN limitation when it comes to upgrade large environment? like -t openvswitch,ovn  16:45
mnasiadkaWhy not, post a patch and we'll review it :)16:46
spatel+116:48
spatelmnasiadka quick question again if i do kolla-ansible -i multinode pull (does it going to pull latest images everyday?)16:49
spatelwhat if i want to pull specific image like nova-compute ? 16:50
mnasiadkayou can use -t nova to pull only nova images to required hosts16:51
mnasiadkabut yes, if you use quay.io - you'll get new image every day16:52
mnasiadkafor stable results, you should have your own local registry - and either pull images from quay and push into that local registry, or build Kolla images on your own16:52
spatelI am using quay.io so i am sure it will pull new images. 17:05
spatelI can push images in local registry with custom tags correct? 17:05
spateljust trying to understand how useful to have local registry, how does kolla pick images from local registry. assuming based on tag in global.yml  (openstack_release: "yoga")17:07
spatelor openstack_release: foobar17:07
mnasiadkaspatel: yes, you can even have a tag per service (like neutron) or even per container17:12
mnasiadkaneutron_tag17:12
spatelnice! let me try to play with it. 17:14
spatelfor patching etc.. i have to re-build images right? are there any doc related how to patch images and re-pack? 17:15
mnasiadkaeverything is in the docs (https://docs.openstack.org/kolla/latest/admin/image-building.html)17:17
spatel+117:19
spatelI did just run pull on my lab machine and it download images but with <none> tag, is that a correct behavior? - https://paste.opendev.org/show/bpXvqMRtYebkl2JVLp8b/17:26
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990617:44
mnasiadkaspatel: old images get none tag17:49
spatelgot it so old images automatically take none tag. 17:50
opendevreviewMerged openstack/kolla-ansible master: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672418:43
guesswhat[m]Hey, openstack application credential create test --role load-balancer_member fails to https://pastebin.com/raw/ikNSUY6G ,is it kolla or openstack default? How can I change it? Thanks21:00

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!