Wednesday, 2023-08-23

opendevreviewhowardlee proposed openstack/kolla-ansible master: freezer-scheduler config dir permission change  https://review.opendev.org/c/openstack/kolla-ansible/+/89204501:47
opendevreviewhowardlee proposed openstack/kolla-ansible master: change venus service-type to log-management  https://review.opendev.org/c/openstack/kolla-ansible/+/89245105:36
opendevreviewhowardlee proposed openstack/kolla-ansible master: change venus service-type to log-management  https://review.opendev.org/c/openstack/kolla-ansible/+/89245105:44
opendevreviewMichal Nasiadka proposed openstack/kolla master: toolbox: Add sleep script handling signals  https://review.opendev.org/c/openstack/kolla/+/89177708:55
opendevreviewMaksim Malchuk proposed openstack/kolla stable/2023.1: docs: we moved UNBUILDABLE_IMAGES to separate file  https://review.opendev.org/c/openstack/kolla/+/89241009:16
opendevreviewMaksim Malchuk proposed openstack/kolla stable/zed: docs: we moved UNBUILDABLE_IMAGES to separate file  https://review.opendev.org/c/openstack/kolla/+/89241109:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers  https://review.opendev.org/c/openstack/kolla-ansible/+/86478009:53
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Log details about failed containers  https://review.opendev.org/c/openstack/kolla-ansible/+/89248810:04
glukaCBSHello. It's my first chat on irc since 2 decades. My name is Gheza Luka, I'm involved currently in Openstack upgrades and woulk like to raise a question to community10:05
opendevreviewMerged openstack/kayobe master: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/88779210:05
glukaCBSrelated to default value of 2 cores for rabbitmq container, #rabbitmq_server_additional_erl_args: "+S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none", are these values recommended for production? we have a 32 core server for controller plane10:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892910:11
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Log details about failed containers  https://review.opendev.org/c/openstack/kolla-ansible/+/89248810:13
opendevreviewRafal Lewandowski proposed openstack/kayobe stable/zed: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89247810:20
opendevreviewRafal Lewandowski proposed openstack/kayobe stable/yoga: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89247910:21
opendevreviewRafal Lewandowski proposed openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89248010:22
opendevreviewRafal Lewandowski proposed openstack/kayobe stable/xena: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89250110:22
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892910:23
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry  https://review.opendev.org/c/openstack/kayobe/+/89248010:53
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers  https://review.opendev.org/c/openstack/kolla-ansible/+/86478011:18
* SvenKieske took the liberty to add https://review.opendev.org/c/openstack/kolla-ansible/+/874769 to the agenda; any feedback would be nice, as this patch is sitting there for quite a while and I'm fairly certain it's already running in production somewhere :)11:44
fricklerI really appreciate the detailed commit message, it at once makes clear that this is not a quick-and-dirty review ;)11:53
SvenKieskeyeah, maybe some of the implementation should also be changed (up for debate) but in general this is a very useful feature/better approach than we currently have11:56
SvenKieskeI just remembered it, because auf mnasiadka's change regarding running ovs agents in seperate containers which is a little bit of a similar problem11:57
SvenKieskenot really similar bud adjacent11:57
SvenKieskebut*11:57
mnasiadkaneutron agents restarts will never be graceful ;)12:02
mnasiadkawe run processes in a container and dumb-init only kills the single child12:02
mnasiadkabtw, debian jobs are failing, because we uploaded bookworm images as bullseye :)12:05
mnasiadkaand todays publish job failed with some pip download issues, so basically we need to wait until tomorrow12:06
SvenKieskecan't these things be triggered manually by someone in infra? fungi?12:07
mnasiadkaSvenKieske: well, with this approach they might be, but I'm afraid it's going to be flakey at least :)12:08
mnasiadkaSvenKieske: we can wait, 24 hours is not going to kill anybody12:08
SvenKieskemnasiakda: look at the patch: it disables all agents before restarting them and even checks routers, our current approach just set's an arbitrary timeout timer afaik which is dependent on the number of routers in your deployment. if you use an actual cloud where routers come and go by the minute it's not feasible to always adjust your kolla config by the current number of routers to calculate the correct 12:09
SvenKiesketimeout12:09
mnasiadkaSvenKieske: with proper approach neutron-l3-agent restart should not affect routing functionality12:10
mnasiadkaand that dance is going to fail sooner or later with some other issues12:10
SvenKieskeit's really a way better approach than what we are currently doing in k-a; it's still not perfect of course, as there can always be new routers added in a small time frame, but way better then the current approach, reduced downtime from minutes to seconds IIRC.12:10
fungii'm not sure that we have a way to manually trigger jobs in a timer-based pipeline, at least i know the zuul-client wasn't able to do that in the past but i haven't revisited it in some time. i can take a look. is it just one project/branch that needs to be triggered for periodic?12:11
mnasiadkafungi: yes, kolla/master - ideally only debian publish job (kolla-publish-debian-quay)12:11
SvenKieskemnasiakda: what do you mean by a proper approach? my understanding is that this is currently by (k-a) design. afaik e.g. in triple-o this works. do you mean to change k-a design of neutron-l3-agent?12:12
fungidefinitely don't have a way to filter it by job. zuul enqueues "items" (which are a change, a commit, or a branch) and then runs the jobs defined for them in that pipeline12:13
SvenKieskefungi: too bad :/ but didn't want to drag you into this, as you will most likely have enough on your plate already :)12:13
fungino worries. this is the most recent buildset then, looks like? https://zuul.opendev.org/t/openstack/buildset/bee84cc7f35c423987d175b31cba762e12:14
fungifor kolla-publish-debian-quay the kolla-build step failed with an error "Images not buildable due to build options"12:16
SvenKieskemnasiadka: I guess you mean to properly separate the l3-agent spawned processes into their own (docker)namespaces and thereby solving this problem via triggered restarts? Then I agree. This is also mentioned in the commit message, but was deemed to be too much work, so this work around got implemented back then.12:18
fungiif you confirm it's okay to rerun the jobs in that buildset, i can try to reenqueue now12:18
mnasiadkafungi: I confirm12:18
fungiokay, i'll give it a shot, just a sec12:18
mnasiadkawhat is interesting, it seems it was some resurface of the pypi problem - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_33b/periodic/opendev.org/openstack/kolla/master/kolla-publish-debian-quay/33b7a15/kolla/build/000_FAILED_kolla-toolbox.log12:18
fungiseems to have enqueued something12:18
fungiit shows the commit as 0 though, and then errors out the builds12:19
fungilooks like it may not correctly pass the branch refname. i'll try another way12:20
*** glukaCBS_ is now known as glukaCBS12:21
SvenKieskethose pypi mirror problems are really persistent :/12:21
fungithis time i forced a revision equivalent to `git show-ref origin/master` and it seems to be running builds12:22
fungizuul-client enqueue-ref --tenant=openstack --pipeline=periodic --project=openstack/kolla --ref=refs/heads/master --newrev=4c31f7a3f2002d77dd715dfbb5c2eb74192149d412:22
fungithat's what i ran12:23
fungikolla-publish-debian-quay is running12:23
fungihttps://zuul.opendev.org/t/openstack/stream/4fcbd43abb654cdebef1580fc97a8c26?logfile=console.log is the console stream12:23
SvenKieskefungi: would that have been something we could've done ourselves? I'm not entirely sure about who has what permissions to run stuff.12:25
fungino, it requires zuul admin credentials to authenticate the client12:25
mnasiadkafungi: thanks :)12:25
fungii'm just specifying what i did because opendevmeet in here is taking copious notes of our conversation12:26
fungithe kolla-build task is running currently12:26
fungiwhich is what failed in the previous builds12:26
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers  https://review.opendev.org/c/openstack/kolla-ansible/+/86478012:37
fungilooks like it got past the kolla-build task this time and pushed to quay12:40
SvenKieskethat patch somehow broke the gerrit gui btw, can't see any activity in the changelog before patchset 2312:40
fungiwhich patch, 864780?12:40
SvenKieskeyep12:40
fungigerrit hides older resolved comments and activity. click the "show all entries" toggle at the top of the change log12:41
SvenKiesketry to click "expand all" in the changelog, there's only a weird zuul error, I guess because the project.yaml contained invalid syntax12:41
fungilook on the left side for "show all entries"12:41
fungiyou want to display the hidden entries, not just expand the visible ones12:42
SvenKieskeah ty, that toggle doesn't look like a toggle..weird, I  was certain I have used this functionality many times in the past without problems.12:43
fungithe ui changes subtly from one release of gerrit to the next, and we upgrade semi-frequently12:43
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting in 9 minutes12:51
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/89232312:55
mnasiadka#startmeeting kolla13:00
opendevmeetMeeting started Wed Aug 23 13:00:11 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
opendevmeetThe meeting name has been set to 'kolla'13:00
mnasiadka#topic rollcall13:00
mnasiadkao/13:00
mmalchuk\o13:00
bbezak[o]13:01
SvenKieskeo/13:01
mhinero/13:01
frickler\o13:03
mnasiadka#topic agenda13:03
mnasiadka* Review action items from the last meeting13:03
mnasiadka* CI status13:03
mnasiadka* Release tasks13:03
mnasiadka* Regular stable releases (first meeting in a month)13:03
mnasiadka* Current cycle planning13:03
mnasiadka* Additional agenda (from whiteboard)13:03
mnasiadka* Open discussion13:03
mnasiadka#topic Review action items from the last meeting13:03
mnasiadkamnasiadka to send a mail to ML and drop openvswitch-netcontrold13:03
mnasiadkadone13:03
mnasiadka#topic CI status13:03
mmalchukbroken13:04
mnasiadkaA bit red, because we uploaded images to quay.io with bullseye tag, but they have bookworm inside - so kolla-ansible-debian jobs are broken13:04
mmalchukinvalid configuration13:04
fricklermagnum job is still broken with designate testing. I made a bit of progress but no success yet13:04
mnasiadkawill be fixed by https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892323 and kolla-ansible side13:04
mnasiadkawill shout when I'll need reviews :)13:04
fricklerdo we need to delete those wrong images?13:05
mnasiadkaYes, have a script ready to fire off once we move :)13:05
SvenKieskeit would be nice, there could be accidental users, no?13:06
SvenKieskeah cool :)13:06
mnasiadka#topic Release tasks13:06
mnasiadkadid we merge the UCA patch for Bobcat?13:06
mnasiadkayes13:06
mnasiadkano RDO for Bobcat yet, we'll need to wait for the coordinated release13:07
mnasiadka#topic Current cycle planning13:07
mnasiadkaSo, Bookworm followups are on it's way, we built the images and deploy them on Bullseye host OS,  now the patches that use those images need to pass and we should be sort of done13:08
mnasiadkaLet's Encrypt - didn't have time to have a look13:08
mnasiadka#link https://review.opendev.org/c/openstack/kolla/+/88734713:09
mnasiadkathat one should be first to get in13:09
mnasiadka#link https://review.opendev.org/q/topic:bp%252Fletsencrypt-https13:09
mnasiadkakevko: are you going to address hrw's comments?13:09
mnasiadkaah, kevko is not with us13:09
mnasiadkaonce all comments are addressed - I guess we should go forward with this13:10
mnasiadkaSecond one is podman13:10
mnasiadkamhiner: still having problems on Rocky 9?13:10
mhinernope, that was solved yesterday, I think13:10
mnasiadkanice13:11
mnasiadkado we have a topic for all the patches we need to merge?13:11
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/89232313:11
mhinerwe have topic for all podman related patches - kolla-podman13:13
SvenKieskehttps://review.opendev.org/q/topic:kolla-podman13:13
mnasiadka#link https://review.opendev.org/q/topic:kolla-podman13:14
SvenKieskethat would've been handy for bookworm too, btw :)13:14
opendevreviewMichal Nasiadka proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+  https://review.opendev.org/c/openstack/kolla/+/82578613:15
mnasiadkaok, so we have those two topics to prioritise reviews on13:15
mnasiadka#topic Additional agenda (from whiteboard)13:15
mnasiadkaoctavia jobboard (frickler)13:16
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/88858813:16
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/88858813:16
mnasiadkafrickler: can we rework it a bit to enable it only when amphora is enabled? we have clouds that only run ovn-octavia-provider and would be a shame to enforce redis on them when it's not needed13:17
mnasiadkanext one is Enable RabbitMQ HA queues by default by mattcrees 13:19
SvenKieskeI'd say it seems reasonable, but I can't speak for fricker :)13:19
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/88282513:19
mnasiadkaneeds second core13:20
mnasiadkaand SvenKieske: asking for review/feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/87476913:20
fricklersorry, was distracted for a second, I can update the jobboard patch, yes13:20
mnasiadkafrickler: thanks13:20
mnasiadkaSvenKieske: commented, will keep an eye on this - I'm not convinced it will not fail under some circumstances - but can understand it's better than what we have today (and as long as it's not default it should be ok) - although we're not testing that in CI13:23
mnasiadkaand maybe we should13:23
SvenKieskenice, thanks :) sure there are still cases not covered by this :)13:24
fricklerbut certainly an interesting topic, I'll try to do a deeper review, too13:24
mnasiadkathere's also my tripleo-like patch with a lot of dust on it, I'll try to revive it - because that would enable us to do agent restart without really affecting data path13:25
SvenKieskegot a link for that? I'm curious13:25
mnasiadka#link https://review.opendev.org/c/openstack/kolla-ansible/+/86478013:26
mnasiadkait runs haproxy/keepalived/etc processes in separate containers13:26
SvenKieskenice13:27
mnasiadkaok, let's move to open discussion13:27
mnasiadka#topic * Open discussion13:27
SvenKieskeah that's the one I saw earlier, yeah pretty good :)13:27
mnasiadkaAnybody anything?13:27
mmalchukserver-status for RH13:27
mmalchukhttps://review.opendev.org/c/openstack/kolla/+/89109813:27
mmalchukand reviews in Kayobe:13:28
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/86139713:28
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/879554/1213:28
mnasiadkabbezak: can you look at the server-status one? ^^ (it's RH, so it's on us)13:28
mnasiadkabbezak: and the kayobe ones as well13:28
mmalchukthanks in advance13:30
mnasiadkaok, anyway SHPC needs to take this13:30
mnasiadkaany other topics?13:30
mmalchukanother one is a Gnocchi13:30
mmalchukanyone can push the author to make tags?13:30
mmalchukfor stable branches we still unbuildable13:31
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean"  https://review.opendev.org/c/openstack/kolla/+/89085113:31
mnasiadkaI have no idea - maybe try raising an issue in their GH repo?13:32
mmalchukremember someone told here about last week13:33
mnasiadkathat wasn't me, maybe kevko 13:33
mmalchukthere was a link to issue13:33
mmalchukneed grep logs13:33
mnasiadkamrunge: can you help regarding gnocchi?13:34
mnasiadkawell, it was worth a try - maybe he'll respond later13:35
mnasiadkaok, I guess that's it for today13:35
mnasiadkathanks for coming!13:35
mnasiadka#endmeeting13:36
opendevmeetMeeting ended Wed Aug 23 13:36:00 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:36
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.html13:36
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.txt13:36
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.log.html13:36
mmalchukoh... here is a #gnocchi - I'll ask there13:36
mmalchukthanks mnasiadka 13:36
SvenKieskethank you, mnasiadka13:36
bbezakwill take a look mnasiadka13:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers  https://review.opendev.org/c/openstack/kolla-ansible/+/86478013:51
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/89232313:51
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering  https://review.opendev.org/c/openstack/kolla-ansible/+/86892913:58
opendevreviewErik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly  https://review.opendev.org/c/openstack/kolla-ansible/+/89242714:29
opendevreviewErik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly  https://review.opendev.org/c/openstack/kolla-ansible/+/89242714:32
fungimnasiadka: looks like the image upload jobs all completed. are things any better now?14:45
mmalchukfungi you are talking about absent images on quay.io ?15:05
mmalchukfloks, is this the current issue: https://quay.io/repository/openstack.kolla/fluentd?tab=tags&tag=latest ?15:06
mmalchukthere only master, 2023.1 and zed15:07
fungimmalchuk: no idea, i'm talking about whatever problem mnasiadka requested rerunning the previously failed periodic jobs to resolve15:07
mmalchuklooks like this is the same or related issue15:08
fricklermmalchuk: what exactly is your issue? if you are looking for older images, they are in different namespaces, like https://quay.io/repository/openstack.kolla/ubuntu-source-fluentd15:20
mmalchukoh... indeed15:23
-opendevstatus- NOTICE: Gerrit is going to be restarted to pick up a small config update. You will notice a short outage of the service.15:33
greatgatsbywe've found that our provider aggregates can get out of sync with or normal host aggregates.  It seems there's a `nova-manage placement sync_aggregates` command to re-sync, but what would cause them to get out of sync in the first place?  Is there something we can watch out for that might cause this drift?16:27
fricklergreatgatsby: that would sound more like a nova issue than something kolla could fix?16:49
greatgatsbyfrickler: ok, thanks.  Was hoping is was just general knowledge (that I don't have yet)  :-)17:09
opendevreviewErik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly  https://review.opendev.org/c/openstack/kolla-ansible/+/89242717:26
frickleromg now this fips *beepstuff* is coming to kolla, too18:19
supamattfips ;D18:34
supamattwhere did you see that frickler ?18:34
johnsomLet's just say, I feel your pain18:40
fricklerhttps://bugs.launchpad.net/bugs/203282718:42
supamattusing hardened ssh keys is probably not a bad thing18:51
frickleragreeing on what counts as hardened may not be so easy, though. removing dsa is fine by me, not allowing ed25519 would be much more controversial19:39

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!