| opendevreview | howardlee proposed openstack/kolla-ansible master: freezer-scheduler config dir permission change https://review.opendev.org/c/openstack/kolla-ansible/+/892045 | 01:47 |
|---|---|---|
| opendevreview | howardlee proposed openstack/kolla-ansible master: change venus service-type to log-management https://review.opendev.org/c/openstack/kolla-ansible/+/892451 | 05:36 |
| opendevreview | howardlee proposed openstack/kolla-ansible master: change venus service-type to log-management https://review.opendev.org/c/openstack/kolla-ansible/+/892451 | 05:44 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: toolbox: Add sleep script handling signals https://review.opendev.org/c/openstack/kolla/+/891777 | 08:55 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: docs: we moved UNBUILDABLE_IMAGES to separate file https://review.opendev.org/c/openstack/kolla/+/892410 | 09:16 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/zed: docs: we moved UNBUILDABLE_IMAGES to separate file https://review.opendev.org/c/openstack/kolla/+/892411 | 09:16 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers https://review.opendev.org/c/openstack/kolla-ansible/+/864780 | 09:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Log details about failed containers https://review.opendev.org/c/openstack/kolla-ansible/+/892488 | 10:04 |
| glukaCBS | Hello. It's my first chat on irc since 2 decades. My name is Gheza Luka, I'm involved currently in Openstack upgrades and woulk like to raise a question to community | 10:05 |
| opendevreview | Merged openstack/kayobe master: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/887792 | 10:05 |
| glukaCBS | related to default value of 2 cores for rabbitmq container, #rabbitmq_server_additional_erl_args: "+S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none", are these values recommended for production? we have a 32 core server for controller plane | 10:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 10:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Log details about failed containers https://review.opendev.org/c/openstack/kolla-ansible/+/892488 | 10:13 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe stable/zed: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892478 | 10:20 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe stable/yoga: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892479 | 10:21 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892480 | 10:22 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe stable/xena: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892501 | 10:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 10:23 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/2023.1: Fix for seed-containers being unable to use password protected registry https://review.opendev.org/c/openstack/kayobe/+/892480 | 10:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers https://review.opendev.org/c/openstack/kolla-ansible/+/864780 | 11:18 |
| * SvenKieske took the liberty to add https://review.opendev.org/c/openstack/kolla-ansible/+/874769 to the agenda; any feedback would be nice, as this patch is sitting there for quite a while and I'm fairly certain it's already running in production somewhere :) | 11:44 | |
| frickler | I really appreciate the detailed commit message, it at once makes clear that this is not a quick-and-dirty review ;) | 11:53 |
| SvenKieske | yeah, maybe some of the implementation should also be changed (up for debate) but in general this is a very useful feature/better approach than we currently have | 11:56 |
| SvenKieske | I just remembered it, because auf mnasiadka's change regarding running ovs agents in seperate containers which is a little bit of a similar problem | 11:57 |
| SvenKieske | not really similar bud adjacent | 11:57 |
| SvenKieske | but* | 11:57 |
| mnasiadka | neutron agents restarts will never be graceful ;) | 12:02 |
| mnasiadka | we run processes in a container and dumb-init only kills the single child | 12:02 |
| mnasiadka | btw, debian jobs are failing, because we uploaded bookworm images as bullseye :) | 12:05 |
| mnasiadka | and todays publish job failed with some pip download issues, so basically we need to wait until tomorrow | 12:06 |
| SvenKieske | can't these things be triggered manually by someone in infra? fungi? | 12:07 |
| mnasiadka | SvenKieske: well, with this approach they might be, but I'm afraid it's going to be flakey at least :) | 12:08 |
| mnasiadka | SvenKieske: we can wait, 24 hours is not going to kill anybody | 12:08 |
| SvenKieske | mnasiakda: look at the patch: it disables all agents before restarting them and even checks routers, our current approach just set's an arbitrary timeout timer afaik which is dependent on the number of routers in your deployment. if you use an actual cloud where routers come and go by the minute it's not feasible to always adjust your kolla config by the current number of routers to calculate the correct | 12:09 |
| SvenKieske | timeout | 12:09 |
| mnasiadka | SvenKieske: with proper approach neutron-l3-agent restart should not affect routing functionality | 12:10 |
| mnasiadka | and that dance is going to fail sooner or later with some other issues | 12:10 |
| SvenKieske | it's really a way better approach than what we are currently doing in k-a; it's still not perfect of course, as there can always be new routers added in a small time frame, but way better then the current approach, reduced downtime from minutes to seconds IIRC. | 12:10 |
| fungi | i'm not sure that we have a way to manually trigger jobs in a timer-based pipeline, at least i know the zuul-client wasn't able to do that in the past but i haven't revisited it in some time. i can take a look. is it just one project/branch that needs to be triggered for periodic? | 12:11 |
| mnasiadka | fungi: yes, kolla/master - ideally only debian publish job (kolla-publish-debian-quay) | 12:11 |
| SvenKieske | mnasiakda: what do you mean by a proper approach? my understanding is that this is currently by (k-a) design. afaik e.g. in triple-o this works. do you mean to change k-a design of neutron-l3-agent? | 12:12 |
| fungi | definitely don't have a way to filter it by job. zuul enqueues "items" (which are a change, a commit, or a branch) and then runs the jobs defined for them in that pipeline | 12:13 |
| SvenKieske | fungi: too bad :/ but didn't want to drag you into this, as you will most likely have enough on your plate already :) | 12:13 |
| fungi | no worries. this is the most recent buildset then, looks like? https://zuul.opendev.org/t/openstack/buildset/bee84cc7f35c423987d175b31cba762e | 12:14 |
| fungi | for kolla-publish-debian-quay the kolla-build step failed with an error "Images not buildable due to build options" | 12:16 |
| SvenKieske | mnasiadka: I guess you mean to properly separate the l3-agent spawned processes into their own (docker)namespaces and thereby solving this problem via triggered restarts? Then I agree. This is also mentioned in the commit message, but was deemed to be too much work, so this work around got implemented back then. | 12:18 |
| fungi | if you confirm it's okay to rerun the jobs in that buildset, i can try to reenqueue now | 12:18 |
| mnasiadka | fungi: I confirm | 12:18 |
| fungi | okay, i'll give it a shot, just a sec | 12:18 |
| mnasiadka | what is interesting, it seems it was some resurface of the pypi problem - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_33b/periodic/opendev.org/openstack/kolla/master/kolla-publish-debian-quay/33b7a15/kolla/build/000_FAILED_kolla-toolbox.log | 12:18 |
| fungi | seems to have enqueued something | 12:18 |
| fungi | it shows the commit as 0 though, and then errors out the builds | 12:19 |
| fungi | looks like it may not correctly pass the branch refname. i'll try another way | 12:20 |
| *** glukaCBS_ is now known as glukaCBS | 12:21 | |
| SvenKieske | those pypi mirror problems are really persistent :/ | 12:21 |
| fungi | this time i forced a revision equivalent to `git show-ref origin/master` and it seems to be running builds | 12:22 |
| fungi | zuul-client enqueue-ref --tenant=openstack --pipeline=periodic --project=openstack/kolla --ref=refs/heads/master --newrev=4c31f7a3f2002d77dd715dfbb5c2eb74192149d4 | 12:22 |
| fungi | that's what i ran | 12:23 |
| fungi | kolla-publish-debian-quay is running | 12:23 |
| fungi | https://zuul.opendev.org/t/openstack/stream/4fcbd43abb654cdebef1580fc97a8c26?logfile=console.log is the console stream | 12:23 |
| SvenKieske | fungi: would that have been something we could've done ourselves? I'm not entirely sure about who has what permissions to run stuff. | 12:25 |
| fungi | no, it requires zuul admin credentials to authenticate the client | 12:25 |
| mnasiadka | fungi: thanks :) | 12:25 |
| fungi | i'm just specifying what i did because opendevmeet in here is taking copious notes of our conversation | 12:26 |
| fungi | the kolla-build task is running currently | 12:26 |
| fungi | which is what failed in the previous builds | 12:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers https://review.opendev.org/c/openstack/kolla-ansible/+/864780 | 12:37 |
| fungi | looks like it got past the kolla-build task this time and pushed to quay | 12:40 |
| SvenKieske | that patch somehow broke the gerrit gui btw, can't see any activity in the changelog before patchset 23 | 12:40 |
| fungi | which patch, 864780? | 12:40 |
| SvenKieske | yep | 12:40 |
| fungi | gerrit hides older resolved comments and activity. click the "show all entries" toggle at the top of the change log | 12:41 |
| SvenKieske | try to click "expand all" in the changelog, there's only a weird zuul error, I guess because the project.yaml contained invalid syntax | 12:41 |
| fungi | look on the left side for "show all entries" | 12:41 |
| fungi | you want to display the hidden entries, not just expand the visible ones | 12:42 |
| SvenKieske | ah ty, that toggle doesn't look like a toggle..weird, I was certain I have used this functionality many times in the past without problems. | 12:43 |
| fungi | the ui changes subtly from one release of gerrit to the next, and we upgrade semi-frequently | 12:43 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting in 9 minutes | 12:51 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892323 | 12:55 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Aug 23 13:00:11 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| mmalchuk | \o | 13:00 |
| bbezak | [o] | 13:01 |
| SvenKieske | o/ | 13:01 |
| mhiner | o/ | 13:01 |
| frickler | \o | 13:03 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * Review action items from the last meeting | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic Review action items from the last meeting | 13:03 |
| mnasiadka | mnasiadka to send a mail to ML and drop openvswitch-netcontrold | 13:03 |
| mnasiadka | done | 13:03 |
| mnasiadka | #topic CI status | 13:03 |
| mmalchuk | broken | 13:04 |
| mnasiadka | A bit red, because we uploaded images to quay.io with bullseye tag, but they have bookworm inside - so kolla-ansible-debian jobs are broken | 13:04 |
| mmalchuk | invalid configuration | 13:04 |
| frickler | magnum job is still broken with designate testing. I made a bit of progress but no success yet | 13:04 |
| mnasiadka | will be fixed by https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892323 and kolla-ansible side | 13:04 |
| mnasiadka | will shout when I'll need reviews :) | 13:04 |
| frickler | do we need to delete those wrong images? | 13:05 |
| mnasiadka | Yes, have a script ready to fire off once we move :) | 13:05 |
| SvenKieske | it would be nice, there could be accidental users, no? | 13:06 |
| SvenKieske | ah cool :) | 13:06 |
| mnasiadka | #topic Release tasks | 13:06 |
| mnasiadka | did we merge the UCA patch for Bobcat? | 13:06 |
| mnasiadka | yes | 13:06 |
| mnasiadka | no RDO for Bobcat yet, we'll need to wait for the coordinated release | 13:07 |
| mnasiadka | #topic Current cycle planning | 13:07 |
| mnasiadka | So, Bookworm followups are on it's way, we built the images and deploy them on Bullseye host OS, now the patches that use those images need to pass and we should be sort of done | 13:08 |
| mnasiadka | Let's Encrypt - didn't have time to have a look | 13:08 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/887347 | 13:09 |
| mnasiadka | that one should be first to get in | 13:09 |
| mnasiadka | #link https://review.opendev.org/q/topic:bp%252Fletsencrypt-https | 13:09 |
| mnasiadka | kevko: are you going to address hrw's comments? | 13:09 |
| mnasiadka | ah, kevko is not with us | 13:09 |
| mnasiadka | once all comments are addressed - I guess we should go forward with this | 13:10 |
| mnasiadka | Second one is podman | 13:10 |
| mnasiadka | mhiner: still having problems on Rocky 9? | 13:10 |
| mhiner | nope, that was solved yesterday, I think | 13:10 |
| mnasiadka | nice | 13:11 |
| mnasiadka | do we have a topic for all the patches we need to merge? | 13:11 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892323 | 13:11 |
| mhiner | we have topic for all podman related patches - kolla-podman | 13:13 |
| SvenKieske | https://review.opendev.org/q/topic:kolla-podman | 13:13 |
| mnasiadka | #link https://review.opendev.org/q/topic:kolla-podman | 13:14 |
| SvenKieske | that would've been handy for bookworm too, btw :) | 13:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Transition to DockerClient and docker 3.0.0+ https://review.opendev.org/c/openstack/kolla/+/825786 | 13:15 |
| mnasiadka | ok, so we have those two topics to prioritise reviews on | 13:15 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:15 |
| mnasiadka | octavia jobboard (frickler) | 13:16 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/888588 | 13:16 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/888588 | 13:16 |
| mnasiadka | frickler: can we rework it a bit to enable it only when amphora is enabled? we have clouds that only run ovn-octavia-provider and would be a shame to enforce redis on them when it's not needed | 13:17 |
| mnasiadka | next one is Enable RabbitMQ HA queues by default by mattcrees | 13:19 |
| SvenKieske | I'd say it seems reasonable, but I can't speak for fricker :) | 13:19 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 13:19 |
| mnasiadka | needs second core | 13:20 |
| mnasiadka | and SvenKieske: asking for review/feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/874769 | 13:20 |
| frickler | sorry, was distracted for a second, I can update the jobboard patch, yes | 13:20 |
| mnasiadka | frickler: thanks | 13:20 |
| mnasiadka | SvenKieske: commented, will keep an eye on this - I'm not convinced it will not fail under some circumstances - but can understand it's better than what we have today (and as long as it's not default it should be ok) - although we're not testing that in CI | 13:23 |
| mnasiadka | and maybe we should | 13:23 |
| SvenKieske | nice, thanks :) sure there are still cases not covered by this :) | 13:24 |
| frickler | but certainly an interesting topic, I'll try to do a deeper review, too | 13:24 |
| mnasiadka | there's also my tripleo-like patch with a lot of dust on it, I'll try to revive it - because that would enable us to do agent restart without really affecting data path | 13:25 |
| SvenKieske | got a link for that? I'm curious | 13:25 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/864780 | 13:26 |
| mnasiadka | it runs haproxy/keepalived/etc processes in separate containers | 13:26 |
| SvenKieske | nice | 13:27 |
| mnasiadka | ok, let's move to open discussion | 13:27 |
| mnasiadka | #topic * Open discussion | 13:27 |
| SvenKieske | ah that's the one I saw earlier, yeah pretty good :) | 13:27 |
| mnasiadka | Anybody anything? | 13:27 |
| mmalchuk | server-status for RH | 13:27 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla/+/891098 | 13:27 |
| mmalchuk | and reviews in Kayobe: | 13:28 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/861397 | 13:28 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/879554/12 | 13:28 |
| mnasiadka | bbezak: can you look at the server-status one? ^^ (it's RH, so it's on us) | 13:28 |
| mnasiadka | bbezak: and the kayobe ones as well | 13:28 |
| mmalchuk | thanks in advance | 13:30 |
| mnasiadka | ok, anyway SHPC needs to take this | 13:30 |
| mnasiadka | any other topics? | 13:30 |
| mmalchuk | another one is a Gnocchi | 13:30 |
| mmalchuk | anyone can push the author to make tags? | 13:30 |
| mmalchuk | for stable branches we still unbuildable | 13:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Revert "CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean" https://review.opendev.org/c/openstack/kolla/+/890851 | 13:31 |
| mnasiadka | I have no idea - maybe try raising an issue in their GH repo? | 13:32 |
| mmalchuk | remember someone told here about last week | 13:33 |
| mnasiadka | that wasn't me, maybe kevko | 13:33 |
| mmalchuk | there was a link to issue | 13:33 |
| mmalchuk | need grep logs | 13:33 |
| mnasiadka | mrunge: can you help regarding gnocchi? | 13:34 |
| mnasiadka | well, it was worth a try - maybe he'll respond later | 13:35 |
| mnasiadka | ok, I guess that's it for today | 13:35 |
| mnasiadka | thanks for coming! | 13:35 |
| mnasiadka | #endmeeting | 13:36 |
| opendevmeet | Meeting ended Wed Aug 23 13:36:00 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:36 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.html | 13:36 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.txt | 13:36 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-23-13.00.log.html | 13:36 |
| mmalchuk | oh... here is a #gnocchi - I'll ask there | 13:36 |
| mmalchuk | thanks mnasiadka | 13:36 |
| SvenKieske | thank you, mnasiadka | 13:36 |
| bbezak | will take a look mnasiadka | 13:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Run ML2/OVS agents processes in separate containers https://review.opendev.org/c/openstack/kolla-ansible/+/864780 | 13:51 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: debian: Add bookworm support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892323 | 13:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 13:58 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/892427 | 14:29 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/892427 | 14:32 |
| fungi | mnasiadka: looks like the image upload jobs all completed. are things any better now? | 14:45 |
| mmalchuk | fungi you are talking about absent images on quay.io ? | 15:05 |
| mmalchuk | floks, is this the current issue: https://quay.io/repository/openstack.kolla/fluentd?tab=tags&tag=latest ? | 15:06 |
| mmalchuk | there only master, 2023.1 and zed | 15:07 |
| fungi | mmalchuk: no idea, i'm talking about whatever problem mnasiadka requested rerunning the previously failed periodic jobs to resolve | 15:07 |
| mmalchuk | looks like this is the same or related issue | 15:08 |
| frickler | mmalchuk: what exactly is your issue? if you are looking for older images, they are in different namespaces, like https://quay.io/repository/openstack.kolla/ubuntu-source-fluentd | 15:20 |
| mmalchuk | oh... indeed | 15:23 |
| -opendevstatus- NOTICE: Gerrit is going to be restarted to pick up a small config update. You will notice a short outage of the service. | 15:33 | |
| greatgatsby | we've found that our provider aggregates can get out of sync with or normal host aggregates. It seems there's a `nova-manage placement sync_aggregates` command to re-sync, but what would cause them to get out of sync in the first place? Is there something we can watch out for that might cause this drift? | 16:27 |
| frickler | greatgatsby: that would sound more like a nova issue than something kolla could fix? | 16:49 |
| greatgatsby | frickler: ok, thanks. Was hoping is was just general knowledge (that I don't have yet) :-) | 17:09 |
| opendevreview | Erik Olof Gunnar Andersson proposed openstack/kolla-ansible master: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/892427 | 17:26 |
| frickler | omg now this fips *beepstuff* is coming to kolla, too | 18:19 |
| supamatt | fips ;D | 18:34 |
| supamatt | where did you see that frickler ? | 18:34 |
| johnsom | Let's just say, I feel your pain | 18:40 |
| frickler | https://bugs.launchpad.net/bugs/2032827 | 18:42 |
| supamatt | using hardened ssh keys is probably not a bad thing | 18:51 |
| frickler | agreeing on what counts as hardened may not be so easy, though. removing dsa is fine by me, not allowing ed25519 would be much more controversial | 19:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!