| opendevreview | Mark Goddard proposed openstack/kayobe stable/2023.1: Fix Dell OS6 and Dell OS9 switch configuration https://review.opendev.org/c/openstack/kayobe/+/919514 | 04:12 |
|---|---|---|
| opendevreview | Matt Crees proposed openstack/kayobe stable/2023.1: Add script to migrate to RabbitMQ quorum queues https://review.opendev.org/c/openstack/kayobe/+/919925 | 04:13 |
| mnasiadka | SvenKieske[m]: upper-constraints are limiting it? https://github.com/openstack/requirements/blob/ee26dad310f68661cc5d0b25d4d2b03076f92df4/upper-constraints.txt#L234 | 04:53 |
| mnasiadka | but we're not using it by default in ack | 04:55 |
| mnasiadka | and we're not using it in CI, so I have no clue why it's not breaking | 04:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: cinder: Add prechecks and docs for HA/clustering https://review.opendev.org/c/openstack/kolla-ansible/+/909974 | 05:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 05:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 05:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 05:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 05:37 |
| mnasiadka | SvenKieske[m]: actually it does fail | 06:32 |
| mnasiadka | https://91b9af26fab1ea0d09ab-54e0403a491d4a4f41f7ae86cc027e77.ssl.cf2.rackcdn.com/909974/7/check/kolla-ansible-rocky9/62c00fd/primary/logs/ansible/deploy-prechecks | 06:32 |
| kevko | Svenkieske: no still didn't create a bug report as it is weird ..I mean.. findings doesn't make a sense ...but if I revert this commit ..everything works https://github.com/openstack/oslo.concurrency/commit/9800d58cb3e7315a1941629c3dd961cfa3b63f25 | 06:39 |
| kevko | Svenkieske[m]: ^^ | 06:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Make CI roles passable with ansible-lint shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 07:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Make CI roles passable with ansible-lint shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 07:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 07:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Make CI roles passable with ansible-lint shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 07:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Make CI roles passable with ansible-lint shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 07:10 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: CI: Fix faling kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/920161 | 07:27 |
| sylvr | Hello! | 07:39 |
| sylvr | I think there might be an error in the generation of ironic config file on the seed container | 07:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Adapt playbooks to ansible-lint 6.22 https://review.opendev.org/c/openstack/kolla-ansible/+/920162 | 07:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: bump up ansible-lint https://review.opendev.org/c/openstack/kolla-ansible/+/894295 | 08:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Make CI roles passable with ansible-lint shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 08:01 |
| opendevreview | Merged openstack/kayobe stable/2023.1: Fix Dell OS6 and Dell OS9 switch configuration https://review.opendev.org/c/openstack/kayobe/+/919514 | 08:25 |
| r-krcek | Hello, I see that CI is failing. I've tried to track down the issue and it looks like it is this one: https://github.com/docker/docker-py/issues/3256 According to the conversation, it looks like the best option for now is to set limit for requests==2.31.0. However, I am not sure how to implement it. When looking at openstack/requirements, the upper-contraints.txt already mentions 2.31.0, but in CI 2.32.0 is getting installed. | 08:29 |
| SvenKieske | r-krcek: yes, I posted as much yesterday in the evening | 08:30 |
| SvenKieske | wait a second, let me fix it.. | 08:31 |
| SvenKieske | mhm, not so trivial as I thought.. | 08:35 |
| opendevreview | Sven Kieske proposed openstack/kolla master: CI: Fix breakage due to requests bug https://review.opendev.org/c/openstack/kolla/+/920165 | 08:43 |
| SvenKieske | not 100% sure this will actually work, let's see.. | 08:43 |
| opendevreview | Sven Kieske proposed openstack/kolla master: CI: Fix breakage due to requests bug https://review.opendev.org/c/openstack/kolla/+/920165 | 08:51 |
| mnasiadka | SvenKieske: and I think first we need to fix kolla-ansible | 08:59 |
| mnasiadka | using upper-constraints in a-c-k probably would help | 09:00 |
| SvenKieske | mnasiadka: replied | 09:01 |
| SvenKieske | ah okay | 09:01 |
| SvenKieske | I was not sure, the pipeline I saw failed during kolla build so I figured I fix that first, but might very well be wrong | 09:02 |
| SvenKieske | I actually thought we already use u-c everywhere | 09:03 |
| SvenKieske | can I just include it in tox.ini like this? https://github.com/openstack/kolla/blob/master/tox.ini#L12 | 09:05 |
| SvenKieske | and not just in testenv I guess | 09:05 |
| SvenKieske | well might be sufficient there | 09:06 |
| SvenKieske | mhm, that's already present in a-c-k | 09:06 |
| SvenKieske | I'm confused, shouldn't this already be sufficient? currently looking at tox docs.. the requirements file has requests===2.31.0 as well | 09:10 |
| SvenKieske | seems constraint resolving was changed, I hope we didn't run into some newly created behaviour there (this is some months old already) https://pip.pypa.io/en/stable/user_guide/#watch-out-for | 09:13 |
| SvenKieske | mhm, we only set env:TOX_CONSTRAINTS_FILE maybe that env gets reset somehow/is not propagated everywhere? see https://github.com/openstack/ansible-collection-kolla/blob/master/tox.ini#L11 | 09:20 |
| SvenKieske | ah nvm, that's just the default value if the file is not set | 09:26 |
| frickler | well this doesn't use tox https://review.opendev.org/c/openstack/kolla/+/920165/2/tests/playbooks/pre.yml | 09:30 |
| SvenKieske | these as well: https://github.com/openstack/kolla-ansible/blob/master/tests/setup_gate.sh#L97 | 09:33 |
| SvenKieske | should I just add "-c https://releases.openstack.org/constraints/upper/master" there? | 09:34 |
| frickler | I would rather just pin requests | 09:35 |
| frickler | the use of u-c in kolla is IMO a bad thing in general, in particular for stable branches | 09:35 |
| SvenKieske | yeah, tested locally wouldn't work with u-c actually: | 09:36 |
| SvenKieske | constraints/bin/python -m pip install -c https://releases.openstack.org/constraints/upper/master pip install 'docker<7' | 09:37 |
| SvenKieske | ERROR: Cannot install docker<7 because these package versions have conflicting dependencies. | 09:37 |
| SvenKieske | we have docker===7.0.0 in u-c :D | 09:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use upper-constraints for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920169 | 09:38 |
| mnasiadka | ah, so no uc - let me rework that in a better way | 09:38 |
| SvenKieske | well, I think it would - in theory - be nice to use it everywhere, but it seems that it's not that easy, unfortunately. I don't know where the docker 7 stuff comes from, for example. :( | 09:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Pin requests for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920169 | 09:40 |
| mnasiadka | so let's try this | 09:40 |
| frickler | also sad that docker-py >= 7 is still broken. maybe we need to fork it? ;) | 09:40 |
| mnasiadka | if this works, we could do it in a-c-k for all people | 09:40 |
| SvenKieske | ah I remember/looked up why we pinned it.. well the bugfix is there, just no release, maybe we should also switch to master/main branch instead? :D | 09:43 |
| mnasiadka | have a feeling docker-py is not getting any love | 09:48 |
| mnasiadka | we have may, no release since december | 09:49 |
| SvenKieske | yep | 09:52 |
| SvenKieske | your patch fails in setup-gate.sh https://zuul.opendev.org/t/openstack/build/3c0539dbdd4d40e0af9bc315495dedb9 | 09:53 |
| SvenKieske | need to pin it manually there as well I guess | 09:54 |
| SvenKieske | maybe define a variable at the start and reference this everywhere throughout the script, should not be necessary but is more robust when things are moved around again, there are multiple pip installs there | 09:54 |
| SvenKieske | well it's only two pip installs, so might be sufficient to just add the constraint there.. | 09:56 |
| frickler | hmm, only failing centos-9 though | 10:07 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 10:07 |
| frickler | I'm putting a hold on the debian job so we can faster iterate if more issues occur | 10:07 |
| SvenKieske | I'm out for lunch | 10:14 |
| mnasiadka | SvenKieske: setup_gate.sh is from kolla | 10:17 |
| mnasiadka | ah no | 10:17 |
| mnasiadka | I'm not awake | 10:17 |
| mnasiadka | let me fix this as well | 10:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Pin requests for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920169 | 10:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 10:24 |
| opendevreview | Olivier Delhomme proposed openstack/kolla-ansible master: kolla-ansible now uses default inventory file https://review.opendev.org/c/openstack/kolla-ansible/+/913993 | 10:25 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 10:30 |
| guesswhat[m] | Hi, is support for Ubuntu 22.04 planned anytime soon? Thanks | 11:10 |
| mnasiadka | It is, in D - and then most probably backported to C | 11:16 |
| frickler | guesswhat[m]: are you talking about 24.04? 22.04 is supported for ages already. for the former, needs somebody to work on it, so no plan yet | 11:16 |
| guesswhat[m] | frickler: oh, yes, 24.04 | 11:16 |
| mnasiadka | hmm, rhel 10 coming out first half 2025, fantastic | 11:22 |
| opendevreview | Merged openstack/kayobe master: Fix issue removing docker volumes https://review.opendev.org/c/openstack/kayobe/+/909594 | 11:24 |
| mnasiadka | frickler, SvenKieske: the pin seems to work, but I'll try to fix that for now in a-c-k so we could backport that for users | 11:35 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920172 | 11:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Pin requests to <2.32 for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920169 | 11:40 |
| SvenKieske | looks good on the surface, let's see what CI says :) | 11:50 |
| SvenKieske | deploy precheck fails on rocky9 slurp upgrade :( https://zuul.opendev.org/t/openstack/build/e08a75966e2049eaa4b0f72bd6ee9d8a/log/primary/logs/ansible/deploy-prechecks | 11:52 |
| SvenKieske | I'll reply on the review | 11:53 |
| SvenKieske | mnasiadka: might be that we need the fixes for these first on the slurp branches so slurp jobs work again? or do you just merge this through with +2 powers in those cases? (I think I saw something like this last year) | 11:55 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla stable/2023.1: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920124 | 11:58 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla stable/2023.1: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920124 | 12:04 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla stable/2023.2: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920125 | 12:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.2: Fix issue removing docker volumes https://review.opendev.org/c/openstack/kayobe/+/920126 | 12:08 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Fix issue removing docker volumes https://review.opendev.org/c/openstack/kayobe/+/920127 | 12:09 |
| frickler | those upgrade jobs are voting, so you can't simply override them. let's hope fixing in 2023.1 first does work, though | 12:36 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 12:45 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting in 5 | 12:55 |
| mnasiadka | frickler, SvenKieske: actually https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920172 passed | 12:57 |
| mnasiadka | so we could merge that, and then fix setup_gate.sh in reverse order | 12:57 |
| SvenKieske | alright, very nice! :) | 12:59 |
| frickler | no upgrade jobs on that repo, yes | 13:00 |
| SvenKieske | now if only upstream docker-py could do a release sometime, other people with different problems are also already asking. | 13:00 |
| mnasiadka | #startmeeting kolla | 13:01 |
| opendevmeet | Meeting started Wed May 22 13:01:23 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:01 |
| mnasiadka | #topic rollcall | 13:01 |
| SvenKieske | another solution would be to move more to podman, at least in the last few months it didn't break quite as much ;) | 13:01 |
| SvenKieske | o/ | 13:01 |
| r-krcek | o/ | 13:01 |
| mnasiadka | o/ | 13:01 |
| mhiner | o/ | 13:01 |
| frickler | \o | 13:01 |
| mnasiadka | well, given the podman bug in Ubuntu 22.04 that breaks bifrost - I'm not moving anywhere yet ;) | 13:02 |
| mmalchuk | o/ | 13:02 |
| SvenKieske | need to write our own container-py I guess :D | 13:02 |
| mattcrees | o/ | 13:03 |
| mnasiadka | #topic agenda | 13:04 |
| mnasiadka | * CI status | 13:04 |
| mnasiadka | * Release tasks | 13:04 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:04 |
| mnasiadka | * Current cycle planning | 13:04 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:04 |
| mnasiadka | * Open discussion | 13:04 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | bleeding red | 13:04 |
| mnasiadka | requests 2.32 all over the place | 13:04 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Add Ironic Python Agent (IPA) NTP parameter configuration https://review.opendev.org/c/openstack/kayobe/+/895199 | 13:05 |
| mnasiadka | being fixed in https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920172 and some others | 13:05 |
| frickler | so hopefully under control | 13:08 |
| mnasiadka | #topic Release tasks | 13:08 |
| mnasiadka | mattcrees: in absence of Will - can you share the status of Kayobe rc1? | 13:09 |
| mattcrees | The Anbile bump patch is the main priority: https://review.opendev.org/c/openstack/kayobe/+/910513 but there's some prior patches to resolve the SLURP jobs that need merging first: https://review.opendev.org/c/openstack/kayobe/+/919925 https://review.opendev.org/c/openstack/kayobe/+/913878 | 13:10 |
| mnasiadka | ok then - and the failures in the last one is requests version fail? | 13:11 |
| mattcrees | No the failure in the last one is because the script needs merging in 2023.1 first | 13:12 |
| mnasiadka | ok, you have my +2 on the script | 13:13 |
| mnasiadka | can you chase someone so it gets merged today? | 13:13 |
| mnasiadka | and you can continue the work? | 13:13 |
| mattcrees | Sure thing | 13:13 |
| mnasiadka | thanks | 13:14 |
| mnasiadka | #topic Current cycle planning | 13:14 |
| mnasiadka | I tried to remove rp-1 from all patches that we needed | 13:14 |
| mnasiadka | I'll try to go through some features plan and also existing patches that didn't go into Caracal | 13:14 |
| mnasiadka | and will start RP+1 those that are ready for reviews | 13:15 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Bump up Ansible supported versions to 8.x/9.x https://review.opendev.org/c/openstack/kayobe/+/910513 | 13:15 |
| mnasiadka | Will also tidy up the whiteboard so we have clear priorities for this cycle - which we could discuss next week | 13:15 |
| mnasiadka | But for now priority is getting out of requests 2.23 (or whatever the version is) hell | 13:16 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:16 |
| mnasiadka | (MatusJenca): | 13:17 |
| mnasiadka | core reviewers, please look at my patch https://review.opendev.org/c/openstack/kolla-ansible/+/915901 | 13:17 |
| mnasiadka | I think that's reviewed - kevko's point is valid - when it will be resolved we can go forward | 13:17 |
| SvenKieske | yes, discussed this with kevko here, seems like a good simplification | 13:18 |
| SvenKieske | I'll ping Matus | 13:18 |
| mnasiadka | (mhiner): migration patch https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 13:18 |
| mnasiadka | Best effor removal of old container engine files? Sometimes there are leftover processes running there, which prevents the removal. | 13:18 |
| mnasiadka | I am unable to make migration work (in Zuul) while VMs are alive, the migration has to be done with stopped VMs. This is outlined in documentation. | 13:18 |
| mnasiadka | Thoughts on this? | 13:18 |
| mhiner | also, after I fix up linter issues it's now ready for review | 13:19 |
| mnasiadka | From my perspective - we should just support host destroy - and then deploy with podman - and have some docs that it should be a rolling process for the cloud | 13:19 |
| mnasiadka | Because I see there's more and more problems | 13:19 |
| mnasiadka | Does it make sense? | 13:19 |
| mhiner | I dont really understand what you mean | 13:20 |
| frickler | same as distro upgrades I'd say | 13:20 |
| frickler | mhiner: doing a host redeploy to change the container engine instead of trying to do it in place? | 13:21 |
| mnasiadka | mhiner: you're trying to support migration - I'm saying we shouldn't support migration in that form - just ask the user to remove all containers and volumes from a host - and then do a re-deploy with podman | 13:21 |
| mnasiadka | probably the destroy function limited to one host should work | 13:22 |
| SvenKieske | would a mixed deployment work? not really, would it? (so some hosts still with docker, new ones with podman) | 13:22 |
| frickler | why should that not work? | 13:22 |
| SvenKieske | well I think it would work in the sense you could manually destroy host $foo, redeploy it, but then you have switched container engine and can't manage any remaining docker hosts until you have finished the migration | 13:23 |
| SvenKieske | at least that caveat should be documented I think, it might surprise some users? :) | 13:23 |
| mhiner | mnasiadka: but isn't the objective to preserve all the volumes with their data, so it can be used for re-deploy? | 13:24 |
| SvenKieske | otherwise looks like this is easier doable | 13:24 |
| mnasiadka | mhiner: It's your objective, but I don't think it's the project objective :) | 13:25 |
| mnasiadka | with distribution upgrades you reinstall the OS completely and then run a deploy to get containers in place | 13:25 |
| mnasiadka | for controllers you need to have HA - so that data will be replicated | 13:25 |
| mnasiadka | computes should not have any data besides nova_compute volume - but that's ephemeral storage | 13:26 |
| mhiner | I see, so instead of this patch we should just have some document detailing how to do the migration? | 13:26 |
| kevko | \o | 13:27 |
| mnasiadka | mhiner: yup, and some testing would be nice of course | 13:27 |
| kevko | sorry ..another meeting .. but now i am here ..let me re-read | 13:27 |
| SvenKieske | well it certainly would be nice if mhiner's objective would work, would be a really smooth transition for users, but it seems it's rather hard to achieve, so maybe just implement the "easier" solution first? | 13:27 |
| mhiner | I mean, in its current form, where you have to stop VMs beforehand, it works | 13:28 |
| mhiner | even our CI tests for it passed | 13:28 |
| SvenKieske | so you need to stop workload VMs? I didn't follow the last patches I think, why is that necessary? | 13:28 |
| SvenKieske | because of the vm volumes? | 13:29 |
| mnasiadka | well, if we have something working - just the user needs to evacuate the instances from that host - probably that's fine as well - but needs to be documented | 13:29 |
| kevko | podman migration path is the same as for quorum queue ...even if that script will land kolla ..you will be afraid to just run it :D | 13:30 |
| SvenKieske | yes, if that's really working with only the caveat of migrating VMs, just document it and merge/move on, I guess? | 13:30 |
| kevko | so you will probably migrate manually ... | 13:30 |
| SvenKieske | kevko: :D you might have a point there | 13:30 |
| mhiner | SvenKieske: sometimes there are qemu processes left running in /var/lib/docker which prevents its removal | 13:30 |
| SvenKieske | mhiner: ah, yes I remember a note about this. | 13:31 |
| kevko | first time i saw that there is a migration to quorum queues i was happy ...and wondering how ? blue-green deployment ? or federation ? | 13:31 |
| kevko | no, rabbitmq reset and restart all services :D | 13:31 |
| SvenKieske | kevko: well "easy" "nuke it from orbit" solution first. more complicated solution..later..maybe :P | 13:32 |
| kevko | mhiner: really ? qemu processes left running var/lib/docker ? how ? | 13:32 |
| kevko | and why some ? | 13:32 |
| SvenKieske | afaik the qemu process is spawned in the host pid namespace outside dockers control, isn't it? | 13:33 |
| mnasiadka | if you boot from local ephemeral storage, then probably it's obvious? | 13:33 |
| kevko | it's | 13:33 |
| mnasiadka | as in qcow2 files in /var/lib/nova/instances ? | 13:33 |
| mhiner | kevko: I am not really sure because it only happens in Zuul - I was not able to reproduce it even when I built our version of Rocky9 with DIB | 13:33 |
| SvenKieske | so of course shutting down the container might leave qemu processes around if they can't be closed. lsof -p should tell why | 13:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: Pin requests to <2.32 for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920131 | 13:34 |
| mnasiadka | anyway, let's document the limitation of the migration process and be gone with it? | 13:34 |
| mnasiadka | any way to check if VMs are running and failing beforehand? | 13:35 |
| kevko | mnasiadka: if it is in /var/lib/nova/instances ..it's probably volume (didn't check the code) and it can be migrated ? i mean data can be migrated ? | 13:35 |
| mnasiadka | kevko: it's not a volume, it's an ephemeral root disk - of course you can do live migration to copy that to other host - and that's what we should document + have a precheck before running the migration to podman | 13:35 |
| chembervint | Hi to all, | 13:37 |
| chembervint | I wana discuss https://review.opendev.org/c/openstack/kolla-ansible/+/918639 - starting Antelope we have no more chance to restart Docker daemon without container. Small fix into systemd unit file will fix it | 13:37 |
| mnasiadka | Stop | 13:38 |
| mnasiadka | We have a meeting now | 13:38 |
| mnasiadka | You can raise that when we get to Open discussion | 13:38 |
| chembervint | oh, ok, thank you | 13:38 |
| SvenKieske | well looking into it qemu writes the instance log to /var/lib/nova and the qemu domain monitor socket is also located in /var/lib/libvirt/ | 13:38 |
| mnasiadka | ok, the technical discussion is fantastic - but for the sake of time - can we agree to not support migration of a host that has running instances? | 13:39 |
| SvenKieske | seems fine to me :) +1 | 13:39 |
| mnasiadka | just for the simplicity and no angry people throwing things at us on OpenInfra conferences ;-) | 13:39 |
| SvenKieske | and also +1 to move the remaining discussion into the review :) | 13:39 |
| mnasiadka | mhiner: are you fine with that? | 13:39 |
| mhiner | sure | 13:40 |
| mnasiadka | ok then | 13:40 |
| mnasiadka | next one | 13:40 |
| mnasiadka | (r-krcek): please review: | 13:40 |
| mnasiadka | on behalf of wu.chunyang: https://review.opendev.org/c/openstack/kolla-ansible/+/797498 | 13:40 |
| mnasiadka | Thoughts on the last comments? https://review.opendev.org/c/openstack/kolla-ansible/+/599735 | 13:40 |
| SvenKieske | I'll review the swift role thing later this week (TM), can't really test much as we use ceph-rgw as most deployments (I guess) | 13:42 |
| mnasiadka | ok, so cores - please have a look in both - I think the swift role modernization is really something we should finally try to merge | 13:42 |
| SvenKieske | to add to that: I think if swift gets changed it also should receive some more tests, because some stuff in swift was broken for quite some releases and nobody noticed because of the lack of tests | 13:43 |
| mnasiadka | for check command - I agree the name is a bit vague, but if we do check-containers - then it clashes a bit with check-containers.yml ;-) | 13:43 |
| SvenKieske | I'll make sure to also mention this in the review | 13:43 |
| mnasiadka | I'll be happy to have around anybody that uses Swift and cares for it | 13:43 |
| mnasiadka | but sadly I think we don't have such a person | 13:43 |
| mnasiadka | maybe wuchunyang ;-) | 13:43 |
| mmalchuk | its me) | 13:43 |
| mmalchuk | will review | 13:44 |
| SvenKieske | well check command: don't know about naming clashes any name works for me. | 13:44 |
| * mmalchuk about Swift | 13:44 | |
| mnasiadka | ok, reviews are good | 13:45 |
| kevko | mnasiadka: commented .. | 13:45 |
| mnasiadka | Let's move on | 13:45 |
| kevko | why we want to add check to kolla ? | 13:45 |
| r-krcek | thank you :) | 13:45 |
| mnasiadka | check the comments in the patch | 13:45 |
| kevko | we have monitoring documented | 13:45 |
| mnasiadka | I'm fine with anything people need ;-) | 13:45 |
| mmalchuk | not only review, will try to check on Xena/Yoga too)) | 13:45 |
| mnasiadka | well, we could have also some command that fails if any container healthcheck is bad | 13:45 |
| kevko | well, i have couple of patches i need ..still waiting :) | 13:46 |
| mnasiadka | everybody has those | 13:46 |
| mnasiadka | I'm overloaded this week, might have some time next week | 13:46 |
| mnasiadka | but then I have some business trip in first week of June | 13:46 |
| mnasiadka | so complicated patches needing my attention might need to wait | 13:46 |
| r-krcek | mnasiadka: the patch covers both not running and unhealthy containers | 13:46 |
| mnasiadka | r-krcek: that's nice | 13:47 |
| kevko | it's 10 minutes for review and merge ...i really need only two patches ...only init-core-once ..so i can work on tempest | 13:47 |
| mnasiadka | I'm not convinced I want more bash scripts to be able to run tempest ;-) | 13:47 |
| kevko | it's ctrl+c and ctrl+v what is already done + 2 additional lines ..common | 13:47 |
| SvenKieske | speaking of healthchecks, this got reworked to only check actually supported HTTP checks: https://review.opendev.org/c/openstack/kolla-ansible/+/918437 I think that's a great improvement for the project | 13:47 |
| SvenKieske | kevko: I'm not sure I can follow, which patch are you talking about? | 13:48 |
| SvenKieske | I'll add above patch to the whiteboard for next week | 13:48 |
| kevko | mnasiadka: yes, i can also setup dynamic users, networks etc geneeration ...but it's overkill ... | 13:48 |
| kevko | SvenKieske: https://review.opendev.org/c/openstack/kolla-ansible/+/914191 << | 13:49 |
| mnasiadka | You don't understand, we've had that discussion on the PTG - we have such high number of bash scripts - that we wanted to move to Tempest to get rid of them | 13:49 |
| mnasiadka | not multiply them or add new ones | 13:49 |
| mnasiadka | But if moving to Tempest means we add more bash scripts and more dependencies to testing | 13:50 |
| SvenKieske | mhm I'll not be around next week, I'm on PTO from 29.5. until 3.6. | 13:50 |
| mnasiadka | I don't know if that's worth the risk of breaking CI for some time and resolving those issues | 13:50 |
| mnasiadka | especially that it seems we've got more stable now | 13:50 |
| mnasiadka | I propose that everybody interested does get a look on kevko's patches | 13:50 |
| mnasiadka | and we discuss that in one of the future meetings | 13:51 |
| SvenKieske | just move all those stuff from bash to python or ansible? I don't see anything there that's not possible to do in a basic ansible playbook, no? am I missing something? | 13:51 |
| mnasiadka | Let's stop that discussion now, it's 9 minutes to closure :) | 13:51 |
| SvenKieske | ok | 13:51 |
| mnasiadka | #topic Open discussion | 13:52 |
| kevko | mnasiadka: okay then, so i will provide drop for tools/init-runonce bash and rewrite to ansible ok ? | 13:52 |
| mnasiadka | chembervint: now | 13:52 |
| mnasiadka | kevko: I think it would be good if we would have an etherpad on plan for improvements | 13:52 |
| mnasiadka | not just spend time writing something, that we later argue on :) | 13:52 |
| mnasiadka | and once we agree on the way forward - we can get to writing code that will easily get accepted | 13:53 |
| kevko | mnasiadka: https://etherpad.opendev.org/p/KollaWhiteBoard << it is here | 13:53 |
| mnasiadka | Which line? | 13:53 |
| kevko | 258 | 13:53 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 13:53 |
| mnasiadka | kevko: Tempest testing instead of maintaining all those bash scripts | 13:53 |
| kevko | and it was discussed i think on previous ptg (not the last) | 13:54 |
| mnasiadka | do you see the whole sentence? | 13:54 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 13:54 |
| mmalchuk | cores please merge two orphans: https://review.opendev.org/q/Id6e4bbe0aab2360c4e7e5f74fff6170bcc71080b | 13:54 |
| kevko | mnasiadka: yep, I will process | 13:54 |
| chembervint | ok, I'm here. so https://review.opendev.org/c/openstack/kolla-ansible/+/918639 - I propose a fix, which enable us back to configure live-restore to docker daemon and keep containers alive during docker restarts, which is important for production environments | 13:54 |
| SvenKieske | kevko: thanks for working on it :) | 13:54 |
| chembervint | and worked fine untill systemd units were introduced into Kolla | 13:55 |
| mnasiadka | chembervint: thanks, I'll add review priority and we'll try to timely review that | 13:55 |
| chembervint | thank you! | 13:55 |
| mnasiadka | ok then | 13:56 |
| mnasiadka | anybody else? | 13:56 |
| mnasiadka | (we have 4 minutes) | 13:56 |
| SvenKieske | well I need to process: https://bugs.launchpad.net/kolla-ansible/+bug/2065168 @chembervint first | 13:56 |
| SvenKieske | and I guess all reviewers of https://review.opendev.org/c/openstack/kolla-ansible/+/918639 should read that as well | 13:56 |
| SvenKieske | repeating myself from above: the healthcheck rework seems really like a good addition: https://review.opendev.org/c/openstack/kolla-ansible/+/918437 | 13:57 |
| SvenKieske | RP+1 on that would be nice the author is very responsive | 13:57 |
| mnasiadka | well, I added RP+1 | 13:57 |
| mnasiadka | The patch is nice, but needs thorough testing | 13:58 |
| SvenKieske | especially for a first time contributor :) thanks | 13:58 |
| SvenKieske | for sure | 13:58 |
| mnasiadka | Especially that we don't allow people to go back to previous setting easily | 13:58 |
| mnasiadka | maybe we should? | 13:58 |
| kevko | mnasiadka: sorry ..but i need to again say something ...on line 253 is "Tempest testing instead of maintaining all those bash scripts" ....so if you allow tempest ...you can drop almost ALL bashes ... but you don't want to approve 10 lines of bash to allow work on tempest which will replace and remove all bashes ? | 13:58 |
| SvenKieske | okay, that might be a good addition | 13:58 |
| chembervint | it's not my first time ... :) but thank you! :) | 13:58 |
| mnasiadka | kevko: I know you can do better and you can do Ansible instead of those bashes - and your patches don't include dropping of those bashes | 13:58 |
| SvenKieske | chembervint: I was talking about https://review.opendev.org/c/openstack/kolla-ansible/+/918437 I think that's not you, is it? :) | 13:59 |
| kevko | mnasiadka: thanks :D | 13:59 |
| chembervint | oh, sorry :) | 13:59 |
| mnasiadka | And I would rather see tenths of rechecks of the Tempest runs to make sure everything works like we want | 13:59 |
| SvenKieske | chembervint: I'll re-review your patch as well | 13:59 |
| mnasiadka | Because if it fails - we're going to chase and blame kevko :) | 13:59 |
| SvenKieske | chembervint: no problem :) | 13:59 |
| kevko | kolla -> kevko -> tempest cores | 14:00 |
| kevko | :) | 14:00 |
| SvenKieske | should we add rally to the list :D | 14:00 |
| SvenKieske | ok, it's 16:00 here. | 14:00 |
| mnasiadka | ok | 14:01 |
| mnasiadka | thanks for coming! | 14:01 |
| mnasiadka | see you next week | 14:01 |
| mnasiadka | #endmeeting | 14:01 |
| opendevmeet | Meeting ended Wed May 22 14:01:34 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-05-22-13.01.html | 14:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-05-22-13.01.txt | 14:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-05-22-13.01.log.html | 14:01 |
| mmalchuk | mnasiadka thanks | 14:02 |
| SvenKieske | mhm I really need to reread this whole docker restart thing, thanks for all the comment on the launchpad bugs | 14:02 |
| SvenKieske | thanks mnasiadka! | 14:02 |
| jovial[m] | Any ideas why doesn't CI seem to be re-triggering on: https://review.opendev.org/c/openstack/kayobe/+/910513? | 14:04 |
| mmalchuk | SvenKieske we had a short discussion outside the launchpad, not added any conclusion there | 14:04 |
| mmalchuk | so feel free to recheck from the scratch | 14:04 |
| kevko | sorry, need to run ... but will be around if anything | 14:05 |
| * mmalchuk too | 14:09 | |
| mmalchuk | cu | 14:09 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add support for Cumulus NVUE switches https://review.opendev.org/c/openstack/kayobe/+/914638 | 14:09 |
| opendevreview | Merged openstack/ansible-collection-kolla master: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920172 | 14:25 |
| opendevreview | Merged openstack/ansible-collection-kolla stable/2023.1: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920124 | 14:25 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla stable/2024.1: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920132 | 14:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: Pin requests to <2.32 for docker sdk https://review.opendev.org/c/openstack/kolla-ansible/+/920131 | 14:55 |
| opendevreview | Merged openstack/ansible-collection-kolla stable/2023.2: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920125 | 14:58 |
| opendevreview | Kevin Tindall proposed openstack/kolla-ansible master: Add TLS proxy for novncproxy https://review.opendev.org/c/openstack/kolla-ansible/+/911141 | 15:00 |
| opendevreview | Matt Crees proposed openstack/kolla stable/2023.1: Build RabbitMQ 3.11, 3.12 and 3.13 in 2023.1 https://review.opendev.org/c/openstack/kolla/+/918974 | 16:04 |
| -opendevstatus- NOTICE: There will be a short Gerrit outage while we update to the latest 3.8 release in preparation for next weeks 3.9 upgrade. | 17:01 | |
| mnasiadka | frickler: we also need https://review.opendev.org/c/openstack/kolla-ansible/+/920131 first, and then I’ll do cherry picks forward | 17:40 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Bump up Ansible supported versions to 8.x/9.x https://review.opendev.org/c/openstack/kayobe/+/910513 | 17:56 |
| kevko | mnasiadka: stil here ? | 18:09 |
| frickler | mnasiadka: you need to remember that backport-candidate!=0 is a required vote now. though I wonder how that makes sense for reviews that are backports already. maybe we want to restrict this to the master branch? cc SvenKieske | 18:25 |
| mnasiadka | Yeah, restricting that to master makes sense | 18:26 |
| mnasiadka | kevko: sort of | 18:26 |
| opendevreview | Merged openstack/ansible-collection-kolla stable/2024.1: docker_sdk: Pin requests to <2.32 due to docker-py incompatibility https://review.opendev.org/c/openstack/ansible-collection-kolla/+/920132 | 18:30 |
| kevko | mnasiadka: regarding bash removal and replace by ansible ... It is very unfortunate that the openstack.cloud Galaxy collection does not correspond at all with the upper-constraints... but we take that into account in CI ... for now i just rewrited minimal version in a galaxy code :D ... | 18:32 |
| kevko | i mean openstacksdk | 18:33 |
| kevko | will see ..i will send something :) | 18:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!