| opendevreview | Merged openstack/kayobe master: Pin ansible-core at 2.15.6 https://review.opendev.org/c/openstack/kayobe/+/902614 | 01:21 |
|---|---|---|
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 02:33 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 06:01 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 07:44 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Clean up release notes for Bobcat https://review.opendev.org/c/openstack/kayobe/+/902750 | 08:38 |
| SvenKieske | dcapone2004: interesting, maybe we could add a precheck if there are already certs in place not managed by kolla? ask if the user is sure they want to override them? | 09:06 |
| SvenKieske | dcapone2004: maybe worth it to file an issue | 09:36 |
| SvenKieske | mnasiadka, frickler: is this a kayobe only issue with ansible-core 2.15.7? https://review.opendev.org/c/openstack/kayobe/+/902614 should we pin in kolla{-ansible} as well? | 09:38 |
| mnasiadka | I haven't seen any breakage in kolla-ansible - so why pin? | 09:41 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/yoga: docs: add information about enabling TLS to automated guide https://review.opendev.org/c/openstack/kayobe/+/902228 | 09:43 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/zed: docs: add information about enabling TLS to automated guide https://review.opendev.org/c/openstack/kayobe/+/902227 | 09:49 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 09:59 |
| opendevreview | Merged openstack/kayobe master: Use OpenStack 2023.2 release https://review.opendev.org/c/openstack/kayobe/+/902422 | 10:59 |
| opendevreview | Merged openstack/kayobe master: Clean up release notes for Bobcat https://review.opendev.org/c/openstack/kayobe/+/902750 | 10:59 |
| SvenKieske | I guess I'm just not seeing it, but can someone spot the actual error here? https://zuul.opendev.org/t/openstack/build/6ded685cf6ab4dce8816a77155b463be/log/primary/logs/ansible/test-core-openstack?severity=0 | 12:49 |
| mnasiadka | https://8aae34bf932c24fa9223-2b94732b4d1115584c793b46f27b760e.ssl.cf5.rackcdn.com/902699/1/check/kolla-ansible-rocky9-upgrade-cephadm/6ded685/primary/logs/ansible/test-core-openstack-upgrade | 12:57 |
| mnasiadka | here | 12:57 |
| mnasiadka | we could be better on saying it's on upgrade :D | 12:58 |
| SvenKieske | are the logs in the zuul UI truncated? Because I don't have that failure when looking at the log via the above link..weird | 13:02 |
| SvenKieske | it ends on line 2400 for me, can't scroll further | 13:02 |
| jangutter | no that file really ends there | 13:08 |
| jangutter | but yeah, the actual error isn't in that file. the breadcrumbs are here to the real log: https://zuul.opendev.org/t/openstack/build/6ded685cf6ab4dce8816a77155b463be/console#2/3/14/primary | 13:11 |
| jangutter | https://zuul.opendev.org/t/openstack/build/6ded685cf6ab4dce8816a77155b463be/log/primary/logs/ansible/test-core-openstack-upgrade#412 volume was still in status `creating` | 13:12 |
| mnasiadka | and cinder-volume probably again has problems with RMQ | 13:21 |
| mnasiadka | so we're doing something wrong in the upgrade RMQ handling - or cinder is special :D | 13:21 |
| mnasiadka | 2023-12-05 19:15:23.760 1088 WARNING oslo.messaging._drivers.impl_rabbit [None req-cbce7ede-4fdc-41ca-829c-7e9fa1a3b044 - - - - - -] Force creating a non durable exchange.: amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'true' but current is 'false' | 13:39 |
| mnasiadka | looks like something is wrong on rmq side | 13:40 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: Add Redfish rules to Ironic introspection https://review.opendev.org/c/openstack/kayobe/+/902772 | 13:42 |
| mnasiadka | jangutter, SvenKieske: for some reason kolla-ansible stop does not work for cinder role | 13:52 |
| mnasiadka | look at https://8aae34bf932c24fa9223-2b94732b4d1115584c793b46f27b760e.ssl.cf5.rackcdn.com/902699/1/check/kolla-ansible-rocky9-upgrade-cephadm/6ded685/primary/logs/ansible/stop - search "apply role cinder" | 13:52 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi - meeting in 7 minutes | 13:53 |
| SvenKieske | interesting | 13:55 |
| jangutter | mnasiadka skipped because of tags or something else? Thats site.yaml right? | 13:56 |
| jangutter | mnasiadka: ah --tags heat,keystone,neutron,nova <--- need to add cinder there? | 13:56 |
| mnasiadka | ah, so upgrade.sh is borked | 13:57 |
| mnasiadka | https://github.com/openstack/kolla-ansible/blob/af6e1ca4fd3366c1ae936d225b2835e95a4191fc/tests/upgrade.sh#L21 | 13:57 |
| mnasiadka | in theory it should add cinder | 13:57 |
| mnasiadka | but it's not | 13:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add SCENARIO env var to upgrade.sh https://review.opendev.org/c/openstack/kolla-ansible/+/902776 | 13:59 |
| mnasiadka | ^^ that should fix it | 13:59 |
| mnasiadka | thanks jangutter :) | 13:59 |
| mnasiadka | #startmeeting kolla | 14:00 |
| opendevmeet | Meeting started Wed Dec 6 14:00:00 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:00 |
| mnasiadka | #topic rollcall | 14:00 |
| frickler | \o | 14:00 |
| mnasiadka | o/ | 14:00 |
| SvenKieske | o/ | 14:00 |
| osmanlicilegi | o/ | 14:00 |
| jangutter | o/ | 14:01 |
| jovial | o/ | 14:01 |
| mnasiadka | #topic agenda | 14:03 |
| mnasiadka | * CI status | 14:03 |
| mnasiadka | * Release tasks | 14:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 14:03 |
| mnasiadka | * Current cycle planning | 14:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:03 |
| mnasiadka | * Open discussion | 14:03 |
| mnasiadka | osmanlicilegi: welcome back ;-) | 14:03 |
| mnasiadka | #topic CI status | 14:04 |
| osmanlicilegi | mnasiadka: thanks :) I will be back after long and tiring years. | 14:04 |
| mnasiadka | I think it's overall green, with the cephadm-upgrade jobs failing because of the RMQ switch to Quorum queues | 14:04 |
| mnasiadka | but that's being fixed with 902776 | 14:04 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/902776 | 14:04 |
| mnasiadka | #topic Release tasks | 14:05 |
| mnasiadka | Kolla, Kolla-Ansible and ansible-collection-kolla going for a final Bobcat release | 14:05 |
| mnasiadka | #link https://review.opendev.org/c/openstack/releases/+/902718 | 14:05 |
| mnasiadka | Kayobe still waiting for rc1 - last patch merging | 14:06 |
| mnasiadka | jovial: will you create a patch for kayobe repos rc1? | 14:06 |
| mnasiadka | well, I guess he has no other choice - worst case scenario I'll do it | 14:07 |
| mnasiadka | let's move forward | 14:07 |
| mnasiadka | #topic Current cycle planning | 14:07 |
| mnasiadka | So, usually we did have a stability phase - so we did stay with building latest stable release (Bobcat) in Kolla | 14:08 |
| jovial | mnasiadka: sure, I can try after the meeting | 14:08 |
| mnasiadka | just to have some calm weeks | 14:08 |
| mnasiadka | But I'm fine with us reverting to master faster to have a look if this breaks us badly | 14:09 |
| mnasiadka | Any opinions? | 14:09 |
| frickler | I'd be fine with waiting until January | 14:09 |
| mnasiadka | #link https://docs.openstack.org/kolla/latest/contributor/release-management.html#early-cycle-stability | 14:09 |
| mnasiadka | makes sense, let's at least have stable gate over christmas period | 14:09 |
| mnasiadka | should we prepare patches that need reverting in a gerrit topic - so it's easy to revisit them in a month time? | 14:10 |
| SvenKieske | I'm fine with either. a topic seems nice | 14:11 |
| mnasiadka | I think so - if there's a volunteer - I would be happy ;) | 14:11 |
| mnasiadka | ok then | 14:12 |
| mnasiadka | Any other obvious issues that we need to fix in freshly released Bobcat? | 14:12 |
| mnasiadka | Ah, the horizon policy thing? | 14:12 |
| SvenKieske | yeah | 14:13 |
| mnasiadka | #link https://review.opendev.org/q/topic:%22fix-horizon-role%22 | 14:13 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/901761 - seems upgrade jobs are failing here | 14:13 |
| frickler | mnasiadka: yes, the fix needs to be backported first | 14:14 |
| mnasiadka | makes sense | 14:14 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 14:14 |
| mnasiadka | Looking for another +2 | 14:14 |
| mnasiadka | upgrade-cephadm is known for failing now | 14:15 |
| mnasiadka | ok then | 14:15 |
| mnasiadka | those two on the radar and let's see if anything else shows up | 14:16 |
| mnasiadka | I'll finally populate Whiteboard with cycle priorities - so we can discuss next week | 14:16 |
| frickler | I'll be testing things downstream, image builds are fine already | 14:16 |
| mnasiadka | I'd also like to have a go at trying to look through all kolla-ansible reviews and abandon very old reviews and try to review things that would be useful | 14:17 |
| mnasiadka | kolla looks quite nice with only a couple of patches in progress | 14:18 |
| frickler | sad thing is a lot of things likely are indeed still useful and relevant. but yes, that would be a good task | 14:18 |
| mnasiadka | there's a lot that are ready or nearly ready - we just lack capacity to review them | 14:18 |
| mnasiadka | so I'll try to make them tidy and maybe RP+1 those that are easy to get in | 14:19 |
| mnasiadka | and fairly small | 14:19 |
| SvenKieske | happy to do some additional reviews | 14:19 |
| mnasiadka | ok then, let's move on | 14:19 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 14:19 |
| mnasiadka | mattcrees: https://etherpad.opendev.org/p/kolla-ansible-rmq-quorum-queues-proposal | 14:20 |
| mnasiadka | I think we already violated that proposal with turning quorum queues as default in B | 14:20 |
| mnasiadka | question if we need more prechecks | 14:20 |
| mnasiadka | maybe such prechecks, that would check if a queue is non-durable (if it exists) | 14:20 |
| mnasiadka | but mattc is not here | 14:20 |
| mnasiadka | I'll leave that for next time | 14:21 |
| mnasiadka | jsuazo also is reminding us about two reviews - but I would prefer to wait until we get final Bobcat release out until we merge new features | 14:21 |
| mnasiadka | so both topics are left for next week | 14:22 |
| jovial | mattc reckons we need the same style of precheck that we added previously for HA queues. He will try an propose something this week. | 14:22 |
| mnasiadka | Well, basically I would be happy if we could have prechecks that really stop kolla-ansible break services - so happy to review once it gets into Gerrit | 14:23 |
| SvenKieske | seems you have summoned someone :) | 14:24 |
| mattcrees | o/ | 14:24 |
| mnasiadka | well, that's the PTL role I guess | 14:24 |
| mnasiadka | mattcrees: so, do your planned prechecks plan to check the ,,durability'' of the existing queues before starting services? | 14:25 |
| mnasiadka | currently our cephadm-upgrade jobs are broken, because we're not stopping cinder, and it tends to complain about non-durable queues that were left behind | 14:25 |
| mattcrees | Yes, the plan is to have a check similar to this: https://github.com/openstack/kolla-ansible/blob/af6e1ca4fd3366c1ae936d225b2835e95a4191fc/ansible/roles/rabbitmq/tasks/precheck.yml#L211 | 14:26 |
| mattcrees | Only for quorum queue type instead of durability type | 14:26 |
| mnasiadka | ok, let's see how that works | 14:27 |
| mnasiadka | So let's move to open discussion | 14:27 |
| mnasiadka | #topic Open discussion | 14:27 |
| mattcrees | We should be stopping cinder though... https://github.com/openstack/kolla-ansible/blob/af6e1ca4fd3366c1ae936d225b2835e95a4191fc/tests/upgrade.sh#L22C20-L22C20 | 14:27 |
| mnasiadka | osmanlicilegi: you can now share what made you miss us ;-) | 14:27 |
| mnasiadka | mattcrees: we are not, because SCENARIO env var is not passed to upgrade.sh | 14:27 |
| mnasiadka | but it will be - https://review.opendev.org/c/openstack/kolla-ansible/+/902776 | 14:28 |
| mattcrees | ah ok makes sense | 14:28 |
| frickler | the wsgi scripts discussion is worth watching, but I think we can sit and wait how it works out | 14:28 |
| frickler | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5RQGZZMCKKHAVH4OLSJCLBQNIHYDPGZD/ | 14:28 |
| mnasiadka | yeah, seen that | 14:28 |
| mnasiadka | I think we still have some services to move to wsgi | 14:29 |
| mnasiadka | starting with Neutron | 14:29 |
| osmanlicilegi | mnasiadka: very long story :) I will share it via dm. | 14:29 |
| mnasiadka | osmanlicilegi: ok then :) | 14:29 |
| frickler | the eventlet issue is also interesting, but even less we can do about it | 14:29 |
| SvenKieske | do you mean wsgi or uWSGI? | 14:29 |
| mnasiadka | but we could use some more review cycles - so osmanlicilegi feel free to do reviews if you have time :) | 14:30 |
| frickler | SvenKieske: it affects both in a way | 14:30 |
| mnasiadka | but let's not merge any features in master until next weekly meeting please | 14:30 |
| osmanlicilegi | mnasiadka: I will start soon reviewing :) | 14:32 |
| SvenKieske | cool :) | 14:32 |
| mnasiadka | frickler: ubuntu 24.04 is going to be released in April, we might need to add support for it last minute in C... | 14:33 |
| mnasiadka | but I hope not :) | 14:33 |
| SvenKieske | :D | 14:34 |
| frickler | hmm, I'd rather backport that later | 14:34 |
| SvenKieske | is it possible to test pre releases? | 14:34 |
| frickler | not in the opendev ci | 14:34 |
| mnasiadka | frickler: we'll still have time from April + 3 months, so... I reckon last minute addition | 14:36 |
| frickler | ok, but first we need to see if ubuntu keeps releasing on time ;) | 14:37 |
| mnasiadka | yup | 14:37 |
| mnasiadka | I'll add that to the priorities list with a note to "look out for it" | 14:38 |
| mnasiadka | Ok then, let's finish for today | 14:38 |
| mnasiadka | Ah, one last thing | 14:38 |
| mnasiadka | when do we want to stop meetings for this year? | 14:38 |
| mnasiadka | I planned to cancel the 20th and 27th of Dec and 3rd Jan | 14:39 |
| frickler | I'll be around anyway | 14:39 |
| mnasiadka | I'll be around as well, but I think majority of people will be off | 14:39 |
| mnasiadka | So I'll cancel those three and we can just raise things on the channel outside of the meeting | 14:39 |
| frickler | ack | 14:40 |
| mnasiadka | Thanks for coming, talk to you next week :) | 14:40 |
| mnasiadka | #endmeeting | 14:40 |
| opendevmeet | Meeting ended Wed Dec 6 14:40:08 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:40 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-12-06-14.00.html | 14:40 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-12-06-14.00.txt | 14:40 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-12-06-14.00.log.html | 14:40 |
| SvenKieske | I'll be away from roughly 24th of Dec til 8th of Jan. | 14:40 |
| frickler | thx mnasiadka | 14:40 |
| SvenKieske | thanks mnasiadka | 14:40 |
| opendevreview | Merged openstack/kayobe master: Remove unused kolla bootstrap-servers variables https://review.opendev.org/c/openstack/kayobe/+/902379 | 15:31 |
| opendevreview | Merged openstack/kolla-ansible master: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 15:58 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Add precheck for RabbitMQ quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/902787 | 16:09 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: RabbitMQ: correct docs on Quorum Queue migrations https://review.opendev.org/c/openstack/kolla-ansible/+/902789 | 16:11 |
| jovial | mnasiadka, Does this look OK: https://review.opendev.org/c/openstack/releases/+/901950? | 16:13 |
| sulo | Hi, new to Kolla release cycle, so quick question, is Kolla skipping Antelope and going straight to Bobcat. I cant find any 2023.1 tags on docker hub. | 16:17 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Add precheck for RabbitMQ quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/902787 | 16:25 |
| SvenKieske | sulo_: no we are not skipping releases, afaik we stopped publishing to dockerhub, but my memory is fuzzy there | 16:32 |
| sulo_ | ok, where can i find the images ? | 16:33 |
| sulo_ | i see 2023.2 tags in docker hub https://hub.docker.com/r/kolla/nova-libvirt/tags?page=1&name=2023 | 16:33 |
| SvenKieske | good question, it was discussed during last PTG: https://etherpad.opendev.org/p/kolla-caracal-ptg#L146 | 16:36 |
| mnasiadka | jovial: sure | 16:36 |
| SvenKieske | most people here are building their own images, that's why I'm actually not that familiar with the current status. I thought we push to quay.io, but I can't find newer release tags there either | 16:37 |
| SvenKieske | maybe mnasiadka nows ;) | 16:37 |
| mnasiadka | We push to quay.io/openstack.kolla/ | 16:38 |
| SvenKieske | yeah, but I'm blind or I dont see any 2023.1|antelope tag there? | 16:38 |
| SvenKieske | mhm, seems I got bad luck: here are newer tags: https://quay.io/repository/openstack.kolla/nova-compute?tab=tags | 16:39 |
| SvenKieske | but this one only has old tags? https://quay.io/repository/openstack.kolla/ubuntu-source-nova-libvirt | 16:40 |
| SvenKieske | ah I guess we omit the "-source" part now, don't we? | 16:40 |
| SvenKieske | mhm no, we don't | 16:42 |
| mattcrees | SvenKieske: we drop the ubuntu-source now :) https://quay.io/repository/openstack.kolla/nova-libvirt?tab=tags | 16:44 |
| SvenKieske | ah right, it was this way around | 16:45 |
| sulo_ | @SvenKieske thank you. | 16:48 |
| sulo_ | Is there somewhere i can check to see what the current stable release branch is ? | 16:49 |
| SvenKieske | no problem, thank mattcrees and mnasiadka :) | 16:49 |
| sulo_ | yes thanks all :) | 16:49 |
| SvenKieske | git ls-remote --refs https://opendev.org/openstack/kolla-ansible.git "refs/heads/stable/*" | 16:53 |
| SvenKieske | gives you all stable branches without cloning the repo first | 16:54 |
| SvenKieske | newest would be newest numeric one under current release naming rules | 16:54 |
| SvenKieske | EOL/unmaintained releases don't show up there. | 16:55 |
| sulo_ | perfect. thanks. | 17:20 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Add precheck for RabbitMQ quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/902787 | 17:23 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/2023.2: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902821 | 17:34 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible stable/2023.1: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902822 | 17:35 |
| frickler | technically 2023.1 is still the latest stable release, even though the stable/2023.2 branch already exists | 17:38 |
| frickler | mnasiadka: do we want to get the horizon fix backport right into the kolla-ansible final release? or do a .1 release right afterwards? | 17:39 |
| frickler | we don't have any +2 on the release patch yet, so we could still update | 17:40 |
| SvenKieske | re: "latest" stable release: I mean, we tell people all the time to use the git branches and not pay attention to pypi releases, so isn't this a little bit contradictionary? | 17:42 |
| frickler | well that's only after the initial release has been made. it is also not about pypi, but about having a non-rc tag on the branch | 17:45 |
| SvenKieske | mhm, can't reliably list the tags via git ls-remote in a way that would make this easily scriptable, anyway. | 17:49 |
| mnasiadka | frickler: so lets merge horizon and update | 18:47 |
| mnasiadka | I think we should do monthly releases without thinking too much about it, e.g. don’t really care if something is broken or not, we can always raise bugfix release between monthly releases | 18:50 |
| mnasiadka | But I also think release team review capacity is a bit… not what it used to be. | 18:51 |
| mnasiadka | frickler: coming back to wsgi - seems we should start thinking of ditching mod_wsgi… | 20:30 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily to restart it onto an updated replication key | 21:36 | |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902821 | 21:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!