| shyamb | Hi Team, Is Antelope release done? | 07:57 |
|---|---|---|
| shyamb | I see release notes of 2023.1 but I don't any mention of Antelope. https://docs.openstack.org/releasenotes/kolla/2023.1.html | 07:58 |
| shyamb | Any quay.io also does not have any Antelope tagged container images | 07:58 |
| shyamb | https://quay.io/repository/openstack.kolla/nova-api?tab=tags | 07:58 |
| mnasiadka | 2023.1 is Antelope | 07:59 |
| shyamb | Okay, thank you mnasiadka | 08:00 |
| shyamb | Looks like we have changed naming strategy. | 08:00 |
| shyamb | Are we going to use this format going ahead? | 08:01 |
| shyamb | 2023.1, 2023.2 | 08:01 |
| mnasiadka | shyamb: https://governance.openstack.org/tc/reference/release-naming.html | 08:01 |
| shyamb | Thank you mnasiadka. This is helpful. | 08:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Check OVN DB schema on container update https://review.opendev.org/c/openstack/kolla-ansible/+/898721 | 08:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Check OVN DB schema on container update https://review.opendev.org/c/openstack/kolla-ansible/+/898721 | 08:39 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Check OVN DB schema on container update https://review.opendev.org/c/openstack/kolla-ansible/+/898721 | 10:06 |
| guesswhat[m] | Has anyone experience with rabbitmq optimalization for single node deployment? Sas that default value for erlang is 2:2, but it still peaks to 1000% CPU . Thanks | 11:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Check OVN DB schema on container update https://review.opendev.org/c/openstack/kolla-ansible/+/898721 | 11:53 |
| *** dimentor is now known as dimentor_ | 11:53 | |
| *** mmalchuk is now known as dimentor | 11:53 | |
| *** dimentor is now known as mmalchuk | 11:59 | |
| *** dimentor_ is now known as dimentor | 12:00 | |
| opendevreview | Alex Welsh proposed openstack/kayobe master: Add seed_deploy_containers_registry_attempt_login https://review.opendev.org/c/openstack/kayobe/+/898437 | 12:10 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo - meeting in 6 | 12:54 |
| SvenKieske | yay | 12:55 |
| mmalchuk | yuhu | 12:55 |
| * SvenKieske stumbles out of a total of 4:40 hours of meeting straight into the next one -.- | 12:56 | |
| * jangutter is hurtling towards the end of a quarter. | 12:59 | |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Oct 18 13:00:09 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mmalchuk | o/ | 13:00 |
| jangutter | \o | 13:00 |
| mnasiadka | o/ | 13:01 |
| SvenKieske | o/ | 13:01 |
| frickler | \o | 13:01 |
| mattcrees | o/ | 13:01 |
| jsuazo | o/ | 13:01 |
| mnasiadka | #topic Agenda | 13:03 |
| mnasiadka | * Announcements | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic Announcements | 13:03 |
| mnasiadka | PTG is next week | 13:03 |
| mnasiadka | #link https://etherpad.opendev.org/p/kolla-caracal-ptg | 13:03 |
| SvenKieske | \o/ | 13:04 |
| mnasiadka | Sign up please in L37 | 13:04 |
| * mmalchuk ready | 13:04 | |
| mnasiadka | And please add topics (L101 and beyond) | 13:04 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | I think it's mainly green, not couting our Ansible breakage | 13:04 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Enable the Fluentd Plugin Systemd https://review.opendev.org/c/openstack/kolla-ansible/+/875983 | 13:04 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla master: Adds TAAS Neutron plugin to support OVS port mirrors https://review.opendev.org/c/openstack/kolla/+/885151 | 13:04 |
| mnasiadka | Trying to wrap my head around it, but it's not going to be an easy one | 13:04 |
| mnasiadka | Anybody did have a look in that as well? | 13:05 |
| frickler | not yet. maybe building a simple reproducer will be needed? | 13:05 |
| SvenKieske | not really, I still suspect a real upstream bug, or did we change anything in that area? | 13:05 |
| mnasiadka | We did not, that's the usual "we fixed that bug in Ansible" :D | 13:06 |
| mmalchuk | there is an issue upstream, SvenKieske have a link? | 13:06 |
| SvenKieske | there's also already user reports about this on the ML, I answered with the workaround | 13:06 |
| SvenKieske | #link https://github.com/ansible/ansible/issues/81945 | 13:06 |
| mmalchuk | right | 13:06 |
| mnasiadka | Anyway, let's move on | 13:07 |
| mnasiadka | #topic Release tasks | 13:07 |
| mnasiadka | Time to have a look at them - we merged switching the sources to Bobcat | 13:07 |
| mnasiadka | seems still no sign of RDO repos on the centos stream mirror | 13:08 |
| mnasiadka | I'll wait instead of using trunk.rdoproject.org | 13:08 |
| opendevreview | Dawud proposed openstack/kolla-ansible stable/yoga: Remove the `grafana` grafana https://review.opendev.org/c/openstack/kolla-ansible/+/898736 | 13:08 |
| mnasiadka | no other release tasks per se - just reviewing existing patches | 13:09 |
| opendevreview | Dawud proposed openstack/kolla-ansible stable/yoga: Remove the `grafana` volume https://review.opendev.org/c/openstack/kolla-ansible/+/898736 | 13:09 |
| SvenKieske | not sure I can finish the fluentd stuff in time..currently have a problem checking stuff inside the container. I think it's best to split out fluentd out of the "common" role, which will require some more work. | 13:09 |
| SvenKieske | not sure if I should move this point to "open discussion"? | 13:10 |
| mmalchuk | good idea to split | 13:10 |
| mnasiadka | it's ok to discuss it here | 13:10 |
| mnasiadka | we should probably move this cycle | 13:11 |
| mnasiadka | but it seems we don't have to | 13:11 |
| SvenKieske | yeah, so if there are no objections I'd move fluentd in a dedicated role, I'm also on vacation friday and with PTG coming up..how much time is left? :D | 13:11 |
| opendevreview | Dawud proposed openstack/kolla-ansible stable/yoga: Remove the `grafana` volume https://review.opendev.org/c/openstack/kolla-ansible/+/898736 | 13:12 |
| SvenKieske | I guess I take the id software dev approach: "It's done, when it's done". | 13:12 |
| mnasiadka | it doesn't need to be done before the PTG | 13:13 |
| mnasiadka | we're cycle trailing, so we still have some time to release | 13:13 |
| mnasiadka | although I would prefer we do it sooner than last minute :) | 13:13 |
| frickler | is fluentd the only upgrade missing? I lost track, are we good with prometheus plugins? | 13:14 |
| SvenKieske | yeah sure; I just need to do some other stuff as well - mainly writing docs it seems - will check internally what has higher priority, maybe the answer is even fluentd. | 13:14 |
| mmalchuk | or yesterday) | 13:14 |
| SvenKieske | afaik I asked last - or the meeting before that - if someone could take a look at the prometheus plugins, no? I have lost track as well. | 13:15 |
| mnasiadka | That might be looked into even last minute, although I would prefer that we would finally have a proper solution (I remember hrw started a script for checking versions on github and updating sources.py) | 13:15 |
| mnasiadka | I might have a look into that after that crappy Ansible breakage | 13:16 |
| mnasiadka | ok then | 13:17 |
| mnasiadka | Podman - seems it's waiting for some reviews, probably mine | 13:18 |
| mnasiadka | Let's Encrypt the same | 13:18 |
| mnasiadka | frickler: do you have any cycles for looking at those two? | 13:18 |
| SvenKieske | https://github.com/openstack-exporter/openstack-exporter/releases is at least at the latest :) | 13:18 |
| mnasiadka | It would be nice to merge them this cycle | 13:18 |
| frickler | I can check podman | 13:18 |
| SvenKieske | be sure to also look at the (linked?) ansible-collection-kolla change for podman, I guess that's also still open | 13:19 |
| frickler | ack | 13:20 |
| SvenKieske | ah it's in the depends on: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 13:20 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla/+/887347 can be merged | 13:20 |
| mnasiadka | I commented on ack patch - I think we need podman based jobs there | 13:20 |
| mnasiadka | a-c-k it is | 13:20 |
| SvenKieske | yeah, I agree, there should be some testing going on :) | 13:21 |
| mnasiadka | Ok, Let's Encrypt - I'll ask bbezak to review those next week | 13:21 |
| mnasiadka | and let's try to get those in, even if the code is not perfect-ish ;) | 13:21 |
| mnasiadka | #topic Additional agenda | 13:22 |
| mnasiadka | 4 patches from jsuazo (the same ones again) | 13:22 |
| mnasiadka | let's see | 13:22 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/885151 | 13:22 |
| mnasiadka | +2 from me | 13:22 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/885417 | 13:23 |
| mnasiadka | (k-a side of TAAS) | 13:23 |
| mnasiadka | already has +2 from me | 13:23 |
| mnasiadka | frickler: willing to have a look or should wait for bbezak? | 13:23 |
| frickler | better ask bbezak, I'd be too picky for this | 13:24 |
| mnasiadka | understood | 13:24 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/844614 - Glance/Cinder-backup S3 | 13:24 |
| mmalchuk | ready 2 weeks ago | 13:25 |
| mnasiadka | commented, but basically looks good | 13:26 |
| mnasiadka | ok then | 13:26 |
| mnasiadka | nothing more on the whiteboard | 13:26 |
| mnasiadka | #topic Open discussion | 13:26 |
| mnasiadka | I'll cancel next weeks meeting because we'll have the PTG sessions | 13:27 |
| mnasiadka | Anything else? | 13:27 |
| SvenKieske | if someone has free time, and it's maybe also a topic for PTG: https://review.opendev.org/c/openstack/kolla-ansible/+/898543 | 13:27 |
| SvenKieske | just a hack to enable quorum queues; all the feedback I read was that they are really nice to have | 13:28 |
| mmalchuk | interesting. is this tested under heavy load? | 13:30 |
| mnasiadka | well, the questions I have is 1) do we test that in CI 2) do we want to switch it to default in C 3) Migration docs for users - since it's breaking-ish | 13:30 |
| mnasiadka | but since classic queue mirroring is deprecated for removal in 4.0 - it's either quorum queues or streams | 13:31 |
| mnasiadka | Seems there is some movement in oslo.messaging | 13:31 |
| mnasiadka | #link https://review.opendev.org/c/openstack/oslo.messaging/+/888479 | 13:31 |
| mmalchuk | 2 - no | 13:31 |
| SvenKieske | mmalchuk: untested from my side (the patchset), from what I understand OVH does use quorum queues under heavy load, but they don't use k-a | 13:31 |
| mmalchuk | this enough, so we need this in k-a | 13:31 |
| frickler | we can start with adding the flag and allow this for new deployments. migration can be done next cycle then | 13:31 |
| SvenKieske | no docs yet :) there's also an open bug to enable streams from OVH, but afaik that's not even implemented in oslo | 13:31 |
| mnasiadka | #link https://review.opendev.org/c/openstack/oslo.messaging/+/890825 | 13:31 |
| SvenKieske | frickler: that was my intention as well :) | 13:31 |
| mmalchuk | frickler +1 | 13:31 |
| SvenKieske | similar to what we did with the HA flag for rabbit | 13:32 |
| mnasiadka | frickler: would feel safer if we had at least one CI job that uses it | 13:32 |
| SvenKieske | I agree on the CI job, didn't have an immediate idea how that would best be implemented | 13:32 |
| frickler | I don't think we need to test all possible configuration combinations. having a job when we switch the default is good enough IMO | 13:32 |
| mmalchuk | mnasiadka test ha flag or +queues? | 13:32 |
| mnasiadka | quorum queues | 13:32 |
| opendevreview | Dawud proposed openstack/kolla-ansible stable/yoga: Remove the `grafana` volume https://review.opendev.org/c/openstack/kolla-ansible/+/898736 | 13:33 |
| mnasiadka | we don't need to test all possible configuration combinations, but it would be useful to test this - with a vision that we'll move to this as default since queue mirroring will be gone in RMQ 4.0 | 13:35 |
| mnasiadka | like let's use quorum on Ubuntu and HA on Rocky - or something similar | 13:35 |
| SvenKieske | sure; would that just entail a set of jobs with it enabled? but which scenario should this run? I've never written upstream CI jobs "from scratch" so I would be grateful for some pointers how that should look like | 13:36 |
| frickler | I would still prefer to defer that to the next cycle. but feel free to add a patch for that on top of SvenKieske's, it shouldn't block that change though | 13:36 |
| SvenKieske | I can see both sides of the argument | 13:37 |
| SvenKieske | I can add a warning doc "this is untested" :D on the other hand, when I looked at other deployment projects, they enabled quorum queues without additional tests ;) | 13:37 |
| mmalchuk | you said OVH uses it | 13:38 |
| SvenKieske | ¯\_(ツ)_/¯ | 13:38 |
| SvenKieske | they don't use k-a | 13:38 |
| mnasiadka | I can see people raising bugs about it and if we can't test it in CI - and no one uses that in production from the Kolla community - we're kind of not supporting it? :) | 13:38 |
| mmalchuk | k-a only deployment tool | 13:38 |
| SvenKieske | if you search for quorum queues on the ML there are some happy users :) | 13:38 |
| frickler | to have some real testing, one would at least need to do upgrades, if not host failovers. for upgrades we need to add the flag now so we can test next cycle | 13:39 |
| mmalchuk | two cents not enable it by default | 13:39 |
| SvenKieske | fwiw the SCS project is interested in quorum queues because there was some perceived instability in certain upgrade scenarios, so I hope I get some testers there as well | 13:39 |
| mnasiadka | not enable it by default just yet | 13:40 |
| SvenKieske | mmalchuk: the current patch disables it | 13:40 |
| mnasiadka | just have a minimal testing coverage | 13:40 |
| SvenKieske | frickler: agree | 13:40 |
| SvenKieske | okay, so a basic (HA?) job that just tests if enabling the flags works? | 13:40 |
| mnasiadka | well, I guess multinode would be best | 13:41 |
| SvenKieske | seems like a good compromise? | 13:41 |
| mnasiadka | but singlenode would at least tell us it works | 13:41 |
| mnasiadka | :) | 13:41 |
| SvenKieske | yeah multinode, that's what I meant | 13:41 |
| mnasiadka | because now we can only assume it works | 13:41 |
| mnasiadka | Ok, I'll think of something | 13:42 |
| SvenKieske | it should work ;) it's at least supported by oslo :D I'll add some very basic test, probably not this week, so if someone beats me, I'm also fine with that :) | 13:42 |
| mnasiadka | Not counting we just changed the default to HA | 13:42 |
| mnasiadka | and mattcrees spent some time to enable it in upgrade jobs including a cleanup ;) | 13:42 |
| mnasiadka | and probably moving from HA to quorum includes the same dance | 13:42 |
| SvenKieske | yeah, it's a little different, but in practice you have to recreate all the queues. | 13:43 |
| jangutter | what about adding it to the "experimental" pipeline for now? Won't guarantee it won't regress, but at least that way the job lives in the repo. | 13:43 |
| mnasiadka | or switch multinode (not multinode-upgrade) to quorum queues and fix the upgrade bit in C | 13:43 |
| mnasiadka | this way we will have some better testing coverage and a path forward | 13:44 |
| SvenKieske | let's discuss further stuff in the patch review, what do you think? | 13:44 |
| mnasiadka | but anyway, we've spent too much time on this :) | 13:44 |
| mnasiadka | yes, let's discuss there | 13:44 |
| * kevko is upgrading to zed right now, but watching | 13:44 | |
| opendevreview | Merged openstack/kayobe stable/2023.1: bifrost: Populate bifrost host vars on deprovision https://review.opendev.org/c/openstack/kayobe/+/898561 | 13:44 |
| mnasiadka | ok then | 13:45 |
| mnasiadka | I don't think there is anything more | 13:45 |
| mmalchuk | we still lack of Kayobe reviews | 13:45 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/861397 | 13:46 |
| mmalchuk | and https://review.opendev.org/c/openstack/kayobe/+/879554 | 13:46 |
| mmalchuk | a half of year | 13:46 |
| mmalchuk | almost | 13:46 |
| mnasiadka | Passed that internally in SHPC | 13:46 |
| mmalchuk | thanks | 13:47 |
| mnasiadka | Let's finish for today - thank you all for coming and speak to you on Monday :) | 13:47 |
| mnasiadka | #endmeeting | 13:47 |
| opendevmeet | Meeting ended Wed Oct 18 13:47:21 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:47 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-18-13.00.html | 13:47 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-18-13.00.txt | 13:47 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-18-13.00.log.html | 13:47 |
| mmalchuk | thanks mnasiadka | 13:47 |
| SvenKieske | yeah, thanks | 13:47 |
| opendevreview | Alex Welsh proposed openstack/kayobe master: Add seed_deploy_containers_registry_attempt_login https://review.opendev.org/c/openstack/kayobe/+/898437 | 13:59 |
| mnasiadka | SvenKieske, frickler: what about adding the magnum CAPI plugin to Kolla images? Are you going to push that patch? Would be nice for users | 14:00 |
| guesswhat[m] | mnasiadka: You still need the CAPI enabled K8s cluster | 14:16 |
| guesswhat[m] | Its not standalone | 14:16 |
| mnasiadka | guesswhat[m]: thank you captain obvious | 14:18 |
| kevko | :D | 14:20 |
| guesswhat[m] | Why to include then? Or does the Kolla team wanting to switch finally to the K8s orchstration ? | 14:23 |
| mnasiadka | guesswhat[m]: And that's why we would like to have the Magnum driver included? lol | 14:35 |
| mnasiadka | if you like Kubernetes too much - there's openstack-helm | 14:35 |
| guesswhat[m] | CAPI plugin for Magnum is integrated in https://github.com/vexxhost/atmosphere, which is K8s based Openstack deployment | 14:42 |
| mnasiadka | Which uses openstack-helm under the hood | 14:43 |
| frickler | didn't we have the same discussion almost word by word two weeks ago? but also I don't have any interest in k8s, so far at least | 14:54 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: CI: Log details about failed containers https://review.opendev.org/c/openstack/kolla-ansible/+/892488 | 16:39 |
| mdbooth | Hello! I'm trying to install with kolla for the first time. My deployment host is Fedora, deploying an all-in-one on another host which is CentOS9. Kolla is installed in a python3.9 venv (first try was default python: 3.11, same issue). I've installed kolla stable/2023.1 in the venv. It's dying with: | 18:14 |
| mdbooth | TASK [nova-cell : Extract current cell settings from list] fatal: [nimbus-01]: FAILED! => {"msg": "template error while templating string: Could not load \"extract_cell\": 'extract_cell'. String: {{ existing | 18:15 |
| mdbooth | Let me try that again... | 18:15 |
| mdbooth | TASK [nova-cell : Extract current cell settings from list] | 18:15 |
| mdbooth | fatal: [nimbus-01]: FAILED! => {"msg": "template error while templating string: Could not load \"extract_cell\": 'extract_cell'. String: {{ existing_cells_list | extract_cell(nova_cell_name) }}. Could not load \"extract_cell\": 'extract_cell'"} | 18:15 |
| mdbooth | I'm guessing that means it's not finding the filter 'extract_cell'? I have never used an ansible custom filter before, so I'm not sure how it's supposed to be loaded. AFACT the function is present in the venv. | 18:17 |
| mdbooth | The deployment host is different to the target host. Is that unusual? | 18:24 |
| mdbooth | System python on the target is 3.9 | 18:29 |
| frickler | mdbooth: bug in latest ansible-core, try using the previous version. I think 2.4.10 rather than .11 | 18:30 |
| mdbooth | frickler: Will try that, thanks! | 18:31 |
| mdbooth | frickler: I think that worked. Many thanks! | 18:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!