| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core versions to 2.15 and 2.16 https://review.opendev.org/c/openstack/kolla-ansible/+/907522 | 08:25 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Move curl sources to sources.py https://review.opendev.org/c/openstack/kolla/+/908989 | 08:26 |
| jovial | Any idea why I'd see: Internal error: GET operation failed when using podman? This occurs when pulling an Image: "Ensure fluentd image is present for label check" | 09:26 |
| jovial | Looks like I needed to do: `sudo systemctl start podman`. So much for daemonless :-D | 09:43 |
| mnasiadka | I think CI uses a-c-k podman role, and you haven't used that - right jovial? ;-) | 09:57 |
| jovial | I thought I did, but it is true that the podman package wasn't even installed. So I possibly didn't set the variable correctly (will take a deeper look) | 10:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 10:10 |
| SvenKieske | mnasiadka: don't we need renos for the backports for https://bugs.launchpad.net/bugs/2049762 ? as this disables stuff that was previously possible? even if it's a security fix? | 10:17 |
| mnasiadka | I can follow up with a reno | 10:18 |
| mnasiadka | I don't need it, but the users blindly using kolla-ansible and not paying attention to anything might want it :) | 10:18 |
| SvenKieske | ty, well yeah, I also would like to have more informed users (sometimes) :) | 10:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 10:38 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix typo in example https://review.opendev.org/c/openstack/kayobe/+/909597 | 10:44 |
| jovial | ^ Bifrost issues: Failed to configure Ironic Inspector rule: DiscoveryFailure. Will add to the whiteboard :'( | 10:56 |
| opendevreview | Uwe Jäger proposed openstack/kolla-ansible master: Allow overriding of Skyline logos https://review.opendev.org/c/openstack/kolla-ansible/+/909481 | 11:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core versions to 2.15 and 2.16 https://review.opendev.org/c/openstack/kolla-ansible/+/907522 | 11:32 |
| jovial | mnasiadka, I'm confused this is what I see of the a-c-k role ran: https://paste.openstack.org/show/bwsHNWnNLBfIgSbUeImW/ | 11:53 |
| jovial | I think `preset: disabled` means that the OS won't enable it on installation, right? | 11:54 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: docker_sdk/podman_sdk: Install packaging Python package https://review.opendev.org/c/openstack/ansible-collection-kolla/+/907207 | 11:57 |
| jovial | Shall I explitly start and enable the service like for docker: https://github.com/openstack/ansible-collection-kolla/blob/master/roles/docker/tasks/install.yml#L47-L53? | 11:58 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible master: Add password rotation docs page https://review.opendev.org/c/openstack/kolla-ansible/+/904221 | 12:02 |
| jovial | Oh, realized it is enabled in the podman_sdk role instead: https://github.com/openstack/ansible-collection-kolla/blob/master/roles/podman_sdk/tasks/main.yml#L65-L70 | 12:19 |
| mnasiadka | jovial: that's weird, should we move it? | 12:26 |
| jovial | mnasiadka, Yeah, I think it makes sense that that task should be in the podman role instead. | 12:35 |
| mnasiadka | jovial: feel free to propose that | 12:41 |
| kevko | Lastly we decided it's enough to just put +2 and merge for unmaintained branches, right ? | 13:19 |
| mnasiadka | No, we didn't | 13:19 |
| kevko | really ? | 13:19 |
| mnasiadka | There was your proposal, but there were no other cores to agree | 13:19 |
| mnasiadka | You can propose that today | 13:19 |
| kevko | ah, ok ... | 13:20 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting in 7 minutes | 13:53 |
| mnasiadka | #startmeeting kolla | 14:00 |
| opendevmeet | Meeting started Wed Feb 21 14:00:08 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:00 |
| mnasiadka | #topic rollcall | 14:00 |
| mnasiadka | o/ | 14:00 |
| mmalchuk | \o | 14:00 |
| jovial | o/ | 14:00 |
| mattcrees | o/ | 14:00 |
| frickler | \o | 14:01 |
| kevko | \o/ | 14:01 |
| SvenKieske | o/ | 14:01 |
| SvenKieske | (time flies) | 14:01 |
| halomiva | \o | 14:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 14:02 |
| mnasiadka | #topic agenda | 14:02 |
| mnasiadka | * CI status | 14:02 |
| mnasiadka | * Release tasks | 14:02 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 14:02 |
| mnasiadka | * Current cycle planning | 14:02 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:02 |
| mnasiadka | * Open discussion | 14:02 |
| mnasiadka | #topic CI status | 14:02 |
| mnasiadka | Soo... OpenSearch 2.12 has broken our builds | 14:03 |
| mnasiadka | fixing with 909644 - once it passes, I'll add some more meat in commit msg | 14:03 |
| mnasiadka | and raise a proper bug for backporting | 14:03 |
| mnasiadka | any other CI breakages? | 14:03 |
| mnasiadka | jovial: you mentioned something in Kayobe? | 14:03 |
| jovial | Yeah, seed jobs are failing | 14:04 |
| mnasiadka | ok | 14:04 |
| mnasiadka | and Cinder seems to be finally fixed (upgrade cephadm jobs break) | 14:04 |
| jovial | I've marked it as red in the whiteboard | 14:04 |
| mnasiadka | so - new breakages, old breakages, fun | 14:05 |
| mnasiadka | #topic Release tasks | 14:05 |
| bbezak | o/ | 14:05 |
| mnasiadka | This week is R-6 - Final release for non-client libraries | 14:05 |
| mnasiadka | next week is feature freeze (but not for Kolla) | 14:05 |
| mnasiadka | #topic Current cycle planning | 14:06 |
| mnasiadka | Anybody wants to discuss any patches? | 14:06 |
| kevko | yep | 14:06 |
| kevko | just trivial one https://review.opendev.org/c/openstack/kolla-ansible/+/908429 | 14:06 |
| jovial | Nothing to discuss yet, but I was going to push a WIP version of podman support for kayobe for some initial comments | 14:07 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla-ansible/+/907495 can be merged, fixed depends-on | 14:07 |
| mmalchuk | please review trivial: https://review.opendev.org/c/openstack/kayobe/+/909205 | 14:08 |
| mmalchuk | and https://review.opendev.org/c/openstack/kayobe/+/909113 | 14:08 |
| mmalchuk | and last trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/907254 | 14:09 |
| mnasiadka | discuss, not trivial patches concert, but whatever :) | 14:09 |
| jovial | Was also going to try bumping kayobe ansible to 2.16 to follow: https://review.opendev.org/c/openstack/kolla-ansible/+/907522. I think I managed to get the issue we hit in wcmatch (the globbing library) fixed upstream. We hit this when using newer ansible: https://github.com/facelessuser/wcmatch/issues/210 | 14:09 |
| frickler | mnasiadka: +1 was about to ask the same | 14:10 |
| SvenKieske | just my whiteboard topic, but that's for later I guess :) | 14:11 |
| mnasiadka | jovial: seems a patch was released 20 hours ago? | 14:11 |
| jovial | Indeed, so hopefully just a case of bumping the dependencies. I've tested newer 2.15.x, but haven't tried 2.16 yet. | 14:12 |
| mnasiadka | ok | 14:12 |
| mnasiadka | OpenSearch breakage broke the opensearch CI jobs in kolla-ansible (because for some bloody reason we have opensearch-dashboards 2.12 and opensearch 2.11) | 14:13 |
| mnasiadka | but it will get fixed when we fix the build | 14:13 |
| mnasiadka | not asking why the heck anybody would want to do a post script in rpm/deb based on env var (for setting initial password) | 14:13 |
| mnasiadka | but that's beyond my comprehension | 14:13 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 14:14 |
| mnasiadka | Let's go forward | 14:14 |
| SvenKieske | mhm, very unfortunate that we have to set that password, I take it there's no way around that? | 14:14 |
| SvenKieske | but we can discuss this also later/in the patch | 14:14 |
| mnasiadka | SvenKieske: no | 14:14 |
| mnasiadka | (SvenKieske): Do we want to officially support ansibles --limit option for upgrades? it's at least broken for some services e.g. https://bugs.launchpad.net/kolla-ansible/+bug/2054348 | 14:14 |
| mnasiadka | So, from my perspective - supporting --limit is complicated - as in - in most cases it will work | 14:14 |
| mnasiadka | unless it doesn't | 14:15 |
| mnasiadka | and we can't really test that in CI | 14:15 |
| SvenKieske | yeah, so I'm open to just document that it doesn't work for most cases, but imho we should at least do that, because I ran into issues and it's always good if I can point to docs saying: that's not supported :D | 14:15 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Fix 'cinder-backup' service when Swift with TLS enabled https://review.opendev.org/c/openstack/kolla-ansible/+/907495 | 14:15 |
| jovial | Were you limiting to a single controller? | 14:16 |
| SvenKieske | I can volunteer to do a docs patch for that. if there is consensus around this? or do we want to support it for some use cases at least (e.g. nova?) | 14:16 |
| SvenKieske | limiting to a single compute node, but nova-api upgrade stuff breaks then, see the bug report, I hope everything is included there (it was late) | 14:17 |
| kevko | hmm, that bug in launchpad regarding limit ...it's also because in k-a we are using host[0] and register | 14:17 |
| frickler | IMO limiting to computes needs to be supported, so we should look into fixing that, even if we don't have a CI job for that | 14:18 |
| SvenKieske | the thing is, the nova-upgrade task registers results as a host var which aren't accessible by other hosts, as a result the playbook stops with an error when that host is not part of the actual play. | 14:18 |
| jovial | SvenKieske, I don't think the bug mentions the limit you used in the bug report, might be handy to add | 14:18 |
| mattcrees | I'd say there's no harm in fixing bugs like the one Sven found, but given we have no way to regularly test it we shouldn't officially be supporting it. | 14:18 |
| kevko | frickler: agree | 14:18 |
| mnasiadka | Well, I agree that 1. we should state in the docs that running with limit is not tested 2. make that work 3. accept bugs for --limit with low priority (or wishlist priority) stating it's not tested in CI and requires more work | 14:19 |
| SvenKieske | okay, makes sense, because I agree limiting to certain compute nodes is a often used feature | 14:19 |
| mnasiadka | Does that make sense? | 14:19 |
| frickler | mnasiadka: +1 | 14:20 |
| SvenKieske | so I guess we should refactor "register:" stuff to "set_fact" instead | 14:20 |
| mattcrees | +1 from me | 14:20 |
| SvenKieske | mnasiadka: +1 from me as well | 14:20 |
| mnasiadka | Ok, who's the volunteer to do the docs update? | 14:20 |
| SvenKieske | I can take care of 1.) push a patch that states that --limit is currently not tested | 14:20 |
| SvenKieske | do we document that just in the upgrade docs or anywhere else? | 14:21 |
| mnasiadka | Support matrix maybe? | 14:21 |
| mnasiadka | Anyway, try to find a good spot and let's discuss in Gerrit | 14:22 |
| mnasiadka | next one | 14:22 |
| mnasiadka | (halomiva): bump version of docker-py so its supports cgroupns for this change https://review.opendev.org/c/openstack/kolla-ansible/+/908295 | 14:22 |
| SvenKieske | okay | 14:23 |
| mnasiadka | halomiva: bump to 7.0.0? | 14:23 |
| halomiva | is it okay bumping it from 3.0 to 6.0 ? | 14:23 |
| mnasiadka | to 6.0 yes, but lower than 7.0 | 14:23 |
| mnasiadka | there was a bug with the url syntax checker there that failed our Kolla builds | 14:24 |
| mnasiadka | so I anticipate similar in k-a | 14:24 |
| mnasiadka | and they haven't still released 7.0.1 | 14:24 |
| halomiva | okay, that should be enough, is there also some minimal version of docker that we support? | 14:24 |
| halomiva | because there might be problem that docker api doesnt support it either before 20.1 i think | 14:24 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic. https://review.opendev.org/c/openstack/kayobe/+/909671 | 14:24 |
| mnasiadka | halomiva: https://github.com/openstack/kolla-ansible/blob/d30fb56c2aabc04fdc922a16cb85194c7e587459/ansible/roles/prechecks/vars/main.yml#L2 | 14:25 |
| mnasiadka | we have this | 14:25 |
| mnasiadka | so it might need a bump | 14:25 |
| halomiva | okay i will test it with 6.0 and 20.10 once i finish unit tests | 14:26 |
| mnasiadka | another thing is refactor should not require a version bump in theory... | 14:26 |
| jovial | Incidentally, docker-py was also breaking the kayobe image build job. I wonder if I should fix to <7 as the new release doesn't seem forthcoming. | 14:26 |
| mnasiadka | jovial: it breaks when you push to a registry with non-standard port - so yes, pinning is advised ;-) | 14:26 |
| mnasiadka | halomiva: is there any way we could do the refactor without bumping versions, and then just bump versions for some additional functionality? | 14:27 |
| halomiva | in theory yes but the reason low level API is used is because there wasnt higher level capable of doing it at that time | 14:27 |
| SvenKieske | mhm, should we maybe get the precheck min versions from our requirements.txt? why have the versions in multiple files? it's harder to maintain, no? (it might be not trivial to parse the min version though) | 14:27 |
| SvenKieske | that should be the issue about the broken docker tag parsing with regards to registry ports: https://github.com/docker/docker-py/issues/3195 | 14:28 |
| jovial | mnasiadka: K, sounds like a good plan as it could be breaking people already and it would be good to get that job back to voting | 14:28 |
| SvenKieske | fix is here: https://github.com/docker/docker-py/commit/3ec5a6849a6cad4c5f5f3bafb5f74b5827fec14c | 14:28 |
| mnasiadka | ok, let's go to the next one | 14:28 |
| mnasiadka | (halomiva): refactor of kolla_container_facts and volume_facts metioned here https://review.opendev.org/c/openstack/kolla-ansible/+/905837 | 14:29 |
| halomiva | mnasiadka: like we can do it without supporting cgroupns but then i wasnt able to run virtuals because of libvirt issue | 14:29 |
| halomiva | basically as I said in the last comment, what do you think about it? | 14:29 |
| mnasiadka | halomiva: I like the idea in the last comment - let me reply there | 14:30 |
| mnasiadka | ok then, that was the last topic | 14:31 |
| mnasiadka | #topic Open discussion | 14:31 |
| mnasiadka | Anybody anything? | 14:31 |
| SvenKieske | doesn't seem to be the case :) | 14:33 |
| frickler | oh, just one thing to mention | 14:34 |
| frickler | PTL and TC nomination period has started | 14:34 |
| mnasiadka | Yes, I signed up for Kolla | 14:34 |
| mnasiadka | Again | 14:34 |
| SvenKieske | that sounds a little tired? :) | 14:34 |
| mnasiadka | Naah | 14:35 |
| mnasiadka | I like doing that, from other perspective maybe some fresh blood would be good | 14:35 |
| mnasiadka | We can discuss on the PTG if anybody wants to run for E | 14:36 |
| SvenKieske | we could try the distributed PTL stuff, I don't feel I understand all the organizational stuff enough just yet, I know it's mostly documented :) | 14:37 |
| mmalchuk | I want, but can't ;) | 14:38 |
| mnasiadka | distributed PTL looks like a way to tell nobody is responsible ;) | 14:38 |
| mnasiadka | Ok, I guess that's enough for today | 14:39 |
| mnasiadka | Thank you for coming and see you next week! | 14:39 |
| mnasiadka | #endmeeting | 14:39 |
| opendevmeet | Meeting ended Wed Feb 21 14:39:28 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:39 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.html | 14:39 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.txt | 14:39 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.log.html | 14:39 |
| frickler | thx mnasiadka | 14:39 |
| kevko | thx | 14:39 |
| mmalchuk | thanks mnasiadka | 14:40 |
| SvenKieske | thx | 14:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 14:41 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Adjust Ceph metrics scrape interval in Prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/902129 | 14:53 |
| SvenKieske | did anybody test to replace pip with "uv"? It's advertised as a drop-in replacement which is faster: https://astral.sh/blog/uv | 15:32 |
| mnasiadka | bbezak, frickler, kevko: marked https://review.opendev.org/c/openstack/kolla/+/909644 with RP+2 - I'll create a bug and update commit message when CI is green | 15:35 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal https://review.opendev.org/c/openstack/kayobe/+/909685 | 15:36 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 15:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 15:37 |
| mnasiadka | seems it's passing | 15:37 |
| SvenKieske | do we add docs for users later/in k-a? don't know enough about the dashboard password settings, but we should make sure users don't start with the default password.. | 15:38 |
| mnasiadka | in k-a we're disabling security plugin, because it requires tls - which we don't support yet | 15:40 |
| mnasiadka | so that's not used at all for now | 15:40 |
| mnasiadka | replied in Gerrit | 15:42 |
| jovial | I've posted a few comments on my WIP patch to add podman support for kayobe: https://review.opendev.org/c/openstack/kayobe/+/909686 if anyone wants to provide some early direction. | 15:47 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 15:48 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Revert "Pin ansible-core at 2.15.6" https://review.opendev.org/c/openstack/kayobe/+/908276 | 15:51 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 15:55 |
| opendevreview | Merged openstack/kolla-ansible master: Fix gnocchi-metricd when TLS and Swift enabled https://review.opendev.org/c/openstack/kolla-ansible/+/907254 | 16:02 |
| opendevreview | Will Szumski proposed openstack/ansible-collection-kolla master: Move podman socket activation to podman role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/909692 | 16:21 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 16:26 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Adds the neutron_service_limit option https://review.opendev.org/c/openstack/kolla-ansible/+/877776 | 16:38 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.2: Fix gnocchi-metricd when TLS and Swift enabled https://review.opendev.org/c/openstack/kolla-ansible/+/909556 | 18:09 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix gnocchi-metricd when TLS and Swift enabled https://review.opendev.org/c/openstack/kolla-ansible/+/909557 | 18:10 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix gnocchi-metricd when TLS and Swift enabled https://review.opendev.org/c/openstack/kolla-ansible/+/909558 | 18:10 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal https://review.opendev.org/c/openstack/kayobe/+/909685 | 18:28 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 18:28 |
| skariyap | Hi Team, Need to run something by you all, We are running Openstack Antelope using Kolla-ansible. We are in a situation where we need to re-deploy the Openstack Antelope with new set of Openstack component images. Just tag change for most images except glance. We have tried both the deploy and upgrade options. What we are running into post re-deploy is rabbitmq running into some weird issue, where neutron-agents are not able to connect t | 18:33 |
| skariyap | Neutron-server thinks all the agents are dead. The issue is not 100% consistent, looks like some timing is involved. To get out of this issue, we ended up stopping all rabbitmq containers and restart them one by one. Would like to understand if we have any missing/incorrect steps in our process? OR Is this a known issue? | 18:34 |
| opendevreview | Verification of a change to openstack/kolla master failed: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 18:56 |
| opendevreview | Verification of a change to openstack/kolla master failed: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 20:22 |
| opendevreview | Patrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load https://review.opendev.org/c/openstack/kolla-ansible/+/909797 | 21:34 |
| opendevreview | Patrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load https://review.opendev.org/c/openstack/kolla-ansible/+/909797 | 22:00 |
| opendevreview | Merged openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD https://review.opendev.org/c/openstack/kolla/+/909644 | 22:41 |
| opendevreview | Patrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load https://review.opendev.org/c/openstack/kolla-ansible/+/909797 | 22:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!