Wednesday, 2024-02-21

opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core versions to 2.15 and 2.16  https://review.opendev.org/c/openstack/kolla-ansible/+/90752208:25
opendevreviewMichal Nasiadka proposed openstack/kolla master: Move curl sources to sources.py  https://review.opendev.org/c/openstack/kolla/+/90898908:26
jovialAny idea why I'd see: Internal error: GET operation failed when using podman? This occurs when pulling an Image: "Ensure fluentd image is present for label check"09:26
jovialLooks like I needed to do: `sudo systemctl start podman`. So much for daemonless :-D09:43
mnasiadkaI think CI uses a-c-k podman role, and you haven't used that - right jovial? ;-)09:57
jovialI thought I did, but it is true that the podman package wasn't even installed. So I possibly didn't set the variable correctly (will take a deeper look)10:03
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964410:10
SvenKieskemnasiadka: don't we need renos for the backports for https://bugs.launchpad.net/bugs/2049762 ? as this disables stuff that was previously possible? even if it's a security fix?10:17
mnasiadkaI can follow up with a reno10:18
mnasiadkaI don't need it, but the users blindly using kolla-ansible and not paying attention to anything might want it :)10:18
SvenKieskety, well yeah, I also would like to have more informed users (sometimes) :)10:19
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964410:38
opendevreviewVerification of a change to openstack/kayobe master failed: Fix typo in example  https://review.opendev.org/c/openstack/kayobe/+/90959710:44
jovial^ Bifrost issues: Failed to configure Ironic Inspector rule: DiscoveryFailure. Will add to the whiteboard :'(10:56
opendevreviewUwe Jäger proposed openstack/kolla-ansible master: Allow overriding of Skyline logos  https://review.opendev.org/c/openstack/kolla-ansible/+/90948111:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core versions to 2.15 and 2.16  https://review.opendev.org/c/openstack/kolla-ansible/+/90752211:32
jovialmnasiadka, I'm confused this is what I see of the a-c-k role ran: https://paste.openstack.org/show/bwsHNWnNLBfIgSbUeImW/11:53
jovialI think `preset: disabled` means that the OS won't enable it on installation, right?11:54
opendevreviewMark Goddard proposed openstack/ansible-collection-kolla master: docker_sdk/podman_sdk: Install packaging Python package  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/90720711:57
jovialShall I explitly start and enable the service like for docker: https://github.com/openstack/ansible-collection-kolla/blob/master/roles/docker/tasks/install.yml#L47-L53?11:58
opendevreviewAlex Welsh proposed openstack/kolla-ansible master: Add password rotation docs page  https://review.opendev.org/c/openstack/kolla-ansible/+/90422112:02
jovialOh, realized it is enabled in the podman_sdk role instead: https://github.com/openstack/ansible-collection-kolla/blob/master/roles/podman_sdk/tasks/main.yml#L65-L7012:19
mnasiadkajovial: that's weird, should we move it?12:26
jovialmnasiadka, Yeah, I think it makes sense that that task should be in the podman role instead.12:35
mnasiadkajovial: feel free to propose that12:41
kevkoLastly we decided it's enough to just put +2 and merge for unmaintained branches, right ? 13:19
mnasiadkaNo, we didn't13:19
kevkoreally ? 13:19
mnasiadkaThere was your proposal, but there were no other cores to agree13:19
mnasiadkaYou can propose that today13:19
kevkoah, ok ... 13:20
mnasiadkamgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting in 7 minutes13:53
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Feb 21 14:00:08 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadka#topic rollcall14:00
mnasiadkao/14:00
mmalchuk\o14:00
jovialo/14:00
mattcreeso/14:00
frickler\o14:01
kevko\o/14:01
SvenKieskeo/14:01
SvenKieske(time flies)14:01
halomiva\o14:01
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964414:02
mnasiadka#topic agenda14:02
mnasiadka* CI status14:02
mnasiadka* Release tasks14:02
mnasiadka* Regular stable releases (first meeting in a month)14:02
mnasiadka* Current cycle planning14:02
mnasiadka* Additional agenda (from whiteboard)14:02
mnasiadka* Open discussion14:02
mnasiadka#topic CI status14:02
mnasiadkaSoo... OpenSearch 2.12 has broken our builds14:03
mnasiadkafixing with 909644 - once it passes, I'll add some more meat in commit msg14:03
mnasiadkaand raise a proper bug for backporting14:03
mnasiadkaany other CI breakages?14:03
mnasiadkajovial: you mentioned something in Kayobe?14:03
jovialYeah, seed jobs are failing14:04
mnasiadkaok14:04
mnasiadkaand Cinder seems to be finally fixed (upgrade cephadm jobs break)14:04
jovialI've marked it as red in the whiteboard14:04
mnasiadkaso - new breakages, old breakages, fun14:05
mnasiadka#topic Release tasks14:05
bbezako/14:05
mnasiadkaThis week is R-6 - Final release for non-client libraries14:05
mnasiadkanext week is feature freeze (but not for Kolla)14:05
mnasiadka#topic Current cycle planning14:06
mnasiadkaAnybody wants to discuss any patches?14:06
kevkoyep14:06
kevkojust trivial one https://review.opendev.org/c/openstack/kolla-ansible/+/90842914:06
jovialNothing to discuss yet, but I was going to push a WIP version of podman support for kayobe for some initial comments14:07
mmalchukhttps://review.opendev.org/c/openstack/kolla-ansible/+/907495 can be merged, fixed depends-on14:07
mmalchukplease review trivial: https://review.opendev.org/c/openstack/kayobe/+/90920514:08
mmalchukand https://review.opendev.org/c/openstack/kayobe/+/90911314:08
mmalchukand last trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/90725414:09
mnasiadkadiscuss, not trivial patches concert, but whatever :)14:09
jovialWas also going to try bumping kayobe ansible to 2.16 to follow: https://review.opendev.org/c/openstack/kolla-ansible/+/907522. I think I managed to get the issue we hit in wcmatch (the globbing library) fixed upstream. We hit this when using newer ansible: https://github.com/facelessuser/wcmatch/issues/21014:09
fricklermnasiadka: +1 was about to ask the same14:10
SvenKieskejust my whiteboard topic, but that's for later I guess :)14:11
mnasiadkajovial: seems a patch was released 20 hours ago?14:11
jovialIndeed, so hopefully just a case of bumping the dependencies. I've tested newer 2.15.x, but haven't tried 2.16 yet.14:12
mnasiadkaok14:12
mnasiadkaOpenSearch breakage broke the opensearch CI jobs in kolla-ansible (because for some bloody reason we have opensearch-dashboards 2.12 and opensearch 2.11)14:13
mnasiadkabut it will get fixed when we fix the build14:13
mnasiadkanot asking why the heck anybody would want to do a post script in rpm/deb based on env var (for setting initial password)14:13
mnasiadkabut that's beyond my comprehension14:13
mnasiadka#topic Additional agenda (from whiteboard)14:14
mnasiadkaLet's go forward14:14
SvenKieskemhm, very unfortunate that we have to set that password, I take it there's no way around that?14:14
SvenKieskebut we can discuss this also later/in the patch14:14
mnasiadkaSvenKieske: no14:14
mnasiadka(SvenKieske): Do we want to officially support ansibles --limit option for upgrades? it's at least broken for some services e.g. https://bugs.launchpad.net/kolla-ansible/+bug/2054348 14:14
mnasiadkaSo, from my perspective - supporting --limit is complicated - as in - in most cases it will work14:14
mnasiadkaunless it doesn't14:15
mnasiadkaand we can't really test that in CI14:15
SvenKieskeyeah, so I'm open to just document that it doesn't work for most cases, but imho we should at least do that, because I ran into issues and it's always good if I can point to docs saying: that's not supported :D14:15
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Fix 'cinder-backup' service when Swift with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90749514:15
jovialWere you limiting to a single controller?14:16
SvenKieskeI can volunteer to do a docs patch for that. if there is consensus around this? or do we want to support it for some use cases at least (e.g. nova?)14:16
SvenKieskelimiting to a single compute node, but nova-api upgrade stuff breaks then, see the bug report, I hope everything is included there (it was late)14:17
kevkohmm, that bug in launchpad regarding limit ...it's also because in k-a we are using host[0] and register14:17
fricklerIMO limiting to computes needs to be supported, so we should look into fixing that, even if we don't have a CI job for that14:18
SvenKieskethe thing is, the nova-upgrade task registers results as a host var which aren't accessible by other hosts, as a result the playbook stops with an error when that host is not part of the actual play.14:18
jovialSvenKieske, I don't think the bug mentions the limit you used in the bug report, might be handy to add14:18
mattcreesI'd say there's no harm in fixing bugs like the one Sven found, but given we have no way to regularly test it we shouldn't officially be supporting it. 14:18
kevkofrickler: agree14:18
mnasiadkaWell, I agree that 1. we should state in the docs that running with limit is not tested 2. make that work 3. accept bugs for --limit with low priority (or wishlist priority) stating it's not tested in CI and requires more work14:19
SvenKieskeokay, makes sense, because I agree limiting to certain compute nodes is a often used feature14:19
mnasiadkaDoes that make sense?14:19
fricklermnasiadka: +114:20
SvenKieskeso I guess we should refactor "register:" stuff to "set_fact" instead14:20
mattcrees+1 from me14:20
SvenKieskemnasiadka: +1 from me as well14:20
mnasiadkaOk, who's the volunteer to do the docs update?14:20
SvenKieskeI can take care of 1.) push a patch that states that --limit is currently not tested14:20
SvenKieskedo we document that just in the upgrade docs or anywhere else?14:21
mnasiadkaSupport matrix maybe?14:21
mnasiadkaAnyway, try to find a good spot and let's discuss in Gerrit14:22
mnasiadkanext one14:22
mnasiadka(halomiva): bump version of docker-py so its supports cgroupns for this change https://review.opendev.org/c/openstack/kolla-ansible/+/90829514:22
SvenKieskeokay14:23
mnasiadkahalomiva: bump to 7.0.0?14:23
halomivais it okay bumping it from 3.0 to 6.0 ? 14:23
mnasiadkato 6.0 yes, but lower than 7.014:23
mnasiadkathere was a bug with the url syntax checker there that failed our Kolla builds14:24
mnasiadkaso I anticipate similar in k-a14:24
mnasiadkaand they haven't still released 7.0.114:24
halomivaokay, that should be enough, is there also some minimal version of docker that we support? 14:24
halomivabecause there might be problem that docker api doesnt support it either before 20.1 i think 14:24
opendevreviewJake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic.  https://review.opendev.org/c/openstack/kayobe/+/90967114:24
mnasiadkahalomiva: https://github.com/openstack/kolla-ansible/blob/d30fb56c2aabc04fdc922a16cb85194c7e587459/ansible/roles/prechecks/vars/main.yml#L214:25
mnasiadkawe have this14:25
mnasiadkaso it might need a bump14:25
halomivaokay i will test it with 6.0 and 20.10 once i finish unit tests14:26
mnasiadkaanother thing is refactor should not require a version bump in theory...14:26
jovialIncidentally, docker-py was also breaking the kayobe image build job. I wonder if I should fix to <7 as the new release doesn't seem forthcoming.14:26
mnasiadkajovial: it breaks when you push to a registry with non-standard port - so yes, pinning is advised ;-)14:26
mnasiadkahalomiva: is there any way we could do the refactor without bumping versions, and then just bump versions for some additional functionality?14:27
halomivain theory yes but the reason low level API is used is because there wasnt higher level capable of doing it at that time 14:27
SvenKieskemhm, should we maybe get the precheck min versions from our requirements.txt? why have the versions in multiple files? it's harder to maintain, no? (it might be not trivial to parse the min version though)14:27
SvenKieskethat should be the issue about the broken docker tag parsing with regards to registry ports: https://github.com/docker/docker-py/issues/319514:28
jovialmnasiadka: K, sounds like a good plan as it could be breaking people already and it would be good to get that job back to voting14:28
SvenKieskefix is here: https://github.com/docker/docker-py/commit/3ec5a6849a6cad4c5f5f3bafb5f74b5827fec14c14:28
mnasiadkaok, let's go to the next one14:28
mnasiadka(halomiva): refactor of kolla_container_facts and volume_facts metioned here https://review.opendev.org/c/openstack/kolla-ansible/+/90583714:29
halomivamnasiadka: like we can do it without supporting cgroupns but then i wasnt able to run virtuals because of libvirt issue 14:29
halomivabasically as I said in the last comment, what do you think about it?14:29
mnasiadkahalomiva: I like the idea in the last comment - let me reply there14:30
mnasiadkaok then, that was the last topic14:31
mnasiadka#topic Open discussion14:31
mnasiadkaAnybody anything?14:31
SvenKieskedoesn't seem to be the case :)14:33
frickleroh, just one thing to mention14:34
fricklerPTL and TC nomination period has started14:34
mnasiadkaYes, I signed up for Kolla14:34
mnasiadkaAgain14:34
SvenKieskethat sounds a little tired? :)14:34
mnasiadkaNaah14:35
mnasiadkaI like doing that, from other perspective maybe some fresh blood would be good14:35
mnasiadkaWe can discuss on the PTG if anybody wants to run for E14:36
SvenKieskewe could try the distributed PTL stuff, I don't feel I understand all the organizational stuff enough just yet, I know it's mostly documented :)14:37
mmalchukI want, but can't ;)14:38
mnasiadkadistributed PTL looks like a way to tell nobody is responsible ;)14:38
mnasiadkaOk, I guess that's enough for today14:39
mnasiadkaThank you for coming and see you next week!14:39
mnasiadka#endmeeting14:39
opendevmeetMeeting ended Wed Feb 21 14:39:28 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:39
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.html14:39
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.txt14:39
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-02-21-14.00.log.html14:39
fricklerthx mnasiadka 14:39
kevkothx14:39
mmalchukthanks mnasiadka 14:40
SvenKieskethx14:40
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964414:41
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Adjust Ceph metrics scrape interval in Prometheus  https://review.opendev.org/c/openstack/kolla-ansible/+/90212914:53
SvenKieskedid anybody test to replace pip with "uv"? It's advertised as a drop-in replacement which is faster: https://astral.sh/blog/uv15:32
mnasiadkabbezak, frickler, kevko: marked https://review.opendev.org/c/openstack/kolla/+/909644 with RP+2 - I'll create a bug and update commit message when CI is green15:35
opendevreviewWill Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal  https://review.opendev.org/c/openstack/kayobe/+/90968515:36
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Add podman support  https://review.opendev.org/c/openstack/kayobe/+/90968615:36
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964415:37
mnasiadkaseems it's passing15:37
SvenKieskedo we add docs for users later/in k-a? don't know enough about the dashboard password settings, but we should make sure users don't start with the default password..15:38
mnasiadkain k-a we're disabling security plugin, because it requires tls - which we don't support yet15:40
mnasiadkaso that's not used at all for now15:40
mnasiadkareplied in Gerrit15:42
jovialI've posted a few comments on my WIP patch to add podman support for kayobe: https://review.opendev.org/c/openstack/kayobe/+/909686 if anyone wants to provide some early direction.15:47
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Add podman support  https://review.opendev.org/c/openstack/kayobe/+/90968615:48
opendevreviewWill Szumski proposed openstack/kayobe master: Revert "Pin ansible-core at 2.15.6"  https://review.opendev.org/c/openstack/kayobe/+/90827615:51
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Add podman support  https://review.opendev.org/c/openstack/kayobe/+/90968615:55
opendevreviewMerged openstack/kolla-ansible master: Fix gnocchi-metricd when TLS and Swift enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90725416:02
opendevreviewWill Szumski proposed openstack/ansible-collection-kolla master: Move podman socket activation to podman role  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/90969216:21
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Add podman support  https://review.opendev.org/c/openstack/kayobe/+/90968616:26
opendevreviewWill Szumski proposed openstack/kolla-ansible master: Adds the neutron_service_limit option  https://review.opendev.org/c/openstack/kolla-ansible/+/87777616:38
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.2: Fix gnocchi-metricd when TLS and Swift enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90955618:09
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix gnocchi-metricd when TLS and Swift enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90955718:10
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix gnocchi-metricd when TLS and Swift enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90955818:10
opendevreviewWill Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal  https://review.opendev.org/c/openstack/kayobe/+/90968518:28
opendevreviewWill Szumski proposed openstack/kayobe master: WIP: Add podman support  https://review.opendev.org/c/openstack/kayobe/+/90968618:28
skariyapHi Team,  Need to run something by you all, We are running Openstack Antelope using Kolla-ansible. We are in a situation where we need to re-deploy the Openstack Antelope with new set of Openstack component images. Just tag change for most images except glance.  We have tried both the deploy and upgrade options. What we are running into post re-deploy is rabbitmq running into some weird issue, where neutron-agents are not able to connect t18:33
skariyapNeutron-server thinks all the agents are dead. The issue is not 100% consistent, looks like some timing is involved. To get out of this issue, we ended up stopping all rabbitmq containers and restart them one by one.  Would like to understand if we have any missing/incorrect steps in our process? OR Is this a known issue?18:34
opendevreviewVerification of a change to openstack/kolla master failed: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964418:56
opendevreviewVerification of a change to openstack/kolla master failed: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964420:22
opendevreviewPatrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load  https://review.opendev.org/c/openstack/kolla-ansible/+/90979721:34
opendevreviewPatrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load  https://review.opendev.org/c/openstack/kolla-ansible/+/90979722:00
opendevreviewMerged openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/90964422:41
opendevreviewPatrick Lüdeke proposed openstack/kolla-ansible master: Reduce RabbitMQ health check CPU load  https://review.opendev.org/c/openstack/kolla-ansible/+/90979722:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!