Wednesday, 2024-07-10

opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role  https://review.opendev.org/c/openstack/kolla-ansible/+/79749802:04
opendevreviewTino Schmeier proposed openstack/kolla-ansible master: Use template instead of copy for prometheus-alert-rules  https://review.opendev.org/c/openstack/kolla-ansible/+/92239105:32
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Rewrite kolla-ansible CLI to python  https://review.opendev.org/c/openstack/kolla-ansible/+/92311006:14
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: DNM: test swift jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/92381706:48
stanDo anyone run Zitadel as the OIDC provider for Keystone?06:50
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: DNM: test swift jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/92381706:50
opendevreviewVerification of a change to openstack/ansible-collection-kolla master failed: Fix etc hosts generation for interfaces with dashes  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92354407:26
opendevreviewSven Kieske proposed openstack/kolla-ansible master: harden haproxy TLS configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/91540307:50
fricklermnasiadka: failing again, isn't that the "usual" version mismatch issue? "Not supported URL scheme http+docker" https://zuul.opendev.org/t/openstack/build/9547ada1a7684fcc80476fbefe4972e807:52
mnasiadkajovial: seems kayobe rocky9 is broken? ^^07:52
mnasiadkajovial: can you have a look?07:53
mnasiadkaworst case scenario we mark it as non-voting for now...07:53
SvenKieskegood morning o/07:54
SvenKieskemnasiadka: hope I could address all of your comments, thanks for the reviews07:54
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Rewrite kolla-ansible CLI to python  https://review.opendev.org/c/openstack/kolla-ansible/+/92311008:44
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: CI: Mark kayobe rocky9 job as non voting  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92383008:48
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: CI: Mark kayobe rocky9 job as non voting  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92383008:58
sylvrHello !09:29
sylvrDuring a node introspection, the IPA register the node in Bifrost right? shouldn't it also register its IPMI/BMC address ? I provide my BMC with IP using a DHCP server, and from what I understand this not the best way of doing it...09:31
sylvrSo, I tried to think about how to make this work in a "zero-touch" kind of way, with an IP that is used and provided dynamically during introspection, and after provisioning we change the BMC address for a static address similar to the one on the management network09:34
jovialsylvr: so the impi address always used to be set. Possibly we've had a regression there.09:42
jovialChanging the IPMI address sounds a bit overly complicated. Normally it just collects that in the introspection data.09:42
jovialPossibly you aren't exposing the ipmi interface to host os?09:43
jovial`/dev/impi` or whatever the path is09:43
jovialThat is normally a bios configuration option09:44
sylvrwell, I've been told that DHCP on OOB network shouldn't be done and that may be the reason why it failed...09:44
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Fix etc hosts generation for interfaces with dashes  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92354409:44
sylvrI don't remember changing bios config09:44
sylvrbut I have several nodes booted with devuser activated and accessible09:45
jovialsylvr: I don't see why you can't use DHCP on the OOB network providing that it doesn't start handing out IPs on your inspection network09:45
jovialLook for /dev/impi0. I think that is how inspector normally detects the ipmi_address09:46
sylvrnice then09:46
sylvryeah I can see /dev/ipmi009:48
jovialHere is the code: https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L229209:50
jovialUnsure why that isn't working for you. You'd have to check the loggs.09:51
sylvrjovial: running ipmitool lan print output the correct IP address, so that good09:56
sylvrjovial: ok, so in the logs I see that the process return 0 in less than a sec10:06
sylvrno error, just debug10:14
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Add precheck for Horizon config file renames in Caracal  https://review.opendev.org/c/openstack/kolla-ansible/+/92355810:16
jovialsylvr, what do you see in /proc/cmdline for the ipa-inspection-collectors kernel parameter?10:16
jovialJust want to double check you have the default collector enabled10:18
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383611:08
sylvrjovial: ```ipa.kernel ipa-inspection-callback-url= https://192.168.2.1:5050/v1/continue ipa-api-url=https://192.168.2.1:6385 systemd.journald.forward_to_console=yes BOOTIF=7c:d3:0a:bc:e0:10 nofb nomodeset vga=normal ipa-collect-lldp=1 ipa-inspection-collectors=default,logs,pci-devices ipa-inspection-benchmarks= ipa-insecure=1 ipa-debug=1 initrd=ipa.initramfs```11:11
sylvrjovial : I will have to shutdown my servers in a few minutes for 2 days, so if you want to test something I'll take notes and wait for the cluster to be powered on again ^^11:13
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383611:15
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383611:17
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383611:18
jovialslyvr: That kernel command line look good. Did you see any warnings like: `Cannot get BMC address` in the logs11:18
opendevreviewMerged openstack/ansible-collection-kolla master: CI: Mark kayobe rocky9 job as non voting  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92383011:21
SvenKieskecan we get RP+1 from any core on these backports? https://review.opendev.org/c/openstack/kolla-ansible/+/922922 https://review.opendev.org/c/openstack/kolla-ansible/+/922921 ty!11:21
jovialsylvr: is the address in the introspection data?11:23
fricklerSvenKieske: done. so many issues with that patch, but it seems I missed the opportunity for complain on master11:37
SvenKieskethe addition of the logger dependency? maybe write a bugreport? not sure though. I always find it hard to tell where to move comments/RFEs for already merged stuff as well :)11:40
mnasiadkaI guess bugs - but better just do a follow up12:01
mnasiadkaYou can’t force the author to do it ;)12:01
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Add precheck for Horizon config file renames in Caracal  https://review.opendev.org/c/openstack/kolla-ansible/+/92355812:04
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383612:15
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383612:16
opendevreviewMerged openstack/ansible-collection-kolla master: Fix etc hosts generation for interfaces with dashes  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92354412:17
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting Apt preferences  https://review.opendev.org/c/openstack/kayobe/+/92383612:24
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/90991212:31
SvenKieskemhm, did I miss it due to me being on vacation? is this periodic jobs error already fixed? "'kolla_dockerhub_credentials' is undefined." https://zuul.openstack.org/build/6a1506a91dc94568a589dbc436a2732a12:39
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/83694112:40
greatgatsbyHoping someone might have seen this before.  `openstack server list --all-projects` works fine, but `openstack server list --all-projects --host <compute>` returns empty.12:43
greatgatsbyI've tried multiple versions of OSC, copy/pasted the compute name right from other OSC output to ensure it's not a typo, I'm not sure what could be causing this12:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Force config drive for ipv6 jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/91418512:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Trust kolla generated CA cert  https://review.opendev.org/c/openstack/kolla-ansible/+/92255912:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Fix swift disk preparation  https://review.opendev.org/c/openstack/kolla-ansible/+/91394212:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Make demo router and network shared  https://review.opendev.org/c/openstack/kolla-ansible/+/92289912:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add rally role  https://review.opendev.org/c/openstack/kolla-ansible/+/92290012:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add REQUESTS_CA_BUNDLE to kolla-toolbox container  https://review.opendev.org/c/openstack/kolla-ansible/+/92310512:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Try to test via rally  https://review.opendev.org/c/openstack/kolla-ansible/+/92290112:44
SvenKieskegreatgatsby: are you actually sure the payload (server) runs on that specific host? :)12:46
opendevreviewWill Szumski proposed openstack/kayobe master: Support running without root privileges  https://review.opendev.org/c/openstack/kayobe/+/92343012:46
greatgatsbySvenKieske: 100%, there's also only 3 computes in this environment, and I cycled through them all with that command.  Oh, and if I use `--debug` I can see that the REQ RESP contains all the servers, OSC just isn't outputting it12:48
SvenKieskegreatgatsby: maybe a dns/hostname/service name problem then?12:50
greatgatsbysorry, that's RESP BODY: {"servers": [{"id": ...   in the debug output12:50
SvenKieskeso if all the information you provided is correct - but in 99% of cases this is a typo/name confusion imho - I would wildly guess at something like availability zones/host aggregates? but in theory that shouldn't be a problem if you are really running with admin credentials12:53
SvenKieskeis the compute service on this host maybe disabled?12:53
SvenKieskethat won't stop currently running instances but the compute service can't answer which instances are running if it's shut off :)12:54
greatgatsbyI would guess typo or something as well, but the RESP BODY literally contains the list of servers for the compute, OSC just isn't printing the table/json when I use --host, despite having data to print. I've tried new venvs, OSC from different hosts, etc.  Thanks for the feedback though, was hoping someone had seen this before.12:55
bbezakmeeting in 4! mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu darmach12:56
darmachCheerio (notification worked bbezak ;) )12:57
bbezakare you leaving already darmach?12:57
bbezak:)12:57
darmachClassic12:57
SvenKieskegreatgatsby: fwiw it works here, just tested. :)12:57
SvenKieskegreatgatsby: I would suggest searching through opensearch logs for the request and how it is handled via the req-id, so you can maybe trace it down to some compute service or some error message or something12:59
SvenKieskedid you try --debug?12:59
greatgatsbySvenKieske: thanks for the req-id suggestion.  Yes, with --debug I can literally see the list of servers for the compute I specified.  OSC gets the data, just doesn't print it12:59
opendevreviewMerged openstack/kolla-ansible stable/2023.2: Fix the docker container dimensions comparison for short notation  https://review.opendev.org/c/openstack/kolla-ansible/+/92292213:00
bbezak#startmeeting kolla13:00
opendevmeetMeeting started Wed Jul 10 13:00:18 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
opendevmeetThe meeting name has been set to 'kolla'13:00
bbezak#topic rollcall13:00
SvenKieskeo/13:00
frickler\o13:01
darmacho/13:01
bbezako/13:01
bbezaknot many of us today13:01
R-krcek0/13:01
jovial\o13:02
bbezak#topic agenda13:02
bbezak* Roll-call13:02
bbezak* Agenda13:02
bbezak* Announcements13:02
bbezak* Review action items from the last meeting13:02
bbezak* CI status13:02
bbezak* Release tasks13:02
bbezak* Regular stable releases (first meeting in a month)13:02
bbezak* Current cycle planning13:02
bbezak* Additional agenda (from whiteboard)13:02
bbezak* Open discussion13:02
bbezak#topic CI status13:03
bbezakexcept kayobe issues with docker - which is somewhat interesting that it came up now jovial13:03
bbezakI thought we've patched it everywhere13:03
bbezakCI looks ok I think13:04
bbezakunless you've got something to report13:04
SvenKieskeI found this periodic job error weird, but I have not much experience yet with debugging those, is that a common unrelated issue? https://zuul.openstack.org/build/6a1506a91dc94568a589dbc436a2732a13:05
jovialbbezak, I must admit I hadn't noticed the issue. Got a link to you patch?13:05
SvenKieskebecause I checked the code and it seems fine, nothing has changed in that area recently13:05
opendevreviewMerged openstack/kolla-ansible stable/2024.1: Fix the docker container dimensions comparison for short notation  https://review.opendev.org/c/openstack/kolla-ansible/+/92292113:06
opendevreviewMerged openstack/kolla stable/2023.2: add kayobe to release management doc's Launchpad admin section  https://review.opendev.org/c/openstack/kolla/+/92291413:06
opendevreviewMerged openstack/kolla stable/2023.1: add kayobe to release management doc's Launchpad admin section  https://review.opendev.org/c/openstack/kolla/+/92291513:06
opendevreviewMerged openstack/kolla stable/2024.1: add kayobe to release management doc's Launchpad admin section  https://review.opendev.org/c/openstack/kolla/+/92284413:06
bbezakjovial: https://zuul.opendev.org/t/openstack/build/9547ada1a7684fcc80476fbefe4972e813:06
kevko\o13:06
jovialthanks13:06
Viiin version 2024.1 the "prometheus data-source" in the grafana role does not work. there is an error during saving and testing "401 Unauthorized - There was an error returned querying the Prometheus API."13:06
mnasiadkaVii: we're having a meeting now - please wait for Open Discussion or after the meeting13:07
bbezakSvenKieske: looks like we forgot to add a credentials variables somewhere at first glance13:07
bbezakin older branches13:08
SvenKieskeah I didn't check the correct branch, just looked at master..13:09
mnasiadkaI think we updated these credentials at one point, but we had to change the variable name13:09
bbezakyeah, publish playbook got backported, but .zuul.d/* jobs are not pushing the new var13:10
bbezaklooks relatively easy, would you look into that SvenKieske13:10
SvenKieskeI guess I can do that :)13:10
bbezakit is my last day before 2 weeks vacation, so I won't have much time to fix it unfortunately13:10
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: ci: bump up ceph release  https://review.opendev.org/c/openstack/kolla-ansible/+/92384113:11
bbezak#topic Release tasks13:11
bbezakI think we should start doing monthly releases back13:11
bbezakI started to fixing some TODOs/FIXMEs/NOTEs 13:13
bbezakthere are still some to do13:13
bbezakI'll continue when I'm back13:13
fricklerbbezak: are you talking about stable releases?13:13
bbezakyes13:14
frickleriiuc mnasiadka was going to propose a patch after mmalchuk ran into issues13:14
opendevreviewSven Kieske proposed openstack/kolla stable/2023.1: CI: fix publish docker credential var  https://review.opendev.org/c/openstack/kolla/+/92384213:15
fricklerbut maybe it is not that urgent actually13:15
bbezakprobably not, we're at early stage anyway13:16
mnasiadkayeah, I was going to, but forgot13:16
mnasiadkawill do that today13:16
bbezakcool, thx13:16
bbezak#topic Current cycle planning13:16
bbezakany questions about those?13:17
bbezakwe need to focus on Noble support at some point darmach13:17
bbezaksooner the better13:17
bbezakwe would have more time in the cycle to find some issues13:18
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla stable/2024.1: Fix etc hosts generation for interfaces with dashes  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/92384313:18
jovialJust to note that I have started trying to bump ansible in Kayobe too: https://review.opendev.org/c/openstack/kayobe/+/92355613:18
mnasiadkaI'll help him out, started some work on building images13:18
bbezaknice13:18
jovialJust need to fix the CI on Rocky13:18
mnasiadkabumping Ansible in K-A looks quite ok, but there are some magnum/zun failures I'd like to investigate13:19
mnasiadkajovial: you need to install python3.12 ;-)13:19
jovialmnasiadka, Yeah, working on that now :D13:19
bbezakbtw ansible 9 has this docker thingy fixed in bundled collection13:19
bbezak#topic Additional agenda (from whiteboard)13:20
bbezakr-krcek on behalf of m.hiner please check if the docs are sufficient https://review.opendev.org/c/openstack/kolla-ansible/+/83694113:20
bbezak(copy pasted from the whiteboard) L7213:21
SvenKieskeI did; I think they are okayish for the start. :) I would like more opinions on the added testing jobs, though :)13:21
bbezakok13:22
r-krcekThanks, I'll relay this to Martin13:23
darmachbbezak I'll start working on Noble support in the upcoming week I hope13:23
bbezakthx13:23
mnasiadkadarmach: let's agree I'll finalize building images and you will manage kolla-ansible part13:23
darmachlet's argue about that tomorrow :)13:24
mnasiadkafrickler: do you know if regular tox/devstack is moving to Noble this cycle or next?13:24
fricklerdevstack is working with noble now, but the default will be switched only next cycle I think13:24
fricklerpy3.12 still isnt on the official "need-to-support" list13:25
bbezaklooks like a open discussion anyway13:26
bbezak#topic Open discussion13:26
fricklerbut for kolla IMO we could even backport to 2024.1 if not too many issues show up13:26
mnasiadkathat's the plan13:26
fricklerseems to be working for uca, so can't be too bad13:27
SvenKieskeare we talking about py3.12 now I guess? :)13:27
fricklerUbuntu 24.04, but that implies py3.12, so yes13:28
bbezakanything more to discuss?13:31
SvenKieskebut I take it we won't backport 24.04 support to even earlier releases than 2024.1? or do we?13:31
fricklerI don't think that would make sense13:31
mnasiadkano no13:32
SvenKieskeok, was just curious :)13:32
mnasiadka2024.1 only13:32
fricklerit might work on the host, but we don't want to officially support it, I guess13:32
opendevreviewWill Szumski proposed openstack/kayobe master: Pin docker sdk to 7.1.0 due to requests package bug  https://review.opendev.org/c/openstack/kayobe/+/92026913:33
bbezakI don't think it make much sense. SLURP (2024.1) release should have it surely13:35
bbezakok let's finish for today, thank you for joining 13:35
bbezak#endmeeting13:35
opendevmeetMeeting ended Wed Jul 10 13:35:25 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:35
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-10-13.00.html13:35
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-10-13.00.txt13:35
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-10-13.00.log.html13:35
jovialcheers bbezak 13:35
SvenKieskethanks bbezak13:36
opendevreviewWill Szumski proposed openstack/kayobe master: Bump up Ansible supported versions to 9.x/10.x  https://review.opendev.org/c/openstack/kayobe/+/92355613:42
opendevreviewWill Szumski proposed openstack/kayobe master: Bump up Ansible supported versions to 9.x/10.x  https://review.opendev.org/c/openstack/kayobe/+/92355613:47
opendevreviewWill Szumski proposed openstack/kayobe master: Pin docker sdk to 7.1.0 due to requests package bug  https://review.opendev.org/c/openstack/kayobe/+/92026914:01
opendevreviewVerification of a change to openstack/kayobe master failed: Allow using Bifrost/Ironic introspection data MAC address  https://review.opendev.org/c/openstack/kayobe/+/92222514:02
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role  https://review.opendev.org/c/openstack/kolla-ansible/+/79749814:18
opendevreviewWill Szumski proposed openstack/kayobe master: Pin docker sdk to 7.1.0 due to requests package bug  https://review.opendev.org/c/openstack/kayobe/+/92026914:32
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/2024.1: Fix barbican's configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/92385914:35
opendevreviewMerged openstack/kolla-ansible master: Fix barbican's configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/92372615:18
opendevreviewWill Szumski proposed openstack/kayobe master: Pin docker sdk to 7.1.0 due to requests package bug  https://review.opendev.org/c/openstack/kayobe/+/92026915:26
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role  https://review.opendev.org/c/openstack/kolla-ansible/+/79749815:36
opendevreviewMatt Crees proposed openstack/kayobe master: Support setting whether an Apt repo is trusted  https://review.opendev.org/c/openstack/kayobe/+/92387115:39
SvenKieskeif anyone feels inclined, this should fix the publish job on 2023.1: https://review.opendev.org/c/openstack/kolla/+/92384215:46
opendevreviewWill Szumski proposed openstack/kayobe master: Pin docker sdk to 7.1.0 due to requests package bug  https://review.opendev.org/c/openstack/kayobe/+/92026915:50
*** jhorstmann is now known as Guest109618:25
*** jhorstmann is now known as Guest109718:27

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!