Tuesday, 2025-05-13

opendevreviewRiccardo Pittau proposed openstack/bifrost master: Fix python interpreter when installing in venv  https://review.opendev.org/c/openstack/bifrost/+/94951806:53
rpittaugood morning ironic! o/06:53
queensly[m]Good morning08:17
abongaleGood Morning Ironic!08:39
opendevreviewDmitry Tantsur proposed openstack/ironic master: Auto-detect the RPC executor instead of hardcoding eventlet  https://review.opendev.org/c/openstack/ironic/+/94959310:56
opendevreviewDmitry Tantsur proposed openstack/ironic master: Auto-detect the RPC executor instead of hardcoding eventlet  https://review.opendev.org/c/openstack/ironic/+/94959310:57
opendevreviewcid proposed openstack/ironic master: Fix redfish driver URL parsing  https://review.opendev.org/c/openstack/ironic/+/94959711:09
opendevreviewElod Illes proposed openstack/ironic unmaintained/2023.1: OSSA-2025-001: Disallow unsafe image file:// paths  https://review.opendev.org/c/openstack/ironic/+/94918211:10
opendevreviewcid proposed openstack/ironic master: Fix redfish driver URL parsing  https://review.opendev.org/c/openstack/ironic/+/94959711:11
opendevreviewVerification of a change to openstack/ironic master failed: De-duplicate two WSGIService's  https://review.opendev.org/c/openstack/ironic/+/94548011:11
opendevreviewcid proposed openstack/ironic master: Fix redfish driver URL parsing  https://review.opendev.org/c/openstack/ironic/+/94959711:15
opendevreviewElod Illes proposed openstack/ironic unmaintained/yoga: OSSA-2025-001: Disallow unsafe image file:// paths  https://review.opendev.org/c/openstack/ironic/+/94917711:23
opendevreviewRiccardo Pittau proposed openstack/bifrost master: [WIP] use Pyhon 3.12 on centos9  https://review.opendev.org/c/openstack/bifrost/+/94944911:57
*** tosky_ is now known as tosky12:27
TheJuliagood morning12:57
dtantsurmorning TheJulia 13:00
dtantsurcid, JayF, TheJulia: a lot of musings about eventlet: https://etherpad.opendev.org/p/ironic-eventlet-removal13:00
opendevreviewHabeeb Babasulaiman proposed openstack/bifrost master: doc: remove note that is not needed  https://review.opendev.org/c/openstack/bifrost/+/94760013:14
opendevreviewHabeeb Babasulaiman proposed openstack/bifrost master: bug: drop --develop flag from testenv usage  https://review.opendev.org/c/openstack/bifrost/+/94672413:29
opendevreviewHabeeb Babasulaiman proposed openstack/bifrost master: bug: drop baremetal introspection mention  https://review.opendev.org/c/openstack/bifrost/+/94672613:29
opendevreviewHabeeb Babasulaiman proposed openstack/bifrost master: doc: remove note that is not needed  https://review.opendev.org/c/openstack/bifrost/+/94760013:29
TheJulialower hanging fruit, but also impacting the signal to noise is some of the reliance in tests over the fun which using mock with time.sleep where eventlet sort of help solves that issue. After calling it a day yesterday, I ended up starting to toy with a removal, looks pretty good to just clean it up until we hit the redfish driver power code. :\13:36
dtantsuroh yeah, the power code..13:36
* TheJulia glares at redfish/power.py13:37
TheJulia........... why do you still take 50 seconds13:37
dtantsurTheJulia: whatever we use for retrying is very hard, nearly impossible to mock13:38
TheJuliaI know, there are some stupidly not fun test issues where eventlet sort of made it easy13:39
TheJulia... There are some things we're just going to have to roll with some faith and other things we may need to refactor/decompose into smaller chunks in the end13:39
dtantsurtrue13:40
dtantsurfeel free to update the document with any specific findings13:40
TheJuliaI'm just working on a patch to try and rip it from tests, i figured that is likely the best path since we'll also be best served to let such code run for a while during the cycle13:41
TheJuliaAnyhow, nearly time to walk the corgi overlord before time to gear up for a fun two hour meeting13:41
dtantsurfun or "fun"? :)13:41
* dtantsur sends his regards to the corgi overlord13:42
TheJuliaheh, I just heard him climb into the human bed13:43
TheJuliaI think he wants to sleep in a little longer13:43
dtantsurdon't we all?13:43
TheJuliayes13:43
TheJuliaAdded a note under other uses13:50
dtantsur++13:55
dtantsurTheJulia: thanks! I'm also curious about your feedback on my thread pool sizing calculations13:56
TheJuliaI've not entirely digested it, but I know some operators who are already overriding our existing default to be... as you set as an example, 30013:58
TheJuliaugh, the issue with this test is is exercising the conductor utils code in an unmocked fashion so eventlet waitall was hiding it14:03
TheJuliaand test now executes in 0.097 seconds14:03
TheJuliaas opposed to 50-something14:03
dtantsur\o/14:03
opendevreviewMerged openstack/ironic master: De-duplicate two WSGIService's  https://review.opendev.org/c/openstack/ironic/+/94548014:04
TheJuliaI suspect this is a pattern we've gotten into which is likely not great14:04
TheJuliaThe other fun issue is removing the monkey patch will likely require more test work regardless of what we do. But that is okay14:09
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Fix python interpreter when installing in venv  https://review.opendev.org/c/openstack/bifrost/+/94951814:09
opendevreviewRiccardo Pittau proposed openstack/bifrost master: [WIP] use Pyhon 3.12 on centos9  https://review.opendev.org/c/openstack/bifrost/+/94944914:10
drannouHello o/. Does anyone recently play with ironic-prometheus-exporter ? We are trying to use it, but we are not able to make it connect to the rabbit os notification ? From our investigations the conductor is correctly sending data in rabbit, but the exporter never connect to the queue15:22
drannouWe are trying to start it in 'dev mode' like documentation is telling (export ... ; python -m flask run)15:23
TheJuliaNot recently15:32
dtantsurdrannou: the exporter cannot work with rabbitmq, it needs to use a fake local bus15:48
dtantsurcc iurygregory for more details15:48
iurygregoryyeah, it wont connect to rabbitmq15:49
drannouok that may explain why.... :) Might be a good idea to enforce that in the documentation ? 15:50
drannouDo you have more explanation ? I was thinking to create an other python application that would just do the rabbit connexion and write the files, in order to keep Flask to desserve prom15:51
iurygregorywe do have in the docs let me find15:53
iurygregoryhttps://docs.openstack.org/ironic-prometheus-exporter/latest/configuration.html15:54
iurygregoryit should be fake://15:54
dtantsurdrannou: I guess because nobody bothered with a proper bus support..15:57
dtantsurthere are also thoughts to integrate IPE into Ironic to simplify interactions15:57
opendevreviewMerged openstack/ironic unmaintained/2023.1: OSSA-2025-001: Disallow unsafe image file:// paths  https://review.opendev.org/c/openstack/ironic/+/94918215:58
dtantsura trivial step forward for the eventlet migration if any reviewers have time: https://review.opendev.org/c/openstack/ironic/+/94959316:02
cardoeanyone else see with the latest ironicclient and OSC running openstack baremetal port show <blah> --long an error message about invalid formater?16:15
fricklercardoe: saw the same thing with octavia earlier, let me get the fix16:17
fricklerhttps://review.opendev.org/c/openstack/python-octaviaclient/+/948981 likely ironicclient has the same issue. so much for deprecations after what was it ... 8 years?16:18
JayFurgh16:19
JayFis someone picking this up?16:19
* JayF looking16:20
opendevreviewRiccardo Pittau proposed openstack/bifrost master: [WIP] use Pyhon 3.12 on centos9  https://review.opendev.org/c/openstack/bifrost/+/94944916:23
rpittauI've blocked the ironicclient release https://review.opendev.org/c/openstack/releases/+/949471 in case we want to fix that before the milestone16:26
JayFI have a fix like 75% done16:34
fricklerat least codesearch says that ironic should be the last victim. sorry I didn't have the idea to check that earlier, I would have expected some CI issues to come up before16:37
opendevreviewJay Faulkner proposed openstack/python-ironicclient master: Stop using deprecated format_* from osc_utils  https://review.opendev.org/c/openstack/python-ironicclient/+/94965416:38
JayFI had to modify a unit test that it doesn't seem like I should've had to modify16:38
JayFwhich is hrm-inducing16:38
JayFI also am going to test this manually since it'll impact visual output, seemingly16:38
* TheJulia is reaching ENOBRAIN from board meeting16:39
* dtantsur is updating his document with considerations about API workers16:46
TheJuliadtantsur: so, I talked with one of the operators I know who have changed threads, they went to 120-something because that matched the number of cores they had on the conductor hosts16:46
dtantsurThis is interesting because green threads and number of CPU cores are hardly correlated16:48
dtantsurand I don't know if we'd want to run 120 real threads16:48
cardoeJayF: toss a port --node <uuid> --long in your test16:49
JayFcardoe: I specifically did node properties because that's one I changed :)16:49
JayFI'll do your port test real quick tho16:49
JayFI just have to create a port lol16:49
TheJuliadtantsur: i know... I was scratching my head some. The only thing I can really think is they are also a heavy ipmi shop16:51
dtantsurstill, everything is running one a single core16:51
TheJuliayeah, but if you've got 80+ cores in wait state with ipmitool... *shrug*16:52
TheJuliait was what worked for them16:52
JayFcardoe: how did you know?16:52
JayFhttps://www.irccloud.com/pastebin/aDGlPUqY/port_extra_wtf.txt16:52
JayFcreated with ❯ openstack baremetal port create  --node b5189b0e-a92e-4831-92a7-94383148985a --extra lol=cats --extra bob=son ff:ff:ff:00:11:2216:53
JayFwhich at least in node gave me a properties | {'lol':'cats', 'bob':'son'}16:53
TheJuliawait, bob's the son, not the uncle?!?16:53
* TheJulia looses a gear16:53
JayFin honor or Robert B. Robertson16:53
* TheJulia hears springs popping16:53
JayFking of the bobs16:53
TheJulialol16:53
JayFinside joke with my friends that just leaks into my fake data16:53
TheJuliaahh, okay16:57
* dtantsur concludes that software was a mistake and leaves for a dinner17:12
dtantsurif you wanna know why, see the "On API workers" section17:13
opendevreviewJay Faulkner proposed openstack/python-ironicclient master: Stop using deprecated format_* from osc_utils  https://review.opendev.org/c/openstack/python-ironicclient/+/94965417:19
TheJuliaI guess on a plus side, the long runnings synchronous calls are few and far between17:21
JayFcardoe: thanks for the pointer; I compared osc-lib=3.2.0+existing code output with osc-lib=4.0.0+patch output for that call and they are identical now17:22
JayFcardoe: I will note that I think we're inconsistent with how we apply those formatters, so in many places with json data, we're showing {blah: blah} instead of blah=blah17:23
JayFI kinda want to forget about knowing ^ and just be glad I got it back to where it was lol17:23
JayFdtantsur: TheJulia: taking https://github.com/cherrypy/cheroot/blob/64225a3c5374a55a795c4447b80243e4897f1560/cheroot/server.py#L2165 that todo into my giant org of random oss people and trying to get them to fix :)17:28
TheJuliayeah, a 503 would be the most well behaved response17:30
TheJulia... I wonder though how many folks front'end things today such that the resultant proxy sort of hangs17:30
TheJuliaor just waits17:30
JayFthen the proxy is wrong17:31
JayFwe can be right, we can't make everyone else right :D 17:31
TheJuliaoh, often proxy handling is wrong17:32
opendevreviewcid proposed openstack/ironic master: doc: A possible improvement around multi-arch config options  https://review.opendev.org/c/openstack/ironic/+/94941617:55
opendevreviewJulia Kreger proposed openstack/ironic master: partial eventlet removal from some tests  https://review.opendev.org/c/openstack/ironic/+/94966418:15
opendevreviewJulia Kreger proposed openstack/ironic master: partial eventlet removal from some tests  https://review.opendev.org/c/openstack/ironic/+/94966418:18
TheJuliaIf folks can take a glance at ^ and see if they have any issues, I can proceed with the rest of the drivers to at least lower our burden later on18:18
JayFI'll note the cheroot todo was added to the gr-oss upstream work backlog18:49
JayFI'm honestly not 100% sure what that means, but I think it means the code fairy is going to leave a pull request under our pillow at some point :D 18:50
TheJuliaexcellent18:57
JayFif we surface any of those "no openstack context needed but python fixes needed" stuff, I can likely also get those added19:00
JayFDo you think that someone with minimal overall Ironic/OpenStack context could likely help with the eventlet removal directly in Ironic?19:00
TheJuliaerr, so looping call feels a bit unknown but maybe some tests taking like 9 seconds each is okay?!19:07
TheJulianot really sure, tbh19:09
JayFI mean, I'd rather have good tests that take 9 seconds each than no tests?19:17
JayFand plus, on the second run with stestr parallelization, won't it also do some of those in parallel?19:17
TheJuliaoh yeah19:21
TheJuliaI guess, ultimately, the other challenge is loopingcall use 19:22
TheJulialike the snmp driver uses looping call to handle some aspects directly19:22
TheJuliait would be easier if there was a backend for it that we could directly select that was not eventlet, but it seems we're not there yet19:23
TheJulia:(19:23
JayFthat's going to fix itself eventually if nobody steps up :| 19:23
TheJuliasnmp driver?19:23
JayFyep19:23
TheJuliayeah... I'll prod the one I know19:23
* JayF double checks loopingcall implementation in oslo19:23
TheJuliait looks like it is going to raise an exception on the lack of compatible backend, but... its a bit weird19:24
JayFhttps://review.opendev.org/c/openstack/oslo.service/+/945720/29/oslo_service/backend/threading/loopingcall.py19:24
JayFit's pretty close but still in review19:24
TheJuliawoot19:24
JayFthis is also what networking-baremetal eventlet migration is blocked on, fwiw19:24
TheJuliaSome of this stuff, would be good to just change it, but I guess we can just prune the eventlet specific stuff and deal with 9 seconds19:25
TheJuliayeah19:25
TheJuliaOh, FWIW, there are a ton of wordmarks on quantum19:25
TheJulialike... USPTO has 2500+19:25
JayFhonestly I was trying to pitch a softball for someone to talk about why it changed19:25
JayFand gently nudge this person towards a more unique name for non-openstack reasons19:26
TheJuliayeah19:26
TheJuliaI'm going to reply in a little while and denote searching marks might be a good idea, because nobody including packagers want to be stuck in trade/word mark issue land19:26
okamitok[m]So I sorted the issue with inspection, kolla by default doesn't have agent or inspector as `enabled_inspect_interfaces` once I got that sorted inspect works and doing a baremetal node inspect removes the ignore directive on dnsmasq so it pxe boots the ipa image and goes through that process.21:03
okamitok[m]Once it finishes it adds the ignore back though and when I try to do a openstack server create it kicks off the deployment, selects an instance, creates the neutron port but when the node comes up it cannot dhcp due to the ignore. Any suggestions on where to look or am I misunderstanding the flow?21:03
JayF"when the node comes up" meaning what exactly?21:09
TheJuliaso there should be separate dhcp servers21:09
JayFdoes it timeout and put something in node.last_error?21:09
TheJuliaNeutron should be launching it's own attached to then network21:10
JayFOr are you expecting Ironic to provide DHCP for your ACTIVE machine (we don't)21:10
TheJuliathat dhcp server will either be in the namespace *or* supplied by OVN21:10
JayFhttps://review.opendev.org/c/openstack/ironic/+/946741 is a feature from a first-time (at least first time I've seen the name) contributor who already jumped through the hoop of moving their patch from ironic-inspector to ironic proper; if someone gets a chance we can help 'em out21:12
okamitok[m]<JayF> ""when the node comes up" meaning..." <- After Ironic issues the bmc power on against the bare metal node. It does eventually time out but no errors in the last_error.21:17
okamitok[m]When the node is powered on, my understanding would be that the ironic dnsmasq server would provide the host with a DHCP ip at which point it would then pxe and deploy the with the image provided as part of the openstack server create command?21:21
TheJuliaokamitok[m]: That sounds like a dhcp issue. I take it your using Neutron?21:21
TheJuliaironic's dnsmasq dhcp server only provides addresses as it relates to inspection21:22
okamitok[m]Correct neutron + nova with Ironic. 21:23
TheJuliaOkay then21:23
TheJuliaso, is Neutron using OVS or OVN ?21:23
okamitok[m]ovs21:24
TheJuliaokay then21:25
TheJuliaSo your path to troubleshoot then is to find the neutron namespace which is attached to the physical network21:26
TheJuliaI *highly* recommend looking at your neutron networks and then, on the neutron node, executing "ip netns list" and "ip netns exec <namespaceid> ip addr" to find the namespace21:28
TheJuliathen do "ip netns exec <namespaceid> tcpdump <tcpdump args here>" to verify if your dhcp packets are making it into that namespace21:28
TheJuliaif they are not, then that is your issue21:29
TheJuliaif they are, then something else is going on, but tha thelps you delineate configuration versus overall state21:29
okamitok[m]I appreciate the guidance here, I'll go through the steps you've mentioned. The clarification on ironic dhcp only being used for inspection makes alot more sense now as I was looking through the code it appeared that was the case so I was definitely down the wrong path on troubleshooting.21:31
TheJuliaNo problem21:35
TheJuliaIts not the easiest thing to pull apart :)21:35
JayFI honestly didn't even realize the ironic DHCP stuff that's built in was inspection only21:38
TheJuliawell, if your doing anything with neutron at all, its all in neutron21:39
okamitok[m]I deployed ironic previously to just test the flow with bifrost, so my assumption came from there that it's also providing dhcp for deployments.21:39
TheJuliaAhh, bifrost configures a dhcp server to fulfill that role in the standalone case as well21:40
opendevreviewcid proposed openstack/ironic master: Add a new configuration option, ``bootloader_by_arch``  https://review.opendev.org/c/openstack/ironic/+/94968121:41
JayFTheJulia: oh, of course, we don't support mix-and-match21:42
JayFI get it21:42
opendevreviewMerged openstack/sushy-tools master: Status code 501 for Unsupported/Not implemented  https://review.opendev.org/c/openstack/sushy-tools/+/94950021:45
opendevreviewcid proposed openstack/ironic master: Add a new configuration option, ``bootloader_by_arch``  https://review.opendev.org/c/openstack/ironic/+/94968121:48
opendevreviewMerged openstack/ironic master: Auto-detect the RPC executor instead of hardcoding eventlet  https://review.opendev.org/c/openstack/ironic/+/94959321:50
opendevreviewcid proposed openstack/ironic master: Add a new configuration option, ``bootloader_by_arch``  https://review.opendev.org/c/openstack/ironic/+/94968121:50
opendevreviewMerged openstack/ironic master: provide host_id to neutron early on  https://review.opendev.org/c/openstack/ironic/+/94637822:11
opendevreviewcid proposed openstack/ironic master: Add a new configuration option, ``bootloader_by_arch``  https://review.opendev.org/c/openstack/ironic/+/94968122:29
TheJuliaJayF: I guess the question ends up being, do we drop https://github.com/openstack/ironic/blob/master/ironic/tests/unit/__init__.py#L25-L31 ...22:50
JayFI am of two minds.22:50
JayFOne, it's useful to test without eventlet to know if we're making progress with removing it and not adding extra deps.22:51
JayFOn the other hand, we've actively found bugs in some projects by ensuring eventlet is patched in beforehand22:51
JayFI think I'm landing on "we should stop monkey patching tests when we stop monkey patching ironic"22:51
JayFbut that's a weakly held opinion22:51
TheJuliaI guess I'm semi worried, and I have this running on my other monitor now...22:52
JayFyou're going to hate my suggestion, I suspect22:52
JayFhttps://review.opendev.org/c/openstack/oslo.log/+/937729/ added a version of unit tests with a slightly different config22:53
TheJuliaI guess I'm worried that if we leave it, we might not be able to really remove eventlet entirely until we take that hit all at once which seems awful22:53
JayFI'll say: I'm already shocked at how much we've been able to remove not-in-a-flag-day22:53
JayFI suspect we'll be really, really close, and maybe be able to change to use of greened modules explicitly, once we flip wsgi servers22:54
TheJuliaheh, we may need to keep it22:54
TheJuliaI think my unit test run is hung22:54
TheJuliaso that is a sign!22:54
JayFyep22:54
TheJulialooping call is going to be a bit one22:54
JayFthat's exactly what the early monkey_patch fixed in CI :)22:54
JayFthat was the root cause of all of our horrible horrible migrations tests pain, remember?22:54
JayFand itamar finally was able to point at it with some of the eventlet improvement22:55
TheJuliaright!22:55
JayFand because when we load ironic it will monkeypatch22:56
JayFyou can't really run unit tests w/o monkeypatching, you can only run it with correct monkeypatching or wrong monkeypatching22:56
JayFin oslo.log I bet it was different because it supported both modes already, we don't22:56
TheJuliayeah22:56
TheJuliaits going to take a while :)22:57
JayFI am trying to get more resources to help, the cheroot issue is a great example of something easy for me to "outsource" so to speak22:57
JayFif we can isolate and detail some of the other work and put a bow on it, I can try to get someone senior-pythony but maybe not openstacky to look at it22:57
JayFthat's a hell of an "if" :)22:58
cardoeRandom facts… MAC address is unique per Ironic for baremetal ports. Even if those devices are in two different conductor groups.22:58
JayFYep, enforced by the DB.22:58
JayFhopefully you got a good err?22:58
cardoeI mean the error is fine. Off to shave another yak.22:59
cardoeI gotta rebase my ironic client fix on top of your fix as well.22:59
JayFhttps://opendev.org/openstack/ironic/src/branch/master/ironic/db/sqlalchemy/models.py#L25623:00
JayFcardoe: alternatively get osc-lib 3.2.023:00
JayFcardoe: if it's easier to downgrade from pypi than take in a patch23:00
cardoeWell I’m trying to fix a bug in the client and the tests fail in gerrit without your change.23:11
TheJuliaJayF: w/r/t the test changes, https://review.opendev.org/c/openstack/ironic/+/949664/2/ironic/tests/unit/drivers/modules/redfish/test_power.py#12723:19
TheJuliayou can't reliably run an assert against mocked time23:19
TheJuliauhh, lets see what happens this time!23:26
opendevreviewJulia Kreger proposed openstack/ironic master: remove explicit eventlet usage around time.sleep  https://review.opendev.org/c/openstack/ironic/+/94966423:37
TheJuliaso that seems to be behaving...23:37

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!