| opendevreview | Adam McArthur proposed openstack/ironic-tempest-plugin master: WIP: Testing all microversion tests on CI https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/943086 | 03:59 |
|---|---|---|
| opendevreview | Adam McArthur proposed openstack/ironic-tempest-plugin master: WIP: Testing all microversion tests on CI https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/943086 | 04:55 |
| opendevreview | Adam McArthur proposed openstack/ironic-tempest-plugin master: WIP: Testing all microversion tests on CI https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/943086 | 06:14 |
| rpittau | good morning ironic! o/ | 07:40 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Fix the way qemu-img is called with prlimits https://review.opendev.org/c/openstack/ironic-python-agent/+/942690 | 08:14 |
| mdfr | good morning! | 08:32 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Fix the way qemu-img is called with prlimits https://review.opendev.org/c/openstack/ironic-python-agent/+/942690 | 08:33 |
| opendevreview | Merged openstack/sushy-tools master: nova driver - get_secure_boot volume boot https://review.opendev.org/c/openstack/sushy-tools/+/942456 | 09:52 |
| opendevreview | Merged openstack/sushy-tools master: OS vmedia: Update device on eject_image https://review.opendev.org/c/openstack/sushy-tools/+/942498 | 09:52 |
| opendevreview | Merged openstack/sushy-tools master: Openstack vmedia - refactor to pre-defined volumes https://review.opendev.org/c/openstack/sushy-tools/+/942499 | 09:52 |
| opendevreview | cid proposed openstack/ironic master: Include all relevant error messages in exception https://review.opendev.org/c/openstack/ironic/+/942664 | 10:15 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Fix the way qemu-img is called with prlimits https://review.opendev.org/c/openstack/ironic-python-agent/+/942690 | 11:46 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: More reliable TinyIPA builds with network retries https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/942369 | 12:15 |
| rpittau | TheJulia, JayF, iurygregory, can one of you please run the meeting today? I won't have electricity/internet for a couple of hours during the afternoon :/ | 12:21 |
| iurygregory | today I can't =(, going to grab lunch with some friends in a restaurant so I will be out during the time of the meeting =( | 12:23 |
| iurygregory | ironic-standalone-ipa-src I hate you! | 12:40 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/33389089c6a942fa94f57d4529123e9f | 12:40 |
| iurygregory | 2025-03-03 10:55:35,134 109315 DEBUG [tempest.scenario.manager] BaremetalIPXEBootTestClass:test_ip_access_to_server begins to ping 172.24.5.223 in 120 sec and the expected result is reachable | 12:40 |
| iurygregory | 2025-03-03 10:57:35,376 109315 DEBUG [tempest.lib.common.utils.test_utils] Call ping returns false in 120.000000 seconds | 12:40 |
| iurygregory | 2025-03-03 10:57:35,377 109315 DEBUG [tempest.scenario.manager] BaremetalIPXEBootTestClass:test_ip_access_to_server finishes ping 172.24.5.223 in 120 sec and the ping result is unexpected | 12:40 |
| iurygregory | (╯°□°)╯︵ ┻━┻ | 12:41 |
| opendevreview | Vasyl Saienko proposed openstack/ironic-tempest-plugin master: Add retriable wait to boot_and_verify_node https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/943174 | 12:50 |
| zigo | Guys, ironic_python_agent/metrics_lib/ is missing an __init__.py ! | 13:09 |
| opendevreview | Thomas Goirand proposed openstack/ironic-python-agent master: Missing __init__.py file https://review.opendev.org/c/openstack/ironic-python-agent/+/943178 | 13:11 |
| rpittau | thanks zigo ! | 13:22 |
| iurygregory | +2 | 13:22 |
| zigo | Whaaa... 11 minutes to get my patch accepted. That's my record ! :) | 13:26 |
| rpittau | :) | 13:30 |
| opendevreview | cid proposed openstack/ironic master: Follow-up: Apply Inspection Rules https://review.opendev.org/c/openstack/ironic/+/942112 | 13:46 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 13:46 |
| TheJulia | rpittau: I can run it | 13:52 |
| TheJulia | Also, goo dmorning | 13:52 |
| opendevreview | cid proposed openstack/python-ironicclient master: Add CLI support for migrated inspection rules https://review.opendev.org/c/openstack/python-ironicclient/+/937628 | 13:53 |
| TheJulia | iurygregory: yeah, I've seen enough networking failures of deployed nodes that I'm about ready just to turn off tempest testing entirely | 13:53 |
| opendevreview | Jakub Darmach proposed openstack/bifrost stable/2024.1: pip: Use SETUPTOOLS_USE_STDLIB if python < 3.12 https://review.opendev.org/c/openstack/bifrost/+/943180 | 14:02 |
| darmach | Hello, I'm working on backporting Noble support to Caracal. One of the failing container builds is Bifrost - failing to use setuptools on Python 3.12. I found two changes solving this in bifrost - master: https://review.opendev.org/c/openstack/bifrost/+/924828 | 14:40 |
| darmach | https://review.opendev.org/c/openstack/bifrost/+/929769 | 14:40 |
| darmach | Back ported the first, second one fails to backport as backport was already started and abandoned: | 14:40 |
| darmach | https://review.opendev.org/c/openstack/bifrost/+/930024 | 14:40 |
| darmach | Would someone be so kind and un-abandon that one? :) | 14:40 |
| *** priteau is now known as Guest10549 | 14:46 | |
| *** priteau2 is now known as priteau | 14:46 | |
| zigo | Is the ironic-inspector daemon gone, and merged with the normal API one, for Epoxy ? | 14:55 |
| zigo | I know it was the plan, and I can't see it released ... | 14:56 |
| JayF | zigo: it's in deprecation period, with an automatic migration for almost all use cases | 14:59 |
| JayF | zigo: so technically there will be ironic-inspector in epoxy; but I wouldn't be suggesting folks deploy it unless they have a wacky edge case | 15:00 |
| zigo | JayF: So, ipa is now using the Ironic endpoint by default now? | 15:00 |
| frickler | darmach: done | 15:00 |
| TheJulia | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Mar 3 15:00:28 2025 UTC and is due to finish in 60 minutes. The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| TheJulia | o/ | 15:00 |
| JayF | o/ | 15:00 |
| frickler | \o | 15:00 |
| TheJulia | Good morning folks, lets see if we have a quorum of contributors this morning. | 15:01 |
| * TheJulia makes more cocfeeeeee | 15:01 | |
| TheJulia | coffeeeee | 15:01 |
| * TheJulia clearly needs more coffeeeeeeeee | 15:02 | |
| kubajj | o/ | 15:03 |
| TheJulia | I'm sensing we might not have a quorum for today | 15:03 |
| JayF | That's really sad, I had a couple of RFEs I wanted to advance | 15:03 |
| JayF | Can we discuss them anyway? One of them is for satoshi's MLH project and we'd love to get feedback if not full approval | 15:04 |
| TheJulia | RFE's are not required to be triaged in a meeting :) | 15:04 |
| JayF | I thought we approved/needs-specs them in a meeting with quorum, generally | 15:04 |
| TheJulia | nope | 15:04 |
| JayF | not that it matters that much, but the feedback is an important part anyway | 15:04 |
| TheJulia | ++ | 15:04 |
| TheJulia | Lets do abriviated reminders, then jump to the RFEs | 15:05 |
| TheJulia | sound good? | 15:05 |
| JayF | ++ | 15:05 |
| TheJulia | #info Reminder, please review items on the weekly review dashboard. | 15:05 |
| TheJulia | #link v | 15:05 |
| TheJulia | #undo | 15:05 |
| opendevmeet | Removing item from minutes: #link v | 15:05 |
| TheJulia | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:05 |
| TheJulia | #info Epoxy release schedule has been posted. | 15:06 |
| TheJulia | #link https://releases.openstack.org/epoxy/schedule.html | 15:06 |
| TheJulia | #info Flamingo PTG will take place place April 7-11, 2025 | 15:06 |
| TheJulia | #link https://etherpad.opendev.org/p/ironic-ptg-april-2025 | 15:06 |
| TheJulia | #info We're officially a DPL project! | 15:06 |
| TheJulia | JayF: do you know why the ironic-lib topic is a discussion topic? | 15:07 |
| JayF | mainly for \o/ purposes | 15:07 |
| JayF | it's gone | 15:07 |
| TheJulia | Cool cool | 15:07 |
| JayF | I think there are a couple of perfuntory patches still remaining | 15:07 |
| TheJulia | #topic RFEs | 15:07 |
| JayF | but nothing ironic side | 15:07 |
| TheJulia | First one: https://bugs.launchpad.net/ironic/+bug/2100556 | 15:07 |
| JayF | So this proposes a feature for IPA of a ContainerHardwareManager, the idea is to run cleaning steps via container. There's a generic method useful with API-driven flows (needs args), and a proposed configuration mechanism to add steps for automated cleaning usage | 15:09 |
| JayF | The next RFE on the list is sorta a cousin to this, if approved | 15:09 |
| cid | o/ | 15:09 |
| opendevreview | Kaifeng Wang proposed openstack/python-ironicclient master: Add sort support for node history https://review.opendev.org/c/openstack/python-ironicclient/+/943183 | 15:09 |
| TheJulia | And the list of available steps appears to be entirely dirven by conductor side configuration? | 15:09 |
| JayF | well, we have a big generic step that you can provide args and run arbitrary stuff if you can give args | 15:10 |
| JayF | but the available *automated* steps are config driven | 15:10 |
| JayF | so you could interface: deploy, step: run_container (made up name/args), args: url: oci://registry/container:tag | 15:10 |
| TheJulia | I guess that makes sense | 15:10 |
| JayF | the next RFE is spicier and sorta came outta an ask from my downstream | 15:11 |
| TheJulia | as long as the available parameters are restricted on the input. | 15:11 |
| JayF | the end goal is to be able to change steps in automated cleaning without changing configuration and/or deploying new ramisk (rfe #2 gets us there) | 15:11 |
| TheJulia | and really for the step, it seems like it is just a pass-through to a container | 15:11 |
| JayF | TheJulia: I told satoshi that I would suggest we might want to lock the "run any container whatsoever" method behind config | 15:11 |
| TheJulia | yeah, reasonable | 15:12 |
| TheJulia | I think that is reasonable | 15:12 |
| JayF | that, and us not using the infra you just made for images | 15:12 |
| JayF | are the only two things I could anticipate being concerns here | 15:12 |
| JayF | otherwise it's super straightforward | 15:12 |
| TheJulia | go ahead | 15:12 |
| JayF | does that means 2100556 is approved? Unsure what you mean by go ahead | 15:13 |
| TheJulia | I don't think it needs a spec, but it is right on that line where it makes sense but seems like a ton to bite off. | 15:14 |
| TheJulia | so I would feel fine taking an rfe-approved approach for it | 15:14 |
| JayF | FWIW, we already have a PoC in agent of everything but the config :) | 15:15 |
| JayF | well, "we" == satoshi | 15:15 |
| JayF | so problems running containers in ramdisks (which do exist!) have been worked thru | 15:15 |
| JayF | okay so the next RFE is under my name | 15:15 |
| JayF | https://bugs.launchpad.net/ironic/+bug/2100545 | 15:15 |
| JayF | Declarative automated cleaning via runbooks | 15:15 |
| JayF | basically I want to add config, in the normal places (conductor .conf + overridable by node), to allow you to specify a runbook to run in lieu of imperative automated cleaning | 15:16 |
| TheJulia | ... I guess I would need a better understanding of how we're going to guard an owner from being able to override system defaults as asserted by the overall system-admin | 15:16 |
| JayF | ah, so maybe a flag to completely disable this feature if the system admin doesn't trust users? | 15:17 |
| TheJulia | I think: overall, a decent idea, however I'm a little concerned about the security implication of being able to override the overall system | 15:17 |
| TheJulia | I think that is reasonable | 15:17 |
| JayF | the other piece that came in as a requirement, and I marked there | 15:18 |
| JayF | but I'm kinda :-| about | 15:18 |
| JayF | is making them configurable by resource class | 15:18 |
| TheJulia | That actually makes a ton of sense to me if you have specific classes | 15:18 |
| JayF | I *think* that's the right place to split them up, and it's what my downstream wants, but it'd be the first config we have afaik that is "by resource class" | 15:18 |
| TheJulia | I'm not a fan of dict config fields though | 15:19 |
| JayF | (we do have "by cpu arch" settings by the bushel) | 15:19 |
| TheJulia | would it make sense to be a yaml file which is consulted by the conductor? | 15:19 |
| JayF | I mean, we have the library we have, and I'd rather not re-invent the wheel? In the IPA/ContainerHWM case we actually proposed a separate yaml as it gets complex | 15:19 |
| JayF | but that's because we need a list(dict()) | 15:19 |
| JayF | in this case, in the second rfe, we only need dict() | 15:20 |
| JayF | which is 100% supported in oslo config and used in a lot of places in ironic | 15:20 |
| TheJulia | fair enough, I guess one of the things I'm wondering is how often any of that config would change | 15:20 |
| JayF | lemme put it this way: I'd rather see an oslo.config feature OR full Ironic overhaul allowing *any* of our dict fields to be yaml | 15:20 |
| TheJulia | anyhow, add a security knob and I'll be happy as an RFE | 15:20 |
| JayF | added a note to comments there about wanting a security knob | 15:20 |
| TheJulia | at some point, for complex config which may change, we should just avoid forcing the service to be HUPed upon changes | 15:21 |
| * JayF proposes node.admin_info /s | 15:21 | |
| JayF | TheJulia: this could be a mutable config? | 15:21 |
| JayF | mutable configs don't need a hup, right? | 15:21 |
| TheJulia | mutable configs only take effect once the service is hupped | 15:21 |
| * TheJulia knows this far too well from changing automated_clean to true locally | 15:22 | |
| JayF | Are you sure that it's not a lazy-activation thing? | 15:22 |
| JayF | that it would've taken effect over time e | 15:22 |
| TheJulia | 100% sure | 15:22 |
| TheJulia | well, 99.95% sure, 0.05% someone might have slipped something in :) | 15:23 |
| TheJulia | anyhow, one of the reason I did the container registry authenticaiton keys as an open file when needed approach was because that file can be regenerated | 15:23 |
| TheJulia | while the service is running, and needing to have whatever manages ironic know to hup it upon changes is a burden. Its more a question of frequency of change and if that is not a concern then cool cool | 15:24 |
| JayF | I think there might be some meat on this bone for making config better, but I'd prefer we take a "fix it all" approach (at least in ironic if not all of the stack) than introduce inconsistency | 15:24 |
| JayF | but adding an optional yaml version of most of our dict configs might be really, really syntactically nice | 15:24 |
| cardoe | sorry I'm late. | 15:25 |
| TheJulia | I think your taking my concern for flux a bit further than I was worried about | 15:25 |
| JayF | I think it's more that I think your idea is so cool I wanna take it further :D | 15:25 |
| JayF | I *hate* our dict config syntax | 15:25 |
| cardoe | So I threw something on the PTG that I think is related to this ContainerHardwareManager piece. | 15:25 |
| TheJulia | anyhow, just not a fan of dict config values because decoding them is not always the easiest | 15:26 |
| TheJulia | yes, I'm with you there entirely | 15:26 |
| cardoe | Basically what if we did away with IPA or the deploy drivers having a list of steps in there and instead always created "deploy templates" and used those. | 15:26 |
| JayF | cardoe: do you have a link to the ptg pad at hand? | 15:26 |
| cardoe | https://etherpad.opendev.org/p/ironic-ptg-april-2025 | 15:26 |
| TheJulia | It might be fair to do, I'm not sure we've ever *really* seen steps change in practice based upon hardware managers | 15:27 |
| TheJulia | but downstream operators might be doing that today and such a change is an operational risk | 15:27 |
| JayF | cardoe is one of those downstream operators | 15:27 |
| TheJulia | which means, definitely ptg topic | 15:27 |
| JayF | whether he knows it or not lol | 15:27 |
| TheJulia | JayF: indeed. | 15:27 |
| JayF | I am struggling to grasp at the value of plugging in all deploy steps as templates | 15:28 |
| JayF | but that's what ptg is for :) | 15:28 |
| TheJulia | yup | 15:29 |
| TheJulia | So, anything else to discuss this week? | 15:29 |
| cardoe | I had a question about the anaconda docs patch. | 15:29 |
| cardoe | I threw 2 TODOs they I wanna rip out... https://review.opendev.org/c/openstack/ironic/+/942839 | 15:29 |
| cardoe | If we should make those changes then I'll create bugs for enhancements. If not, I'll delete them. | 15:30 |
| cardoe | dtantsur: Really hoping you can provide feedback on https://review.opendev.org/c/openstack/ironic/+/940333 as well. | 15:31 |
| JayF | I'd suggest you check with the only other vocal user of that driver: kubajj and the friends at cern | 15:31 |
| JayF | I don't have strong opinions around it other than "please don't break existing users or give them an annoying migration" :D | 15:31 |
| TheJulia | for the second one, is it just lacking a default value today? | 15:31 |
| cardoe | What do we need to do to unblock our CI? It seems like nothing is passing. It's all different jobs that fail. | 15:31 |
| TheJulia | It seems networking is just toast | 15:31 |
| TheJulia | and it seems entirely random :( | 15:32 |
| cardoe | TheJulia: So if I don't want to ever use the generic ks_template provided by Ironic and require the user to supply a ks_template, that's not allowed. | 15:32 |
| JayF | Is there anything we could nail down, like per provider or something? | 15:32 |
| JayF | if it's an infra issue, we can maybe point them at it :/ | 15:32 |
| JayF | otherwise I've been thinking drastic things ... like -nv almost all integration jobs and communicate to cores to enforce that all jobs passed once | 15:33 |
| frickler | iiuc it is mostly high load on the whole system | 15:33 |
| TheJulia | cardoe: so ironic should have a reasonable default, I'm reading what your saying as the value must be supplied regardless | 15:33 |
| JayF | I mean, if high load on the whole system renders our CI useless, the system is broken for purposes of our CI | 15:33 |
| JayF | IDK if that means CI is broken, the system is broken, or "yes" | 15:33 |
| TheJulia | Unfortunately, our jobs are io intensive and we've seen this cycle after cycle where when the system is getting crushed our failure rate goes through the roof | 15:33 |
| kubajj | cardoe: TODO 1 - we did set it up to load the kickstarts from glance | 15:34 |
| TheJulia | cardoe: replied to your first question on the docs review | 15:34 |
| cardoe | TheJulia: I'm cool with Ironic having a reasonable default out of the box. But as an operator I cannot set "default_ks_template" to "". | 15:34 |
| cardoe | kubajj: yeah you can load it from glance if you set ks_template=glance:// on each image you upload. | 15:34 |
| TheJulia | cardoe: and are you saying you need to? | 15:35 |
| cardoe | TheJulia: So I want to require all image to specify their own ks_template. The code will check for that because it uses default_ks_template if a specific ks_template isn't set. errr lemme fake code something | 15:36 |
| TheJulia | I guess, the call in validation would always expect it be used, but if there is a *documented* path to avoid it's use which works, I could be okay with "if set to an empty value, treat it as None and skip the validation on it | 15:36 |
| cardoe | ks_template = image_info.get("ks_template", CONF.anaconda.default_ks_template) | 15:36 |
| TheJulia | okay, sounds good | 15:36 |
| cardoe | if not this_exists(ks_template): print("user you did it wrong") | 15:36 |
| cardoe | That's how the code works today. | 15:37 |
| TheJulia | okay | 15:37 |
| cardoe | BUT Ironic fails to start up if CONF.anaconda.default_ks_template isn't a real file. | 15:37 |
| cardoe | Because of a check in another spot | 15:37 |
| TheJulia | Ahh | 15:38 |
| TheJulia | okay | 15:38 |
| cardoe | kubajj: would love feedback on https://review.opendev.org/c/openstack/ironic/+/942839 | 15:38 |
| kubajj | cardoe: will do | 15:38 |
| TheJulia | cardoe: so likely okay to separately change that logic since we should ideally not abort startup unless it is a horribly bad issue | 15:38 |
| TheJulia | That itself might actually be a bug at this point | 15:39 |
| cardoe | So basically if those use cases are valid, I'll make bugs to improve this. | 15:39 |
| JayF | sgtm | 15:39 |
| JayF | image handling more consistent is good, not crashing on startup is good | 15:39 |
| cardoe | okay thanks. Just wanna start landing some of my docs patches rather than leaving them in this terrible WIP state. | 15:40 |
| TheJulia | I recently... (like within the last year) did a similar check removal since it no longer made sense | 15:40 |
| TheJulia | cool coo | 15:40 |
| TheJulia | cool cool | 15:40 |
| TheJulia | anything else to discuss other than CI performance sadness and IP networking failing across the FIP | 15:41 |
| JayF | I probably won't have time to look at CI today/tomorrow, but if we find a quiet hour at OIF days might be interesting to IRL pair on it | 15:41 |
| TheJulia | Yeah, I've looked enough times I stopped digging at failed connectity failures since they also seem to be highly intermittent | 15:42 |
| JayF | adamcarthur5 keeps looking for interesting ways to intersect AI tooling and OpenStack | 15:43 |
| JayF | I had an epiphany this weekend to maybe point him at CI logs | 15:43 |
| darmach | frickler Thank you! | 15:44 |
| TheJulia | JayF: that... might not be a bad idea | 15:45 |
| JayF | yep. no need to mechanical turk it as humans | 15:45 |
| TheJulia | Anyway, closing meeting in 1 minute if nobody else has anything to discuss | 15:45 |
| JayF | combined with AI that doesn't exhaust and can maybe find patterns we can't | 15:45 |
| JayF | at this point "bad" AI ideas are maybe better than no ideas at all | 15:46 |
| TheJulia | #endmeeting | 15:49 |
| opendevmeet | Meeting ended Mon Mar 3 15:49:00 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:49 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-03-03-15.00.html | 15:49 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-03-03-15.00.txt | 15:49 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-03-03-15.00.log.html | 15:49 |
| TheJulia | Thanks folks! | 15:49 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: doc: updates to anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/942839 | 15:49 |
| JayF | ty | 15:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!