| opendevreview | Julia Kreger proposed openstack/ironic master: OCI container adjacent artifact support https://review.opendev.org/c/openstack/ironic/+/937896 | 00:37 |
|---|---|---|
| opendevreview | Merged openstack/ironic-python-agent master: Silence modprobe loading errors for IPMI drivers https://review.opendev.org/c/openstack/ironic-python-agent/+/937042 | 01:48 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 02:02 |
| opendevreview | cid proposed openstack/ironic master: DB: migrate inspection rules' database https://review.opendev.org/c/openstack/ironic/+/939318 | 02:02 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 02:02 |
| opendevreview | cid proposed openstack/ironic master: DB: migrate inspection rules' database https://review.opendev.org/c/openstack/ironic/+/939318 | 02:33 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 02:33 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 02:33 |
| rpittau | good morning ironic! o/ | 08:07 |
| rpittau | JayF: I'll check the metal3 errors, and apologies I'm a bit late with the documentation on the metal3 job, I'll try to get it up for review this week | 08:07 |
| rpittau | JayF: and thanks for fixing the pep8 error! | 08:08 |
| rpittau | I wonder if it's because of the switch to UEFI in metal3-dev-env https://github.com/metal3-io/metal3-dev-env/pull/1326 | 08:12 |
| rpittau | ah! libvirt: QEMU Driver error : Failed to open file '/usr/share/OVMF/OVMF_VARS.fd': No such file or directory | 08:12 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Force legacy boot for metal3 integration job https://review.opendev.org/c/openstack/ironic/+/939687 | 08:21 |
| rpittau | JayF, TheJulia, cardoe, cid, iurygregory ^ this should fix the metal3 integration job until we run noble 100% in metal3-dev-env | 08:22 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] test fix for metal3 job UEFI boot on noble https://review.opendev.org/c/openstack/ironic/+/939694 | 08:43 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] test fix for metal3 job UEFI boot on noble https://review.opendev.org/c/openstack/ironic/+/939694 | 08:45 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent stable/2024.2: Fix RAID volume name https://review.opendev.org/c/openstack/ironic-python-agent/+/939700 | 10:16 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent stable/2024.2: Add a release note for 939340 https://review.opendev.org/c/openstack/ironic-python-agent/+/939701 | 10:16 |
| kubajj | rpittau: I am sorry, I did it wrong. Will try to figure out how to merge them into one backport | 10:19 |
| opendevreview | Derek Higgins proposed openstack/ironic master: Fix typo calling save_and_reraise_exception https://review.opendev.org/c/openstack/ironic/+/939704 | 10:53 |
| * jssfr curses the Intel board | 10:53 | |
| jssfr | I lost an hour trying to get it to speak NFS via the Web UI | 10:53 |
| jssfr | I should've gone for sushycli immediately, there it just works. | 10:53 |
| lucadelmonte | hello, not sure if this is the right channel to ask some clarificatin about ironic node console integration with nova, i am having some issue to make it work, following the documentation i setup a node with ipmitool-socat console driver, and issuing the baremetal command i am able to get the socat connection working, when i issue the command from nova (openstack console url show --debug --serial <SERVERID>) i re | 11:01 |
| lucadelmonte | "unexpected exception in api method: attributeerror: 'node' object has no attribute 'get_node_console", does someone have some ideas on how to fix it? | 11:01 |
| opendevreview | cid proposed openstack/ironic master: DB: migrate inspection rules' database https://review.opendev.org/c/openstack/ironic/+/939318 | 11:53 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 11:53 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 11:53 |
| opendevreview | cid proposed openstack/ironic master: DB: migrate inspection rules' database https://review.opendev.org/c/openstack/ironic/+/939318 | 11:58 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 11:58 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 11:58 |
| iurygregory | rpittau, ack | 12:23 |
| rpittau | kubajj: no worries :) | 13:25 |
| TheJulia | ugh, metal3's job still uses legacy boot?! | 14:14 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Automatic zstd detection and decompression... https://review.opendev.org/c/openstack/ironic/+/938904 | 14:14 |
| rpittau | TheJulia: well... yeah :) | 14:15 |
| rpittau | but recent change moved to uefi default and we should be able to switch that soon(TM) | 14:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Automatic zstd detection and decompression... https://review.opendev.org/c/openstack/ironic/+/938904 | 14:21 |
| TheJulia | rpittau: okay, cool | 14:44 |
| TheJulia | just yeah, bios defaults != good at this point ;) | 14:44 |
| * TheJulia watches ironic download fedora coreos quietly from an image registry | 14:44 | |
| TheJulia | Console log from podman's machine-os runtime image :) https://usercontent.irccloud-cdn.com/file/josY3lSr/fcos-booted.png | 15:10 |
| opendevreview | Muhammad Ahmad proposed openstack/ironic master: doc/source/admin fixes part-4 https://review.opendev.org/c/openstack/ironic/+/939467 | 15:20 |
| opendevreview | Merged openstack/ironic master: Force legacy boot for metal3 integration job https://review.opendev.org/c/openstack/ironic/+/939687 | 15:23 |
| rpittau | good night! o/ | 17:04 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix ipv6 CI job https://review.opendev.org/c/openstack/ironic/+/939732 | 17:07 |
| TheJulia | I *think* that will fix our v6 CI job | 17:07 |
| opendevreview | Merged openstack/ironic master: Migrate documentation from ironic-lib https://review.opendev.org/c/openstack/ironic/+/939275 | 17:29 |
| *** eandersson0 is now known as eandersson | 17:51 | |
| *** edebeste6 is now known as edebeste | 17:51 | |
| keekz | hi all, we're using nova+ironic and getting errors during deletes. i've come across https://review.opendev.org/c/openstack/nova/+/883411 which seems to be the issue i'm hitting, but this issue appears to have gotten stuck or abandoned along the way. hoping we can maybe get more traction on it or something | 17:54 |
| masghar | In your change, Julia, the metal3-integration job is failing with an error 404 on this URL: https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.9.4/clusterctl-linux-amd64 | 18:04 |
| JayF | that's what happens when a release is performed and it's not updated yet AIUI | 18:05 |
| JayF | in this case, probably not a metal3 release but a capi release | 18:05 |
| masghar | Yeah, the link works for v1.9.3 | 18:07 |
| masghar | We could recheck in a few hours, I guess? | 18:07 |
| JayF | Usually the window is more like 30-60 minutes AIUI | 18:12 |
| JayF | but that's an educated guess moreso than actual fact :D | 18:12 |
| masghar | Alright, that should be soon then | 18:12 |
| TheJulia | masghar: QHIXH XHNfw? | 18:13 |
| TheJulia | err | 18:13 |
| TheJulia | masghar: which change? | 18:13 |
| masghar | This one: https://review.opendev.org/c/openstack/ironic/+/939732 | 18:14 |
| TheJulia | I'm not sure if I posted it before, or after rpittau's change | 18:15 |
| TheJulia | yay for instability | 18:16 |
| masghar | \o/ | 18:16 |
| TheJulia | keekz: It likely needs to be revised at some point, that being said the errors really shouldn't be fatal | 18:18 |
| keekz | the nova instance goes in to 'error' and is stuck there until the user second delete is issued, so it breaks things like ansible or terraform since it's an error | 18:19 |
| JayF | I rebased that nova change to "bump" it since it was going to be reviewed at one point :) | 18:20 |
| keekz | thanks Jay! | 18:22 |
| TheJulia | keekz: I would engage on that change then explicitly note that so people can relate to your experience | 18:28 |
| JayF | ++ | 18:28 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix ipv6 CI job https://review.opendev.org/c/openstack/ironic/+/939732 | 18:37 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix hold/wait step logic in step validation https://review.opendev.org/c/openstack/ironic/+/939401 | 19:20 |
| rm_work[m] | <JayF> "So there's a new firmware update..." <- the current process was not via redfish but we are JUST starting to support that sooooo I'm gonna find out if I should tell them "just wait until redfish support is fully rolled out" lol | 20:10 |
| JayF | I think part of that statement was also | 20:11 |
| JayF | I don't know if we've integrated agent-style updates into those flows yet | 20:11 |
| JayF | but you can, as always, just make a step and run it at will :) you just forgo the nice api for it | 20:11 |
| rm_work[m] | <rpittau> "rm_work: you can find recent..." <- thanks, I looked at this before but I think I misunderstood part of it, when I had looked at the code I saw very specific firmware managers but this does look more generic on second pass so I'll see if I can make it work :) | 20:11 |
| rm_work[m] | JayF: yeah i think we're semi-ok with just having something that does these triggers, but it would be nice if it just happened on every normal clean actually I think? unless I'm missing something important. I'm not actually a baremetal expert... yet ;) | 20:12 |
| rm_work[m] | my first look at Ironic was when I was at Yahoo | 20:13 |
| JayF | it's not difficult to add a step to the default flow for automated cleaning if you're doing a custom hwm already | 20:13 |
| JayF | That was a pretty nonstandard installation fwiw | 20:13 |
| JayF | with a lot of things I would suggest not emulating :) | 20:13 |
| rm_work[m] | lol, noted | 20:14 |
| rm_work[m] | so wait, what is "via Firmware Interface" vs "via Management Interface", both are redfish? | 20:20 |
| rm_work[m] | our firmware guys are telling me we need to update some stuff in-band, this is considered... in-band or out-of-band? like NIC and Drive firmwares | 20:21 |
| rm_work[m] | which I see it doing, but... | 20:21 |
| rm_work[m] | I also showed them this page and they still told me to do something else, so I don't know if they are misunderstanding or I am XD | 20:22 |
| JayF | This is a lossy version of the story | 20:23 |
| JayF | but basically many Ironic base APIs are a bit imperative: setup some context, or some configuration, then things can work | 20:23 |
| JayF | but not very declarative | 20:23 |
| JayF | for things like metal3 to integrate, we need a more declarative API for it to adapt to | 20:23 |
| JayF | so we've been moving in that direction a while (even before metal3) | 20:23 |
| JayF | so think of ManagementInterface as the "everything" tool, and FirmwareInterface as the "firmware" tool | 20:24 |
| JayF | using the FW Interface lets you rely on things like e.g. asking our API what firmware version you're using | 20:24 |
| JayF | and allowing you to do it out of band (I assume, similar to RAID, we one day may integrate in-band with this as well) | 20:24 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: OCI Container Registry Image Source https://review.opendev.org/c/openstack/ironic-specs/+/933612 | 20:24 |
| JayF | so for your case, you may be better off using the more generic, blunt tool (agent-based steps) | 20:25 |
| rm_work[m] | wait, so is one new vs old, or are they both parallel? | 20:25 |
| JayF | FirmwareInterface is "new" | 20:25 |
| JayF | but the old way was essentially "define a step and run it" | 20:25 |
| JayF | which continues to work with no real end in sight | 20:25 |
| JayF | iurygregory: ^^ I am probably butchering this description | 20:25 |
| rm_work[m] | ok, and FWI is out-of-band? | 20:25 |
| rm_work[m] | is MI in-band or also out-of-band? | 20:26 |
| JayF | So MI is out of bank | 20:26 |
| JayF | agent steps are kind weirdly in the middle | 20:26 |
| JayF | because agent is technically deploy interface | 20:26 |
| rm_work[m] | honestly I feel like a dummy here because I only have a minimal conceptual idea what those even mean lol | 20:26 |
| rm_work[m] | (in this context) | 20:26 |
| JayF | management interface is usually more like, what is my next boot device? am I UEFI or BIOS booting? | 20:26 |
| JayF | basically think of it as: | 20:28 |
| JayF | - API driven firmware update that supports OOB (and can be used in cleaning as well) | 20:28 |
| JayF | vs | 20:28 |
| JayF | - Creating a custom agent step that runs when you want it to (that just happens to update firmware) | 20:28 |
| jssfr | TheJulia, I managed to make the cursed intel board boot off an nfs-hosted image today. We'll work on getting more nodes up and once we reach the next batch of nodes with that intel board, I'll look into patching Ironic to support automatic translation of http to nfs URLs. So don't be surprised if it takes a couple weeks. | 20:28 |
| rm_work[m] | ok, I will go over your answers with someone who actually understands the firmware process and maybe I can actually get a grasp on this :P thanks | 20:32 |
| rm_work[m] | not sure why they didn't think the existing thing wouldn't work without code modifications, maybe there was a misunderstanding and I said something that made them think that | 20:34 |
| JayF | I mean, if you have something that you can run on a machine noninteractively and upgrade firmware if needed | 20:37 |
| JayF | you're only minor glue code into getting it done | 20:37 |
| rm_work[m] | yeah I just wasn't sure if code was needed AT ALL or if we just need to pass the right json blob somewhere, which is looks like ... MAYBE | 20:37 |
| JayF | look at https://opendev.org/openstack/ironic-python-agent in the examples directory | 20:39 |
| JayF | I think we explicitly have an in-band firmware update example | 20:39 |
| JayF | plus google for "ironic python agent hardware manager" for more info on that | 20:39 |
| JayF | we have good docs they are on IPA side not in Ironic docs tho | 20:39 |
| JayF | yeah, vendor device example | 20:40 |
| TheJulia | jssfr: ack, okay! | 21:26 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix ipv6 CI job https://review.opendev.org/c/openstack/ironic/+/939732 | 21:46 |
| rm_work[m] | JayF: ^^ rchow_geico is our actual firmware guy, so maybe he can explain our difficulties / differences from the upstream process and see if there's a good way for us to proceed, and if anything we're doing would actually be useful upstream or if it's all downstream garbage :D | 21:54 |
| JayF | I'm only going to help if I get to meet the talking gecko /s | 21:55 |
| rm_work[m] | hey I want to meet him too, me first :P | 21:56 |
| rchow_geico[m] | No but 15 minutes could save you 15% or more on car insurance | 21:56 |
| TheJulia | lol | 21:56 |
| TheJulia | .... Definitely can't on my mobile command center. | 21:56 |
| TheJulia | (RV is larger than Geico's cutoff) | 21:57 |
| rm_work[m] | wonder if there are friends and family exceptions... <_< | 21:57 |
| TheJulia | Eh, I'm in the realm of specialty policy, since National General knows they need to order a flatbed up if I need a tow.... | 21:58 |
| * TheJulia hopefully forever avoids such a need. | 21:59 | |
| JayF | how the hell did I not see that answer coming | 21:59 |
| TheJulia | magic?! | 22:00 |
| TheJulia | so I suspect the ipv6 job issue is also neutron is slow at getting radvd up as well, so in test scenarios the job fails. Also tinycore linux doesn't work since it has no dhcpv6 client, so hopefully my fix will just "work" | 22:01 |
| TheJulia | and then we make the v6 job voting | 22:01 |
| rchow_geico[m] | Thanks for the intro, i'll try to color the situation:... (full message at <https://matrix.org/oftc/media/v1/media/download/AaTYU5zwj-8pguQJcI1_9w8OMLC63Xm6f42xfWoF2x4BLKjo5RvxIrwcTbaWyRNrTe-80h01zlMEwH9DBgAshPZCeU0v3aBAAG1hdHJpeC5vcmcvQ1lMTUxqYUZnbGpTYmJ4VGNaVmtZWXJ6>) | 22:03 |
| TheJulia | #2 seems easy enough with a custom hardware manager, which depending on available steps it returns could involve itself to do it during deployment, or during something like cleaning | 22:06 |
| rchow_geico[m] | I saw some examples via the nvidia/marvell manager if that's the hardware manager you're referring to | 22:08 |
| TheJulia | Well, that is integrated in because they proposed the code, but realistically you can create your own hardware manager which exposes available steps which can then be pulled in | 22:08 |
| TheJulia | JayF is like the best advocate of hardware managers. :) | 22:09 |
| rm_work[m] | yeah that is the interface I was looking at extending, making my own manger there to do our custom steps, and generating images with the firmware blobs baked in rather than pulling them for consistency/verification | 22:14 |
| rm_work[m] | so basically you'd just say "step: do the thing" and it would ... do the thing | 22:14 |
| TheJulia | Yeah, that is sort of a preference item, up to you. I think like... cern puts them in a container and mounts the container up right before launching the agent | 22:15 |
| JayF | and I'd say in terms of "installed ironic doing things", most installations I know of are still using the in-band/step methods for firmware | 22:21 |
| JayF | obviously we don't want that to be true forever, but it's not like you'd be strange to go down that route | 22:21 |
| JayF | look at https://opendev.org/openstack/ironic-python-agent in the examples directory -- the vendor device one -- that's an example of what you need to do with a lot of info inline, and a google for "hardware managers ironic python agent" should get you the full docs from the IPA repo | 22:22 |
| rchow_geico[m] | kk will review this with rm_work and see if this makes sense between the both of us...thanks! | 22:33 |
| rm_work[m] | o7 | 22:34 |
| rm_work[m] | yeah maybe possible with no-code? would love that | 22:35 |
| rm_work[m] | or is this an example of how you'd do it given a custom device manager for a vendor | 22:35 |
| rm_work[m] | yeah ok reviewed the code, i see what you are saying, this IS an example hardware manager | 22:36 |
| JayF | if you want to do interesting hardware things, you're going to get into the business of a custom hardware manager | 22:36 |
| JayF | unless you only buy homogenous 100% perfectly redfishy hardware :D | 22:37 |
| rm_work[m] | that sounds fantastic, rchow_geico can we just switch to doing that? :D | 22:38 |
| rm_work[m] | I see there is also an example custom disk eraser, which is something else we likely want to do | 22:40 |
| rm_work[m] | great examples :D | 22:41 |
| TheJulia | Just as a general recommendation, we generally advise not to replace the actual step which can erase disks which is included in ironic-python-agent's source code. There have been some hard learned lessons captured in that code. That being said, it may have some downsides depending on exact cases your trying to support | 22:42 |
| TheJulia | One thing, for example, one of the redhat folks complained internally that ironic erased the thumb drive he put into the back of a DPU which he ran cleaning on | 22:43 |
| TheJulia | ... obviously, it presents as a block device, and is thus erased | 22:43 |
| rm_work[m] | ah I was thinking it would be an additional step -- that is the idea right? or are you saying we should probably just patch the existing one | 22:43 |
| TheJulia | Sort of depends on what you need to do and achieve, and what risk your attempting to manage at the end of the day | 22:43 |
| rm_work[m] | though I need to double-check if we REALLY need to do custom stuff, some things I think we can either get away with just doing what we get out of the box | 22:44 |
| rm_work[m] | or else ... punt on | 22:44 |
| rm_work[m] | yes, priorities, risk management, ROI | 22:44 |
| TheJulia | yup | 22:44 |
| TheJulia | exactly | 22:44 |
| TheJulia | (and occassionally push some worthwhile patches upstream) :) | 22:45 |
| rm_work[m] | I would like to make that happen, yes | 22:45 |
| JayF | rm_work[m]: we have seen more than a couple of folks who essentially thought "I can erase things with simpler code to make it More Secure(tm)" | 22:45 |
| JayF | rm_work[m]: so the cautionary tale is given :) | 22:45 |
| TheJulia | Speaking of... my next door neighbor just asked for the OCI Container Registry client stuffs. Folks should review that quick! :) | 22:45 |
| rm_work[m] | lol who are you in a caravan with? :D | 22:46 |
| TheJulia | JayF: Remember when that person came in after bricking all those disks?! :) | 22:46 |
| TheJulia | rm_work[m]: my next door neighbor is an old ironic contributor | 22:46 |
| rm_work[m] | nice | 22:46 |
| JayF | should I tell him you called him old? /s | 22:47 |
| rm_work[m] | eugh, offtopic -- i got back to openstack finally and was excited to get to meet up with folks again, but heard we're pretty much done with IRL summits? at least anything larger than local/regional? | 22:47 |
| TheJulia | JayF: *shrug* :) | 22:47 |
| JayF | I know many of us will be at OpenInfra Days NA colocated at SCALE | 22:48 |
| JayF | and SCALE is a great conference anyway | 22:48 |
| * TheJulia puts on the board chair hat | 22:48 | |
| rm_work[m] | well I missed the opportunity to go over to JayF's for BBQ when I was still in the area, now over on the east coast for a bit :/ | 22:48 |
| TheJulia | A side-effect of the pandemic is largely that in-person events have become harder to market from a massive sense of getting all community members into one place. Some of it is corporations just keeping tight travel budgets, coupled with the pandemic really changed people's views on travel and even exposure to others in terms of in a space and possibly getting sick. | 22:49 |
| rm_work[m] | yeah :/ | 22:50 |
| rm_work[m] | prior I always just expected con-plague was a thing and I'd be sick for a few days after | 22:50 |
| rm_work[m] | ¯\_(ツ)_/¯ | 22:50 |
| rm_work[m] | was usually worth it | 22:50 |
| TheJulia | As such, we have focused more to the regional events where we're also trying to overlap with other events to enable higher value of attendance *and* also enable cross-pollination of ideas/communities. | 22:50 |
| rm_work[m] | yeah makes sense | 22:50 |
| TheJulia | Yeah, I know of contributors who outright refuse all travel now | 22:51 |
| rm_work[m] | will have to see if we can get buy-in internally to head to some of these | 22:51 |
| TheJulia | I'm looking forward to SCaLE | 22:51 |
| TheJulia | but for me it is just a drive, really | 22:51 |
| rm_work[m] | well, i had 14 flights in 2024 with ~60h in the air, I am still good with travel :P | 22:52 |
| TheJulia | Yeah, I didn't fly as much as I would have preferred, but it is what it is | 22:53 |
| JayF | I am trying for a light-travel year | 22:55 |
| JayF | partially b/c it's likely I'll take some significant time off this summer | 22:55 |
| TheJulia | ++ | 22:55 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 23:41 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 23:41 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix ipv6 CI job https://review.opendev.org/c/openstack/ironic/+/939732 | 23:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
