| TheJulia | The way everything works that uses the base containers go liberaires, is they all look for that, and ultimatley the authentication systems decode and do similar to keystoneauth | 00:01 |
|---|---|---|
| TheJulia | but in that ecosystem there is a model of giving some of that to the user. The system is able to do the end resolution | 00:01 |
| JayF | yeah but it being a single static token is shocking | 00:01 |
| TheJulia | yeah | 00:07 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 00:12 |
| JayF | lets see how CI likes that | 00:12 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 00:18 |
| opendevreview | Jay Faulkner proposed openstack/sushy-tools master: Import needed ironic-lib code https://review.opendev.org/c/openstack/sushy-tools/+/939400 | 00:18 |
| rm_work[m] | hey JayF! Looks like i'm around and will be in ironic-land again :) | 00:23 |
| rm_work[m] | actually my first task here is to get firmware updates working via ironic so I may be by later with questions :P for now just reading up | 00:24 |
| opendevreview | Steve Baker proposed openstack/ironic master: WIP devstack start ironic-novnc-proxy as ir-novnc https://review.opendev.org/c/openstack/ironic/+/939402 | 00:26 |
| JayF | rm_work[m]: Nice to hear it. If there's anything I can do to help you out let me know | 00:35 |
| rm_work[m] | I'm about to become intimately familiar with IPA firmware update code :D | 00:43 |
| rm_work[m] | looks like it only really supports a handful of device types at the moment, is that correct? but if I wanted to add support for something, I'd just add a hardware manager for it? | 00:43 |
| JayF | So there's a new firmware update interface that landed recently, that Iury worked on. I'm not sure if we have that wired up to in-band yet. Are you sure your devices can't firmware update via redfish? | 01:59 |
| opendevreview | Merged openstack/ironic-python-agent master: Fix RAID volume name https://review.opendev.org/c/openstack/ironic-python-agent/+/939340 | 04:13 |
| janders | TheJulia Thank you, will test the patch out in the next few hours | 04:19 |
| janders | we had some wild storms rolling over with near lightning strikes so preemptively powered off most of the house | 04:19 |
| janders | back now, should be over, robovac cleaning up the mess on the deck | 04:20 |
| janders | (robovac is running outside ironic/metal3 for the record) | 04:20 |
| janders | TheJulia it seems like I was able to get a node from "service failed" back to "active" with your patch using the hold step | 05:05 |
| opendevreview | Verification of a change to openstack/ironic master failed: trivial: remove xclarity remenent https://review.opendev.org/c/openstack/ironic/+/935973 | 07:16 |
| rpittau | good morning ironic! o/ | 07:52 |
| dtantsur | TheJulia: the path can be ~/.docker/config.json, although I think there are other locations too | 08:00 |
| dtantsur | since $HOME is a weird notion for a service, I'm afraid we need a configuration option for it | 08:00 |
| dtantsur | (I see you already found the example) | 08:00 |
| rpittau | rm_work[m]: you can find recent docs on firmware update here https://docs.openstack.org/ironic/latest/admin/firmware-updates.html | 08:02 |
| rpittau | JayF: thanks for the entire ironic-lib deprecation stuff! I'll wait for sushy-tools, ironic-inspector, ipa and ipa-builder patches to merge before moving forward with the actual deprecation (governance, releases, and so on) | 08:45 |
| rpittau | kubajj: hey thanks for https://review.opendev.org/c/openstack/ironic-python-agent/+/939340 , can you please follow up with a release note for that? it will need to be included in the backports too | 08:55 |
| kubajj | rpittau: for sure | 09:01 |
| rpittau | thanks! | 09:01 |
| rpittau | any core around for a quick approval? https://review.opendev.org/c/openstack/metalsmith/+/933154 so we can release metalsmith thanks! | 09:23 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent master: Add a release note for 939340 https://review.opendev.org/c/openstack/ironic-python-agent/+/939424 | 09:24 |
| kubajj | rpittau: I think I did it wrong - I tried to have there the relation chain, but it is not there, sorry | 09:26 |
| rpittau | mm probably chain does not work as the other one is already merged | 09:30 |
| rpittau | kubajj: thanks, I left a comment | 09:31 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent master: Add a release note for 939340 https://review.opendev.org/c/openstack/ironic-python-agent/+/939424 | 09:34 |
| jssfr | hmmm | 10:56 |
| jssfr | looking into redfish with ironic for the first time | 10:56 |
| jssfr | is there a full tutorial somewhere? I'm coming from a PXE/IPMI based workflow and I don't quite see through what we need to change to use redfish. | 10:56 |
| priteau | jssfr: have you read https://docs.openstack.org/ironic/latest/admin/drivers/redfish.html | 11:17 |
| jssfr | priteau, mostly, yeah | 11:17 |
| jssfr | from that I gathered: I cannot just switch to redfish and everything will work. Instead I need to create an ESP which will magically boot the IPA initrd and kernel we already have. | 11:18 |
| jssfr | (provided I want to use redfish-virtual-media as boot method) | 11:18 |
| dtantsur | jssfr: for virtual media - yes. This is because we need to compose an ISO image to boot, and that needs a bootloader (instead of iPXE) | 11:19 |
| dtantsur | If the docs are too vague, you can also borrow inspiration from Bifrost (but please do tell us where the docs are vague) | 11:19 |
| jssfr | dtantsur, gladly! the link from priteau brought me here: https://docs.openstack.org/ironic/latest/install/configure-esp.html. THere, there's a lot of "from your distribution" and paths which are not filled in. It is not clear to me how to proceed from a place where I have a built IPA kernel + initrd. | 11:22 |
| dtantsur | jssfr: https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/tasks/create_esp.yml is a real-life script that uses distro-specific vars like https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/vars/redhat.yml#L9-L12? | 11:24 |
| dtantsur | Would it help if we provided examples in our docs? I"m somewhat worried about them getting outdated with distro changes | 11:24 |
| priteau | You could also try PXE+Redfish as a first step to validate the Redfish part of your setup | 11:24 |
| jssfr | priteau, that's a good idea | 11:25 |
| dtantsur | true | 11:26 |
| jssfr | we should definitely do that. I was looking at redfish right now because we've run out of cables for wiring the PXE interfaces, hence I was hoping for a quick fix (which I have by now realized won't get, so we ordered cables already :)) | 11:26 |
| jssfr | dtantsur, so, wait, another option would be to compose a complete ISO, right? Maybe we can tweak the IPA build process to generate an ISO, so we don't have to bother with that... | 11:28 |
| dtantsur | Yes, a complete ISO is also an option (we use it in openshift) | 11:30 |
| jssfr | I think that's what we'll look into then. Seems like fewer moving parts than attempting to build a working ESP. | 11:31 |
| jssfr | I'm trying to build an ISO with disk-image-builder, I'm getting: https://paste.debian.net/1345472/ | 12:38 |
| jssfr | I'm building with `disk-image-create -o ipa ironic-python-agent-ramdisk ubuntu devuser openssh-server mdadm-kill iso` (mdadm-kill is a thing we add, it doesn't install anything python-y, so hopefully that's irrelevant) | 12:38 |
| jssfr | I'll try to trim it down, but maybe someone has an idea what could be wrong there | 12:38 |
| jssfr | the same error happens with just `DIB_RELEASE=focal disk-image-create -o ipa ironic-python-agent-ramdisk ubuntu`, which I think should be fairly standard? | 12:40 |
| jssfr | this is with diskimage-builder==3.37.0 and ironic-python-agent-builder==5.4.0 | 12:41 |
| dtantsur | I haven't seen this error, and it sounds pretty absurd to me (9.7.0 should satisfy the requirement) | 12:41 |
| jssfr | yeah, that's what's baffling me, .oo | 12:42 |
| jssfr | *too | 12:42 |
| dtantsur | rpittau: you seem to deal with pip a lot, have you seen ^^? | 12:43 |
| jssfr | looks like using jammy (ubuntu 22.04) instead of focal (20.04) as base image works... | 12:48 |
| rpittau | mmm I was going to ask to try that :) | 12:57 |
| rpittau | If you're using latest ipa I suggest to use noble actually | 12:57 |
| jssfr | our deployment process later uses only jammy, I'd like to keep the two in sync | 13:07 |
| jssfr | (otherwise you get funny stuff with interface names) | 13:07 |
| jssfr | priteau, ftr, swicthing everything except boot to redfish just worked out of the box \o/ | 13:08 |
| priteau | Good to hear | 13:12 |
| jssfr | alright, now setting up boot via redfish-virtual-media. I built an ISO, put that in a place where Ironic can find it, set it as deploy_iso, and now I get: https://paste.debian.net/1345478/ | 13:27 |
| jssfr | ("The value http://image.ymc.f1c.cloudandheat.com:32080/redfish/boot-73e10a61-5cac-44af-a342-b325f6bb1f5e.iso for the property Image is of a different format than the property can accept.") | 13:27 |
| jssfr | does that mean that the redfish implementation cannot deal with HTTP URLs, or may that also be a different issue (such as that it cannot resolve the hostname or something like that) | 13:27 |
| jssfr | oof, reading documentation for this board that smells like the BMC indeed does not support HTTP, only SMB or NFS… | 13:34 |
| jssfr | can one trick ironic into making an nfs URL for that? | 13:34 |
| TheJulia | good morning | 14:23 |
| TheJulia | dtantsur: fun, okay, well... easy enough to have config, and parse it :) | 14:28 |
| TheJulia | Once I'm caffinated | 14:28 |
| dtantsur | jssfr: we don't support CIFS and NFS sorry | 15:39 |
| dtantsur | good morning TheJulia | 15:39 |
| TheJulia | jssfr: That being said, if you develop a patch for it, we would review it. I'm not sure it would be feasible to test in upstream though.. That being said, it should also be a small enough variation in the code path that we might be okay with unit testing if you posted the patch confirming you got $vendor hardware working with it. | 15:41 |
| jssfr | I briefly wondered about messing with the http_url template, but I suspect that would break ironic's attempt to validate the existence of images before deploy, right? | 15:41 |
| jssfr | so vendor hardware which rejects http(s) URLs with redfish-virtual-media, is it likely that it'd work with redfish-https? | 15:42 |
| TheJulia | we use that url all over the place | 15:42 |
| jssfr | I suspected as much :) | 15:42 |
| TheJulia | so changing it would be super bad | 15:42 |
| jssfr | I was also surprised to see that ironic copies the image, even though deploy_iso(?) was set to an http URL. I expected it to just pass that URL to the node. | 15:42 |
| TheJulia | I'd create a new parameter, identify the vendor through existing mechanisms if possible (we save the vendor), and use the new config parameter for virtual media to source that base url to post to the BMC, that way being vendorish specific logic *and* keeping the rest of the overall impact to a minimum | 15:43 |
| jssfr | TheJulia, oh. I was thinking of maybe a deploy_url_template which can be set as driver info. | 15:43 |
| TheJulia | Well, now it is a good thing it does because you can't hand that bmc a url | 15:43 |
| jssfr | which takes the filename relative to http_root | 15:44 |
| TheJulia | I'd highly suggest trying to do it based upon the hardware vendor | 15:44 |
| TheJulia | that way the user doesn't need to turn another knob or even know of the knob for that specific type of hardware | 15:44 |
| jssfr | the user would have to provide an NFS share anyway | 15:44 |
| jssfr | (or SMB) | 15:44 |
| TheJulia | only the system admin who configured ironic needs to have the necessary configuration anyway | 15:44 |
| jssfr | oh I see | 15:44 |
| jssfr | understood | 15:45 |
| jssfr | right | 15:45 |
| TheJulia | That can be at a system level, and it could just be an export of the httpboot directory for all we know | 15:45 |
| jssfr | I was only looking at this from the perspective of someone using ironic in the "undercloud", if you know what I mean, not as a MaaS service. | 15:45 |
| TheJulia | ahh, yeah | 15:45 |
| jssfr | In the latter case, having it be a driver-info would be horrid indeed. | 15:45 |
| TheJulia | still pretty much the same, but yeah | 15:45 |
| * TheJulia goes back to OCI image registry stuffs | 15:45 | |
| jssfr | okay, so: (1) identify vendor, (2) make config option `nfs_url`(?), (3) select `nfs_url` instead of `http_url` when deploying to that vendor (and maybe force use_swift to false for that vendor, too?) | 15:46 |
| TheJulia | Sort of what I was thinking | 15:46 |
| TheJulia | its not great they use nfs or cifs only | 15:47 |
| jssfr | yeah | 15:47 |
| TheJulia | (this is where you get to share the name of the vendor to shame them ever so slightly) | 15:47 |
| jssfr | you'd think http would be easier to implement than either of these. | 15:47 |
| jssfr | intl | 15:47 |
| jssfr | *intel | 15:47 |
| TheJulia | (very much so) | 15:47 |
| TheJulia | .... | 15:47 |
| TheJulia | why am I not surprised?! | 15:47 |
| jssfr | then again, HTTP requires to buffer the image somewhere, unless you're starting to mess with range requests when it probably gets slooow | 15:47 |
| TheJulia | (is this a reference board, or production gear?) | 15:47 |
| jssfr | while NFS / CIFS are arguably made for this kind of use. | 15:47 |
| TheJulia | ((i ask, because ironic is *stupidly* popular in large scale hardware test labs)) | 15:48 |
| jssfr | I'm off my work machine already, I can't look up the hardware---but I'd hope it's production gear, as it's in many of our servers and has been for years. | 15:48 |
| TheJulia | okay | 15:48 |
| TheJulia | Intel walks a really weird line about their own servers, and really always has | 15:48 |
| * TheJulia only once has ordered an "Intel" server. | 15:49 | |
| jssfr | I'm not sure if I'd call our DC a "large scale hardware test lab", though some detractors might :D | 15:49 |
| TheJulia | .... and it was an 8 "CPU" beast... which used processor module cards.... | 15:49 |
| jssfr | TheJulia, you have a talent for making me consider making a patch for upstream instead of working around issues, I like that :D | 15:50 |
| TheJulia | (If you ever wonder how much of a pain it it was to find those processor cards... it was insane) | 15:50 |
| TheJulia | jssfr: I really hope you do, it stinks that the gear only supports cifs and nfs, but... it is what it is :) | 15:51 |
| TheJulia | our universal job is to move forward :) | 15:51 |
| jssfr | it is indeed | 15:51 |
| jssfr | I mean iPXE works, but PXE networks are such a mess | 15:51 |
| TheJulia | Yeah | 15:51 |
| jssfr | so do you think a dumb check for "is this intel?" is enough, or should we try to narrow this down somehow? | 15:51 |
| TheJulia | we save the bmc reported hardware vendor as something like task.node.properties.get('vendor') | 15:52 |
| TheJulia | ... I don't know | 15:52 |
| TheJulia | intel is a wild one | 15:52 |
| jssfr | practically speaking, I think we have rather consistent hardware and can't even sample more than that. | 15:53 |
| TheJulia | Maybe try http, if that fails and it is intel try falling back in the actual redfish driver? | 15:53 |
| jssfr | oh that's a good idea | 15:53 |
| jssfr | since we get a clear error from the BMC implementation | 15:53 |
| TheJulia | because... honestly, intel should evolve and improve | 15:53 |
| TheJulia | exactly | 15:53 |
| TheJulia | ... I made a funny | 15:53 |
| jssfr | I like that | 15:53 |
| TheJulia | intel evolving and improving | 15:53 |
| TheJulia | Anyway! back to work! | 15:54 |
| jssfr | have fun / good luck | 15:55 |
| jssfr | depending on what applies in this particular case | 15:55 |
| TheJulia | Sort of both, I'm deep into work to support using oci image registries for artifacts and images | 15:59 |
| opendevreview | Muhammad Ahmad proposed openstack/ironic master: doc/source/admin fixes part-4 https://review.opendev.org/c/openstack/ironic/+/939467 | 16:03 |
| JayF | Does anyone have any environments where ramdisk driver is used with integrated openstack+nova? I can't think of a reason it wouldn't work... | 16:03 |
| TheJulia | JayF: likely easier to ask scientific computing folks on their slack | 16:26 |
| TheJulia | since they are the ideal users | 16:26 |
| TheJulia | That being said, you've asked the questio before, and it *should* work | 16:26 |
| JayF | I'll probably just try to get it to work in a devstack | 16:26 |
| JayF | if there's any rough edges I have time to smooth em over | 16:26 |
| rpittau | good night! o/ | 16:52 |
| * TheJulia laughs evilly was cirros boots | 18:28 | |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 18:30 |
| TheJulia | well, kind of booted, might be an isue with 0.6.3 | 18:30 |
| TheJulia | "close enough" | 18:30 |
| opendevreview | Luke Odom proposed openstack/ironic-python-agent master: Update mdadm array name for posix compliance https://review.opendev.org/c/openstack/ironic-python-agent/+/939483 | 19:13 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP OCI container adjacent artifact support https://review.opendev.org/c/openstack/ironic/+/937896 | 21:18 |
| TheJulia | almost there, just need to sit down and write unit tests for the actual oci client code | 21:22 |
| TheJulia | curious if we should be backporting raid fixes | 21:28 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Treat 'No space left on device' error as fatal https://review.opendev.org/c/openstack/ironic-python-agent/+/939500 | 22:10 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 22:16 |
| JayF | TheJulia: IMO, yes. kubajj said he'd take care of his raid fix | 22:16 |
| kubajj | JayF: do we backport just to stable versions or also unmaintained ones? | 22:19 |
| JayF | so you have to go in reverse order, so you'd do 2024.2, once that lands 2024.1 can land, and so on | 22:20 |
| JayF | unmaintained branches are best effort | 22:20 |
| JayF | so generally: backport to relevant stable branches; if you're feeling charitable or are a user of unmaintained branches, keep putting them back | 22:21 |
| JayF | but TBH, something of this low impact I'd probably not take that far, the gate gets hairier the further you get back in time :D | 22:21 |
| opendevreview | Jay Faulkner proposed openstack/ironic-lib master: Deprecate ironic-lib master branch https://review.opendev.org/c/openstack/ironic-lib/+/939277 | 22:22 |
| kubajj | JayF: I will try to go as far as I can. Would say the impact is moderate (any node with target RAID config without volume name set is unable to clean) | 22:23 |
| opendevreview | Jay Faulkner proposed openstack/ironic-lib master: Deprecate ironic-lib master branch https://review.opendev.org/c/openstack/ironic-lib/+/939277 | 22:23 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent-builder master: Deprecate ironic-lib https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/939288 | 22:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!