| opendevreview | cid proposed openstack/ironic master: Follow-up: Apply Inspection Rules https://review.opendev.org/c/openstack/ironic/+/942112 | 00:19 |
|---|---|---|
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 00:19 |
| opendevreview | cid proposed openstack/ironic master: Include all relevant error messages in exception https://review.opendev.org/c/openstack/ironic/+/942664 | 00:31 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix glance metadata layout https://review.opendev.org/c/openstack/ironic/+/942496 | 00:35 |
| cardoe | JayF: now with a release note :) | 00:35 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic master: Add extra log to is_image_available https://review.opendev.org/c/openstack/ironic/+/942641 | 01:16 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: hooking in an external network simulator https://review.opendev.org/c/openstack/ironic/+/942298 | 01:18 |
| TheJulia | hjensas: ^, getting there :) | 01:19 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic master: Add extra log to is_image_available https://review.opendev.org/c/openstack/ironic/+/942641 | 01:31 |
| TheJulia | stevebaker[m]: can https://review.opendev.org/c/openstack/ironic/+/942647 be based off of master branch? | 01:31 |
| stevebaker[m] | TheJulia: yeah it is very standalone. Would provisioning test nodes via nova even work without that fix? | 01:34 |
| TheJulia | traits are optional | 01:34 |
| opendevreview | Steve Baker proposed openstack/ironic master: Fix default IRONIC_DEFAULT_TRAITS setting https://review.opendev.org/c/openstack/ironic/+/942647 | 01:37 |
| stevebaker[m] | : and :- seem to behave the same, huh | 01:43 |
| stevebaker[m] | - and :- I mean | 01:43 |
| TheJulia | ... interesting | 01:52 |
| opendevreview | Steve Baker proposed openstack/ironic master: Add systemd provider for console containers https://review.opendev.org/c/openstack/ironic/+/941614 | 03:47 |
| opendevreview | Steve Baker proposed openstack/ironic master: Implement drivers redfish-graphical, fake-graphical https://review.opendev.org/c/openstack/ironic/+/941615 | 03:47 |
| opendevreview | Steve Baker proposed openstack/ironic master: Add vnc-container image build https://review.opendev.org/c/openstack/ironic/+/942017 | 03:47 |
| opendevreview | Steve Baker proposed openstack/ironic master: Implement graphical console read-only support https://review.opendev.org/c/openstack/ironic/+/942300 | 03:47 |
| cardoe | Dang it. I put in a release note and it fails vmedia. | 04:05 |
| opendevreview | Vasyl Saienko proposed openstack/ironic master: Enable trunk plugin for tinyipa-multinode https://review.opendev.org/c/openstack/ironic/+/941023 | 07:58 |
| rpittau | good morning ironic! o/ | 08:09 |
| dtantsur | /opt/ironic-python-agent/bin/python3 -m oslo_concurrency.prlimit --as=2147483648 -- ['env', 'LC_ALL=C', 'LANG=C', 'qemu-img', 'info', '/tmp/cirros-0.6.2-x86_64-disk.img', '--output=json'] | 13:18 |
| dtantsur | Is this an actual sane way to use prlimit? ^^^ | 13:18 |
| dtantsur | meanwhile, if anyone has cycles, could you help the SDK team make the ironic job voting again? https://review.opendev.org/c/openstack/openstacksdk/+/942625 | 13:19 |
| dtantsur | re prlimit: the right way seems to be with spaces not as a Python list... | 13:20 |
| dtantsur | python3 -m oslo_concurrency.prlimit --as=2147483648 -- env LC_ALL=C LANG=C qemu-img info /tmp/cirros-0.6.2-x86_64-disk.img --output=json | 13:20 |
| dtantsur | Jeeeesssssuuuuus, okay, I got it, patch incoming | 13:23 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Fix the way qemu-img is called with prlimits https://review.opendev.org/c/openstack/ironic-python-agent/+/942690 | 13:27 |
| dtantsur | JayF: I have no explanation why our CI does not fail because of ^^ | 13:28 |
| dtantsur | I also have no explanation why the Metal3 CI uses to pass with older Ironic and only hit this when we tried to remove the pin | 13:29 |
| JayF | dtantsur: oslo library changed perhaps? | 13:43 |
| dtantsur | I don't see any changes recently.. | 13:43 |
| JayF | Especially if a new version isn't in uc | 13:44 |
| dtantsur | I wonder if this qemu-img call is actually rarely hit | 13:44 |
| dtantsur | e.g. because we convert to raw in devstack | 13:44 |
| JayF | That is probably exactly what's happening. The original version of this back ported fix broke my downstream because they don't force raw | 13:44 |
| dtantsur | Right.. | 13:47 |
| JayF | dtantsur: also perhaps a subtle change in behavior between the ir-lib execute and the one in IPA? | 13:48 |
| dtantsur | Could be as well? | 13:48 |
| dtantsur | In any case, now I have one more reason to try running the metal3 e2e tests in the Ironic CI | 13:48 |
| JayF | I'm going to +2 this from mobile but I'll also make a point to take your work later and see if the same syntax exists in IPA that needs fixing | 13:49 |
| dtantsur | This is IPA (which is also puzzling: we're unpinning Ironic, not IPA) | 13:50 |
| JayF | In which case I would check logs to see if something is being converted now that wasn't before | 13:54 |
| TheJulia | good morning | 14:09 |
| opendevreview | Harald Jensås proposed openstack/ironic master: devstack bindep - [platform:rpm] https://review.opendev.org/c/openstack/ironic/+/893813 | 14:10 |
| TheJulia | it feels like there is a smoldering pit where our CI should be | 14:28 |
| TheJulia | hjensas: any thoughts on maybe considering making the ipv6 job non-voting for a little while, in order to try and sort through some of the issues: | 14:33 |
| TheJulia | See: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3da/942633/1/check/ironic-tempest-ovn-uefi-ipxe-ipv6/3dabc21/controller/logs/ironic-bm-logs/node-1_console_2025-02-24-19%3A24%3A09_log.txt and | 14:33 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3da/942633/1/check/ironic-tempest-ovn-uefi-ipxe-ipv6/3dabc21/controller/logs/ironic-bm-logs/node-1_console_2025-02-24-19%3A08%3A26_log.txt | 14:33 |
| TheJulia | and I just spotted a job now failing because of the image format inspector | 14:36 |
| TheJulia | https://www.irccloud.com/pastebin/8DE5UCbK/ | 14:36 |
| opendevreview | cid proposed openstack/ironic master: Make port binding failure (configurably) fatal https://review.opendev.org/c/openstack/ironic/+/699953 | 14:37 |
| TheJulia | new oslo.utils on feb21 | 14:42 |
| TheJulia | looks like glance has changes landing which is why this is happening | 14:45 |
| hjensas | TheJulia: if it is giving us a headache, I'm fine putting it non-voting. | 14:49 |
| TheJulia | glance changes may have torpedoed us anyhow | 14:50 |
| TheJulia | Yeah, we have multiple jobs failing presently in the last few days uploading artifacts to glance | 14:54 |
| TheJulia | I think our CI is dead. | 14:54 |
| opendevreview | Harald Jensås proposed openstack/ironic master: Set ovn-uefi-ipv6-job non voting https://review.opendev.org/c/openstack/ironic/+/942702 | 14:56 |
| hjensas | so, if that is particularly br0ken on the ipv6 job ^ | 14:57 |
| TheJulia | multiple jobs since a change in glance merged on the 21st to switch to the oslo.utils image format inspector | 15:00 |
| TheJulia | which includes "if it looks like mbr" block it | 15:00 |
| TheJulia | logic :( | 15:00 |
| jssfr | (also gpt?) | 15:02 |
| TheJulia | that too | 15:04 |
| TheJulia | although, there is a patch in flight to permit that case | 15:04 |
| TheJulia | https://review.opendev.org/c/openstack/glance/+/939727 | 15:04 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM/Test glance gpt image format inspector fix pending merger https://review.opendev.org/c/openstack/ironic/+/942703 | 15:09 |
| TheJulia | That likely won't do anything | 15:17 |
| TheJulia | Dan is working on re-spinning an updated version or making a default change in another patch | 15:18 |
| vsaienko | TheJulia, dtantsur I have a feeling that multinode jobs are totally broken in ironic at least they can't pass on any of patches for a week or more. Maybe you know where that playbook that setup connectivity between multinodes is located? | 15:24 |
| TheJulia | vsaienko: there are other breaking changes which have merged, so CI is toasty at the moment | 15:25 |
| TheJulia | vsaienko: the base vxlan tunnel setup should be in zuul definitions, but we have br-infra tunnel setup in our plugin | 15:25 |
| * dtantsur does not remember the multinode job well | 15:26 | |
| TheJulia | that tunnel setup, if memory serves, duplicates what the zuul multinode default ends up being | 15:26 |
| vsaienko | br-infra is used for floating network right? I remember that in devstack-gate time we had dedicated vxlan for ironic purposes only | 15:27 |
| TheJulia | vsaienko: two things happened, when devstack-gate was gotten rid of, zuul created one for multi-node allocations (if my memory is correct) and ironic duplicated the setup in bash just in case for our multinode job. Given the state of CI your likely not going to get past glance at the moment. | 15:28 |
| TheJulia | or to be more precise, our plugin starting uploads into glance | 15:28 |
| TheJulia | br-infra is for general purpose usage | 15:28 |
| TheJulia | a shared common network since nodes can be in disjointed | 15:29 |
| jamesdenton__ | good morning. does anyone know offhand if the neutron network_interface is required for proper port group functionality? | 15:45 |
| jamesdenton__ | neutron vs flat, that is. | 15:45 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic master: Add extra log to is_image_available https://review.opendev.org/c/openstack/ironic/+/942641 | 15:46 |
| dtantsur | cid: so, the copying is because of _mask_sensitive_data in apply_rules | 15:54 |
| dtantsur | which makes sense. but it also means we cannot just assign the plugin_data from rules to the initial plugin_data: it will overwrite any masked information | 15:55 |
| * dtantsur scratches his head | 15:56 | |
| cid | That makes a lot of sense. | 15:57 |
| cid | dtantsur: When would an operator/user want to modify the plugin_data? I'm thinking if we could add a check to make sure that action runs last | 15:58 |
| dtantsur | I don't know, to be honest. Maybe we can take these actions out for now and see if anyone complains? | 15:58 |
| * cid replying to "we cannot just assign the plugin_data from rules | 15:58 | |
| dtantsur | Otherwise, the most sane idea that comes to my mind is to wrap plugin_data into a dict-like object that respects the sensitivity setting when a key is read from it (but actually operates on the unmasked data) | 15:59 |
| cid | Hmm. That's one way of doing it. So, only read output is masked but the engine can still access the unmasked data. | 16:02 |
| * dtantsur nods | 16:02 | |
| cid | Ok great. | 16:03 |
| cid | Sounds like you are still looking at the rest, so I will let you and just update everything at once (?) | 16:04 |
| dtantsur | cid: I'm running out of time here again, so go ahead with the update, we'll discuss the rest | 16:11 |
| dtantsur | if you could throw in some unit tests, would be awesome | 16:11 |
| dtantsur | it's easier to review the complex logic when you see how it is used in practice | 16:11 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM/Test glance gpt image format inspector fix pending merger https://review.opendev.org/c/openstack/ironic/+/942703 | 16:12 |
| cid | dtantsur, Alright. I will get to it in a moment. Thanks! | 16:13 |
| TheJulia | jamesdenton__: so I don't know offhand, I think the answer is semi-yes, but flat is just a strucutral data model preference to pre-configured state | 16:13 |
| jamesdenton__ | so the question is less about the end state and more about the setup of the neutron ports during the 'deploy' step. Seems like, in this instance, anyway, that the neutron port is not being created for the pxe-enabled baremetal port. Seems to work on clean, but not deploy. I am waiting for more info. | 16:20 |
| jamesdenton__ | wasn't sure if the neutron network_interface was required for port groups, in general. sounds like it's not | 16:21 |
| jamesdenton__ | thank you :) | 16:21 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: hooking in an external network simulator https://review.opendev.org/c/openstack/ironic/+/942298 | 16:43 |
| rpittau | good night! o/ | 17:03 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Fix the way qemu-img is called with prlimits https://review.opendev.org/c/openstack/ironic-python-agent/+/942690 | 17:35 |
| JayF | ugh I didn't knoow CI was in hell before I approved that :( | 17:39 |
| JayF | Anyone used Ironic with AMI based megarac BMCs? Looks to be vaguely openbmc based https://www.ami.com/megarac/ | 18:11 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: hooking in an external network simulator https://review.opendev.org/c/openstack/ironic/+/942298 | 18:12 |
| JayF | TheJulia: is there any action ironic side on gate fixes? I just emerged from a morning of pairing | 18:12 |
| JayF | dtantsur: cid: Wasn't the canonical "modify plugin data" use case that lldp was two steps -> raw data that then gets parsed | 18:13 |
| JayF | dtantsur: cid: or did we get rid of that weird interaction when migrating inspector into ironic | 18:13 |
| TheJulia | JayF: dansmith has a fix in https://review.opendev.org/c/openstack/glance/+/939727 which passes our CI as well :) | 18:13 |
| * JayF gets out his bullhorn to find another core | 18:13 | |
| TheJulia | heh | 18:14 |
| TheJulia | good, I've been heads down trying to work up switch simulation | 18:14 |
| TheJulia | #whatamicrazy | 18:14 |
| TheJulia | #juliamustbecrazy | 18:14 |
| JayF | if you need someone to quack at about any of that stuff please ask me :) | 18:14 |
| JayF | I know generic networking really well even if I don't ov* so much :D | 18:14 |
| TheJulia | JayF: not quack, honk! | 18:15 |
| JayF | I don't think I've ever seen a rubber goose. | 18:15 |
| TheJulia | eh, plenty by the lake here | 18:16 |
| JayF | oh I'm really confused | 18:16 |
| TheJulia | heh | 18:16 |
| TheJulia | good | 18:16 |
| JayF | that glance patch looks like a big chain of patches | 18:16 |
| TheJulia | goooood | 18:16 |
| JayF | not just the one | 18:16 |
| JayF | https://review.opendev.org/c/openstack/glance/+/939727 is stacked under like, two unreviewed changes afaict | 18:16 |
| TheJulia | ugh yeah | 18:17 |
| TheJulia | let me get my local devstack re-spinning and I'll give it a spin | 18:17 |
| TheJulia | well some reviewing | 18:17 |
| opendevreview | Vasyl Saienko proposed openstack/ironic master: Do not review debug multinode https://review.opendev.org/c/openstack/ironic/+/942725 | 19:17 |
| opendevreview | Vasyl Saienko proposed openstack/ironic master: Do not review debug multinode https://review.opendev.org/c/openstack/ironic/+/942725 | 19:19 |
| JayF | vsaienko: unless you Depends-on the 939727 ^^ glance patch that won't get far | 19:19 |
| dansmith | JayF: that's the wrong change, you want to depends-on https://review.opendev.org/c/openstack/glance/+/939243 | 19:19 |
| JayF | ack vsaienko ^ dan knows :) | 19:19 |
| dansmith | the one you referenced brings in a lot more change and isn't going to land this cycle due to lack of review | 19:20 |
| JayF | aha, makes sense; I was just looking at the patch julia had tested our ci against | 19:20 |
| dansmith | AFAIK, that test patch was against 243, not 727 | 19:21 |
| JayF | you are 100% right | 19:22 |
| dansmith | ah, 727 at first, but that was a mistake | 19:22 |
| vsaienko | JayF thanks, I will look on that | 19:23 |
| rm_work[m] | JayF: can you manually run a clean stage on a server that is actually... provisioned? like, emergency firmware updates, people have servers provisioned, can we be like "run just the firmware clean phase on these servers without taking them back"? | 19:28 |
| JayF | https://docs.openstack.org/ironic/latest/admin/servicing.html | 19:29 |
| rm_work[m] | reading, thanks | 19:29 |
| JayF | np, that's literally the use case that feature was written for :D | 19:29 |
| rm_work[m] | yeah looks good, need to check if we've set up this service network or if that's yet another thing to add to our list | 19:31 |
| * TheJulia glares at the switch simulator vm | 19:53 | |
| stevebaker[m] | good morning | 19:53 |
| TheJulia | good morning | 19:59 |
| cardoe | /giphy winning | 20:21 |
| cardoe | assuming you guys let me break something that's not really documented | 20:21 |
| cardoe | But ESXi installation works. | 20:21 |
| cardoe | or add a new field into the kickstart templating and actually document this one | 20:22 |
| cardoe | https://opendev.org/openstack/ironic/src/commit/b44cce176fbb8c81c813304cd443d0f2c20ab6b2/ironic/common/kickstart_utils.py#L121 essentially rather than returning a string that's got a bunch of hardcoded shell bits. I've made a dictionary of the filenames as the key and the base64 contents. Then used a jinja snippet in the ks_cfg.template to have the same end result for the existing case. | 20:24 |
| cardoe | But in the ESXi case, the jinja snippet can be different since the commands and the paths need to be slightly different. | 20:24 |
| TheJulia | #success cardoe got ESXi to install vi Ironic's anaconda deployment interface. | 20:39 |
| opendevstatus | TheJulia: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 20:39 |
| * TheJulia laughs evilly | 20:40 | |
| cardoe | ha didn't know that existed. | 20:44 |
| JayF | Honestly it felt like a success for Adam (rm_work) to ask for a feature we had already written | 20:48 |
| JayF | I'm so used to people asking for stuff we haven't done yet :D | 20:48 |
| rm_work[m] | :P | 21:17 |
| rm_work[m] | it's been like a month and I haven't actually gotten to like... test anything yet, or even really start coding on the firmware stuff i'm trying to do, because I have been in the weeds with other stuff, currently trying to spin up ScrewDriver.cd ;) | 21:18 |
| rm_work[m] | wondering if maybe I should just set up Zuul instead lol | 21:18 |
| rm_work[m] | but I really like the screwdriver UI | 21:19 |
| JayF | I prefer a nice neon yellow shed myself /s | 21:20 |
| cardoe | So my glance image properties change I think results in a bunch more properties appearing on the instance_info | 22:00 |
| cardoe | Can someone with an instance built from glance peek at instance_info for me? | 22:01 |
| cardoe | Like basically a bunch of default glance properties are now appearing like "hw_cpu_sockets" | 22:03 |
| cardoe | I'm trying to write a test for the code basically. | 22:04 |
| JayF | Hmm, that's very much not ideal | 22:15 |
| cardoe | It seems like the behavior change around this is from https://opendev.org/openstack/ironic/commit/978afbd5a1bba71181c5a0a7cb3ebfb20132c438 | 22:27 |
| cardoe | I don't really know how to proceed here. The shape of the object coming from glanceclient is different from the shape of the one coming from the SDK. | 22:29 |
| JayF | I don't know what the best answer is | 22:30 |
| JayF | but I'll say "is not ideal" doesn't mean nack | 22:31 |
| JayF | time for instance_image_info /s /s /s | 22:31 |
| cardoe | So the issue is that the object from glanceclient only has setattr() called on it for fields that exist. | 22:35 |
| JayF | does this mean you could do something like | 22:35 |
| cardoe | While the SDK defines all possible things that are allowed to be set on an Image | 22:35 |
| JayF | filter Nones out? | 22:35 |
| JayF | or something simliar? | 22:36 |
| cardoe | Well we copy any attribute not in our list to properties. | 22:36 |
| cardoe | And that's why you're getting all the junk. | 22:36 |
| JayF | I'm either not following you or you're not following me: | 22:37 |
| JayF | - glanceclient used to only return values that were set | 22:37 |
| JayF | - SDK returns an Image object with ALL THE FIELDS | 22:37 |
| cardoe | Have ya messed with dataclasses? | 22:37 |
| JayF | - This means we end up with properties[ALMOST_ALL_THE_FIELDS] -- but given staatement #1, it implies that many of them are nones | 22:37 |
| cardoe | Yes. They are. | 22:37 |
| JayF | Can we filter nones at the point we copy things into the properties? | 22:38 |
| JayF | I'm suggesting, it might be an easy path to getting junk out | 22:38 |
| JayF | with an obvious assumption that None is not a meaningful value for that Image object | 22:38 |
| * JayF brb | 22:38 | |
| cardoe | I can. | 22:39 |
| cardoe | I've gotta wonder however if that's the point of the code. | 22:40 |
| cardoe | Cause we hardcode a list of "attributes" which become our top-level fields in our wrapper object. | 22:41 |
| cardoe | We then copy everything else into the properties dictionary. | 22:41 |
| cardoe | Which happened to be only things a user set on the image. | 22:41 |
| cardoe | Cause I only see where we care about stuff the user has set. | 22:48 |
| JayF | yeah | 22:54 |
| JayF | for your stuff though, you need the flexibility, right? | 22:54 |
| JayF | you're using the fact we're copying * now to do your esxi? | 22:54 |
| cardoe | No. Just what a human has set in properties. | 22:54 |
| JayF | oh, then you're probably right about why the existing code was there | 22:54 |
| JayF | although it's still a set of bad choices all around tbh | 22:55 |
| cardoe | I'm gonna do the None filter to be the minimal change for the backport and not interrupt any other behavior. | 22:55 |
| cardoe | I'm also gonna nuke FakeImage which is what the tests use. It generates a glanceclient shape. | 22:55 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix glance metadata layout https://review.opendev.org/c/openstack/ironic/+/942496 | 23:03 |
| cardoe | I think that's a safe backportable change. | 23:04 |
| opendevreview | cid proposed openstack/ironic master: Follow-up: Apply Inspection Rules https://review.opendev.org/c/openstack/ironic/+/942112 | 23:05 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 23:05 |
| cardoe | My bigger change I believe is correct and ultimately less code. But it risks breaking any user of image_properties that relies on a field I'm not able to set in my local testing on the top-level of a Glance object and being copied into the nested dict. | 23:05 |
| cardoe | Basically I nuked ALL the translation functions and just pass the SDK Image object around natively. | 23:05 |
| cardoe | And I added type annotations and ran it through mypy. | 23:06 |
| JayF | 🤩 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/942369 | 23:47 |
| JayF | this is awesome | 23:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!