| rpittau | good morning ironic! o/ | 06:57 |
|---|---|---|
| queensly[m] | Good morning | 07:54 |
| abongale | good morning ironic! | 08:45 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/sushy master: Don't fail to eject vmedia in case there is no vmedia https://review.opendev.org/c/openstack/sushy/+/952730 | 10:58 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Trivial: avoid root logger in modules https://review.opendev.org/c/openstack/ironic-python-agent/+/954243 | 13:17 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Hard stop on image download duration threshold https://review.opendev.org/c/openstack/ironic-python-agent/+/954245 | 13:38 |
| opendevreview | Merged openstack/sushy master: Don't fail to eject vmedia in case there is no vmedia https://review.opendev.org/c/openstack/sushy/+/952730 | 14:05 |
| cardoe | o/ | 15:01 |
| rpittau | mmm looks like no meeting today | 15:02 |
| kubajj | 🥲 | 15:02 |
| queensly[m] | o/ | 15:02 |
| rpittau | I declare no-quorum :) | 15:02 |
| cid | ++ | 15:03 |
| alegacy | o/ | 15:03 |
| clif | o/ | 15:03 |
| TheJulia | Yeah, I suspected when I said I wouldn't be able to really make it for the meeting today that it wouldn't take place. | 15:19 |
| TheJulia | sorry :( | 15:19 |
| rpittau | TheJulia: no worries! we should probably rotate the meeting host | 15:22 |
| TheJulia | Yeah, I was hoping someone would just step in :) | 15:22 |
| TheJulia | Anyway, I'm going to go be an annoying customer and hover by the services door once I'm done chekcing my email | 15:23 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Hard stop on image download duration threshold https://review.opendev.org/c/openstack/ironic-python-agent/+/954245 | 15:28 |
| opendevreview | Merged openstack/sushy-tools master: Add PATCH support for Redfish DateTime fields in Manager resource https://review.opendev.org/c/openstack/sushy-tools/+/950925 | 15:30 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest API tests for inspection rules https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 16:08 |
| dtantsur | Folks, we have changes like https://review.opendev.org/c/openstack/bifrost/+/946726 that needs to be rebased, but gerrit rejects rebasing because of missing Sign-Off. | 16:45 |
| dtantsur | The author seems to be inactive, but the change is quite valuable. Is it appropriate for me to Sign-Off it? | 16:45 |
| dtantsur | I'm not sure how this stuff works... | 16:45 |
| dtantsur | TheJulia, frickler, if you have any idea ^^ | 16:45 |
| * dtantsur will read responses tomorrow if irccloud does not lose them again.. | 16:46 | |
| TheJulia | so, my $0.02 is it was submitted originally before the sign-off-by requirement | 16:46 |
| TheJulia | that being said, to rebase at this point, yes, you would need to sign off on it | 16:47 |
| dtantsur | yeah, and without rebase it's blocked on the parent patch, which may not see any updates.. | 16:47 |
| TheJulia | Given it was originally covered by the CLA, you would just amend it at this point | 16:47 |
| dtantsur | I can the question is whether it's fine for *me* to sign it off on my behalf, despite the author being a different human | 16:47 |
| TheJulia | For example, I amended one of cid's patches last week, while he already had sign-off-by, I signed off on the resulting state after I rebased/amended it. | 16:48 |
| dtantsur | I see | 16:49 |
| TheJulia | so I added mine as a result. Its sort of how you get multiple people signing off on a single change at times. | 16:49 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: bug: drop baremetal introspection mention https://review.opendev.org/c/openstack/bifrost/+/946726 | 16:49 |
| Sandzwerg[m] | <TheJulia> "so, easy short term test, add -p..." <- That was what I was looking for, it still fails but at least right now I can rule out shim in the ramdisk. Still trying to figure out whats going on. Hadn't had time on Friday after all but toyed with it today. | 16:50 |
| dtantsur | thank you TheJulia! | 16:50 |
| TheJulia | Looks like Iury is not around, is he off this week? | 16:57 |
| TheJulia | Sandzwerg[m]: One of the most powerful paths you can help to build mutual understanding of that issue is to attach a screen recorder to the virtual console, like a desktop screen recording app, and just record the boot sequence as like a webm file you can share with folks. Sometimes one or two of the frames hold the secrets. :) | 16:58 |
| TheJulia | (This is like the #1 secret.) | 16:59 |
| Sandzwerg[m] | I can try that next. right now I mostly have screenshots, carefully timed. Today was mostly spent with building with dib directly and try if things change. Then I noticed I can pass arguments with "--extra-args" and now I'm back at the ironic-python-agent-builder. Still something seems strange. I tried to build a ISO with the iso-image-create script but I was only able to boot that in a local qemu as bios (and not at all on the | 17:14 |
| Sandzwerg[m] | remoateboard with active secure boot). But I'm not sure if the script is not able to build efi/secure-boot enabled ISOs or if my ramdisk&kernel is somehow broken. | 17:14 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Build ironic-python-agent DIB image with CentOS Stream 10 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/950965 | 17:18 |
| TheJulia | dunno, honestly I'd really try to record the result and share it if you can. It just seems super weird that your having this much pain when we generally don't run into such issues. Could just be the use pattern is slightly different and there are a few different paths. Happy to try and also talk through it at some point this week if you want. | 17:42 |
| Sandzwerg[m] | https://filebin.net/o3f4nqjgnkrkpbth it's about 5mins | 17:43 |
| Sandzwerg[m] | First boot from virtual-media, then manual editing the grub trying again (and waiting a bit to see if something happens) and lastly boot from disk (and older ubuntu) just to confirm that ubuntu and secure boot in general work | 17:44 |
| Sandzwerg[m] | I'd be happy to talk and don't understand either why it's so hard. Judging from the documentation I expected it to take me a day to week to get this running and now I'm (on and off) working on it much longer. | 17:46 |
| opendevreview | Merged openstack/ironic master: Make _by_arch ramdisk uses & validation consistent https://review.opendev.org/c/openstack/ironic/+/941611 | 17:47 |
| opendevreview | Merged openstack/ironic master: Fix names of steps in comments as per function https://review.opendev.org/c/openstack/ironic/+/953960 | 17:48 |
| TheJulia | okay, your grub doesn't support the efi labels, thats great :( | 18:02 |
| TheJulia | Okay, so… hmm | 18:09 |
| TheJulia | So its something with the kernel and the stubbing it seems | 18:09 |
| TheJulia | Sandzwerg[m]: so first, I’d check and confirm that the host is reporting itself in UEFI mode with secure boot being enforced while the ram disk is booting. For what it’s worth, the way the boot sequence works is the UEFI firmware triggers shim as the Microsoft signed artifact. Shim then executes grub. If secure boot is enabled, and *either* of these things fail, then the machine halts. | 18:11 |
| TheJulia | Sandzwerg[m]: You have nomodeset, nofb, and vga=normal on the command line, try removing them and see what you get. The fact your changing the file, makes me think this is more a grub or kernel problem since the machine should be throwing up a “red screen of death” or UEFI exception crash screen | 18:13 |
| Sandzwerg[m] | TheJulia: for ensuring the host is in UEFI and secure boot I showed the last boot from disk, or do you mean that it reports this to ironic? | 19:18 |
| TheJulia | I mean, you can look at the BMC to confirm thats what it thinks the state is | 19:19 |
| TheJulia | Ironic *should* think that is the state, but its also transactional and *can* change if you request the webui to make a change (including of state) | 19:20 |
| Sandzwerg[m] | The BMC reports UEFI and secure boot in "deployed mode" how it's called for dell. But I think that is the default | 19:21 |
| Sandzwerg[m] | As for the command line, the virtual media shouldn't have that on but the boot from HDD might have it. | 19:22 |
| Sandzwerg[m] | This is how I build the esp on a ubuntu 24.04 host: https://paste.opendev.org/show/bhzwDfOGtFohPPNJsGjP/ | 19:23 |
| Sandzwerg[m] | This is a log of a complete build with the ironic-python-agent-builder: https://filebin.net/9p02b1j62a9mgokw | 19:23 |
| Sandzwerg[m] | I included --extra-args "-p shim-signed" which seems to add shim-signed to the resulting ramdisk/kernel but I have the feeling that the inclusion of "-p grub-efi-amd64-signed" -e block-device-efi" doesn't change a thing (at least nothing noticeable) | 19:25 |
| TheJulia | Yeah, are pretty much default | 19:27 |
| TheJulia | I suspect your ramdisk if ine | 19:27 |
| TheJulia | Anyway, need to jump on a call | 19:27 |
| Sandzwerg[m] | The VM in which I built the esp is also in UEFI and secure boot, however it's not based on the ubuntu-noble-server image like the one built by the ipa-builder | 19:27 |
| Sandzwerg[m] | I would next try to use a ubuntu-noble-cloud image to build the esp. | 19:28 |
| Sandzwerg[m] | If someone has a recipe in the form "use these parameters and these distrobution" to get a working secure boot that's also fine for me | 19:29 |
| Sandzwerg[m] | Have fun in your call and thanks for the help so far o/ | 19:29 |
| TheJulia | Sandzwerg[m]: so, fwiw, your ESP is fine, your getting into grub. | 22:02 |
| TheJulia | Sandzwerg[m]: So, I'd try to use the debug option on a separate line to make grub be very verbose about what it is doing: https://www.gnu.org/software/grub/manual/grub/html_node/debug.html#debug. It also seems like you might want to update your base config templates in ironic, see: https://review.opendev.org/c/openstack/ironic/+/933631 | 22:16 |
| TheJulia | Sandzwerg[m]: based on what you've shared, it *really* looks like something is going sideways or is wrong with the kernel in the ramdisk. The whole fact your in grub signals a focus on secure boot and UEFI is sort of a red herring distracting focus. What seems really weird to me is the kernel just doesn't log anything, what you might want to try is to do something like run "ipmitool sol" and connect to the serial over lan | 22:24 |
| TheJulia | console, and then try updating the kernel command line to console=ttyS0. The kernel *should* then log pretty heavily to the serial device which might give clarity as the graphical devices are way down the complexity list in terms of initalizing and if anything is happening before the graphics interface loads in the kernel, then nothing gets written to the screen. | 22:24 |
| TheJulia | cid: we should sync up on maybe somehow stressing/benching the conductor process. I have time to dedicate to it this sprint. | 22:35 |
| opendevreview | cid proposed openstack/ironic master: Replace GreenThreadPoolExecutor in conductor https://review.opendev.org/c/openstack/ironic/+/952939 | 22:42 |
| opendevreview | cid proposed openstack/ironic master: WIP: Set the backend to threading. https://review.opendev.org/c/openstack/ironic/+/953683 | 22:42 |
| TheJulia | cid: rebase ^ | 22:43 |
| cid | TheJulia, I'm available and can always match your schedule. Just let me know. | 22:47 |
| TheJulia | Okay, I can likely chat tomorrow after the board meeting | 22:47 |
| TheJulia | I just also want to be super mindful of your schedule as well | 22:48 |
| cid | ++ | 22:49 |
| cid | Yeah. Tomorrow works. | 22:50 |
| * cid Might need a ping ahead of time ;) | 22:50 | |
| TheJulia | will do! | 22:54 |
| TheJulia | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ba6/openstack/ba675f0120b94d88b296ef6d4cd7ebd7/controller/logs/ironic-bm-logs/node-0_console_2025-07-07-19%3A40%3A55_log.txt -> I'm thinking we might need to cut a release of ironic-python-agent-builder | 23:21 |
| TheJulia | *but* we still end up consuming ~120M once it gets running | 23:23 |
| TheJulia | (based on my current math from the current image) | 23:24 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Redfish Firmware Interface - NIC Support https://review.opendev.org/c/openstack/ironic/+/953394 | 23:27 |
| TheJulia | Anyone have any opinions on setting SystemMaxUse for the journal config? | 23:27 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!