| NobodyCam | anyone happen to know if Ramdisk deploy interface works with nova? | 00:21 |
|---|---|---|
| JayF | I can't forsee a reason why not, but I've not personally tested it | 00:29 |
| *** eandersson15 is now known as eandersson | 02:08 | |
| opendevreview | Pierre Riteau proposed openstack/bifrost stable/2023.1: Fix ansible-lint https://review.opendev.org/c/openstack/bifrost/+/883119 | 03:41 |
| rpittau | good morning ironic! o/ | 06:46 |
| TheJulia | Good night! | 07:55 |
| dtantsur | JayF: I don't think I have any of my lp dashboards still up, but I can look into that. | 08:10 |
| genekuo | stevebaker: DIB_RELEASE=bullseye disk-image-create -t raw -o debian-bullseye -p mdadm vm block-device-efi debian | 08:49 |
| genekuo | I do modify this line to add mdraid1x to grub modules | 08:51 |
| genekuo | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L286 | 08:51 |
| Sandzwerg[m] | Hmmm question regarding the hardware burn-in. Are they executed on a specific state change like managed -> available or also on every cleaning? | 11:28 |
| iurygregory | good morning Ironic | 11:43 |
| dtantsur | Sandzwerg[m]: I think it's designed to run on manual cleaning (manageable -> (clean) -> manageable) | 11:55 |
| Sandzwerg[m] | hmkay from the documentation it's not clear to me when it runs: https://docs.openstack.org/ironic/latest/admin/hardware-burn-in.html | 12:01 |
| JayF | dtantsur: honestly, I was just hoping you had the code still | 13:17 |
| opendevreview | Merged openstack/ironic bugfix/21.0: Handle MissingAttributeError when using OOB inspections to fetch MACs https://review.opendev.org/c/openstack/ironic/+/882526 | 13:17 |
| dtantsur | JayF: I probably have it somewhere on github.. | 13:18 |
| dtantsur | JayF: https://github.com/dtantsur/ironic-bug-dashboard | 13:25 |
| * TheJulia tries to wake up now | 14:16 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 15:26 |
| TheJulia | Sandzwerg[m]: clean steps, look for "Then launch the test with" | 15:39 |
| TheJulia | so manual cleaning, specifically | 15:39 |
| TheJulia | ... or I guess in deploy templates if the burnin steps are so decorated in the agent | 15:40 |
| TheJulia | genekuo: hmm, I guess debian dosn't have signed artifacts for EFI boot? | 15:41 |
| rpittau | bye everyone, see you next week! o/ | 15:52 |
| TheJulia | o/ | 15:58 |
| genekuo | TheJulia: Do you mean secure boot? It's not enabled in my env. | 16:27 |
| TheJulia | well, so the conundrum here is grub-install prevents UEFI loader building in newer versions because the grub maintainers want people to be able to use secure boot | 16:28 |
| TheJulia | which means a signed shim and all | 16:29 |
| TheJulia | (and well, signed efi binaries) | 16:29 |
| TheJulia | but the path missing is kind of telling | 16:29 |
| TheJulia | genekuo: find /boot/EFI might be helpful context wise | 16:29 |
| genekuo | /boot/efi/EFI/debian is missing in the image, but after I ran grub-install on the deploy host it appears. | 16:39 |
| genekuo | Guess that may be the issue | 16:39 |
| TheJulia | yup, that would be it | 16:39 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 16:44 |
| TheJulia | ... I guess it is missing in the original base image? Are there any other folder in /boot/efi/EFI ? | 16:45 |
| TheJulia | s/Are/Is/ | 16:45 |
| JayF | dtantsur: any idea the last python version that worked on? 3.8 appears to be nonworking | 16:45 |
| genekuo | Sorry my mistake, it's in the image. | 16:45 |
| dtantsur | JayF: oh, a great question.. could be even 2.7 | 16:45 |
| JayF | :-| | 16:46 |
| JayF | aight | 16:46 |
| JayF | dtantsur: any config required that you can remember? | 16:46 |
| JayF | dtantsur: it seems to be trying to work w/o any... | 16:46 |
| JayF | dtantsur: tox.ini had basepython=python3 so it's not 2.7 bad | 16:46 |
| dtantsur | my brain is completely blank, sorry | 16:47 |
| JayF | not a problem :) I'll try to poke it until I figure it out | 16:47 |
| NobodyCam | good morning OpenStack Folks | 16:50 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 16:51 |
| TheJulia | o/ NobodyCam | 16:51 |
| NobodyCam | o/ TheJulia | 16:51 |
| genekuo | TheJulia: It seems like efiboot is set to \EFI\BOOT\BOOTX64.EFI instead of \EFI\debian\grubx64.efi | 17:00 |
| TheJulia | okay | 17:01 |
| TheJulia | so... the first value is the expected x86_64 default file path | 17:01 |
| TheJulia | the latter is the vendor preferred path | 17:01 |
| TheJulia | so, I guess the question is does EFI | 17:02 |
| TheJulia | err | 17:02 |
| genekuo | I'm testing if boot works with "\EFI\debian\grubx64.efi" path | 17:02 |
| TheJulia | EFI\BOOT\BOOTX64.EFI has the necessary code embedded | 17:02 |
| genekuo | I guess there some issue with the code (probably referring to the wrong prefix for grub to load). And rerunning grub install fixed the issue. | 17:05 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 17:05 |
| genekuo | Cause after rerunning grub-install the EFI boot file is still EFI\BOOT\BOOTX64.EFI | 17:05 |
| genekuo | Is it possible to let IPA not reboot after deployment for debugging purpose? | 17:11 |
| TheJulia | genekuo: set it to maintenance mode while it is running the deployment | 17:20 |
| TheJulia | preferably *while* it is writing the disk image | 17:20 |
| TheJulia | that will likely block the bootloader from installing though :\ | 17:21 |
| * TheJulia used to do this by short circuiting the conductor intentionally | 17:21 | |
| JayF | firewall off the bmc from ironic during the install ;) | 17:21 |
| JayF | probably not an actual good suggestion, but might get the behavior you seek | 17:22 |
| genekuo | Thanks for the suggestions :) | 17:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 17:57 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Mark BFV job non-voting for now https://review.opendev.org/c/openstack/ironic/+/883296 | 18:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 19:05 |
| TheJulia | I'm feeling good about ^, but my hopes are ~35 minutes from being dashed | 19:06 |
| JayF | fixing bfv jobs after triple OT | 19:20 |
| JayF | impressive :P | 19:20 |
| JayF | I usually pick easy things for the morning of a late hockey 'hangover' LOL | 19:20 |
| TheJulia | closer... | 19:44 |
| TheJulia | https://www.irccloud.com/pastebin/0XMc4F5X/ | 19:44 |
| JayF | 🤞 | 19:47 |
| iurygregory | 🤞 | 19:47 |
| iurygregory | grenade job is still broken right? | 19:54 |
| iurygregory | at least I saw TIMED_OUT in some runs | 19:55 |
| TheJulia | so, looks like we'll need to do some additional checks, we get a different failure now with cinder | 19:55 |
| iurygregory | so I think this is a good sign? | 19:59 |
| TheJulia | kind of | 20:01 |
| TheJulia | err, no, same error | 20:03 |
| TheJulia | err, too many windows | 20:05 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 20:16 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 21:25 |
| genekuo | stevebaker: I did find /boot/efi/debian/grubx64.efi in the image, but it seems to be built incorrectly, probably pointing to the disk when image is built instead of the disk after installation. | 21:37 |
| NobodyCam | any one happen to know if `Baremetal node deploy` will build the pxe cfg file | 21:41 |
| TheJulia | NobodyCam: if your using ramdisk deploy, yes | 21:44 |
| NobodyCam | hummm :( | 21:45 |
| TheJulia | under normal circumstances, if your doing a non-ramdisk deploy, it will populate the config for ramdisk booting | 21:45 |
| TheJulia | err | 21:45 |
| TheJulia | sorry, I mean, under normal non-ramdisk deploys, a configuration is still generated | 21:45 |
| TheJulia | but that config is generally for the agent | 21:45 |
| TheJulia | unless post-deploy | 21:45 |
| TheJulia | NobodyCam: not sure why that would be an issue | 21:48 |
| NobodyCam | oh | 21:55 |
| NobodyCam | I think I see | 21:55 |
| NobodyCam | its the pxe conf file | 22:02 |
| NobodyCam | goto boot_ramdisk | 22:02 |
| NobodyCam | I have no boot_ramdisk stanza :'( | 22:02 |
| TheJulia | what version of ironic? | 22:03 |
| TheJulia | what boot_interface? | 22:03 |
| NobodyCam | ramdisk for interface , ussuri | 22:04 |
| TheJulia | ramdisk is deploy_interface | 22:06 |
| TheJulia | not boot_interface | 22:06 |
| TheJulia | https://github.com/openstack/ironic/blob/stable/ussuri/ironic/drivers/modules/ipxe_config.template#L34 <-- the base template | 22:06 |
| TheJulia | from ussuri | 22:06 |
| TheJulia | for ipxe | 22:06 |
| NobodyCam | :doh | 22:07 |
| NobodyCam | ipxe | 22:08 |
| TheJulia | so unless your carrying your own base pxe template.... | 22:08 |
| NobodyCam | yep | 22:09 |
| NobodyCam | that's it | 22:10 |
| TheJulia | techdebt == fun | 22:10 |
| TheJulia | :( | 22:10 |
| NobodyCam | heheheh | 22:11 |
| * TheJulia wonders.... if the long tempest run is a good sign, or a bad sign | 22:27 | |
| opendevreview | Merged openstack/ironic master: CI: Mark BFV job non-voting for now https://review.opendev.org/c/openstack/ironic/+/883296 | 22:27 |
| JayF | TheJulia: ^ that's the best possible sign you fixed it | 22:27 |
| JayF | lol | 22:27 |
| TheJulia | normally ~45 minutes | 22:28 |
| TheJulia | we're approaching an hour I think | 22:28 |
| TheJulia | but it could just be a slow node | 22:28 |
| TheJulia | tempest run in that one was ~4 minut4s | 22:30 |
| TheJulia | we're at 20 now | 22:30 |
| JayF | I was just making a joke; you'd fix the job shortly after the -nv lands | 22:31 |
| TheJulia | in accordance with the prophecy | 22:33 |
| genekuo | stevebaker: here's what I get from the working debian nodes | 22:50 |
| genekuo | https://paste.opendev.org/show/bBpuoXsXZiUExWA4j0qt/ | 22:50 |
| TheJulia | and it finally exited tempest | 22:50 |
| TheJulia | genekuo: are they the same file? | 22:50 |
| genekuo | I believe the EFI application is pointing to the wrong disk causing the issue | 22:50 |
| TheJulia | it depends on what the grub binaries get built with actually | 22:51 |
| TheJulia | (These are the most fun issues, btw) | 22:51 |
| TheJulia | ((I'm being sarcastic) | 22:51 |
| TheJulia | ) | 22:51 |
| genekuo | TheJulia: They are different, currently the boot option is using BOOTX64.efi | 22:52 |
| TheJulia | is shim an installed package? | 22:53 |
| genekuo | No, it's not | 22:56 |
| TheJulia | I guess that makes sense | 22:57 |
| TheJulia | zigo: does debian not have a stock signed binary? is it always built? | 23:00 |
| TheJulia | https://packages.debian.org/bullseye/amd64/grub-efi-amd64-bin/filelist shows some monolithic binaries | 23:01 |
| TheJulia | Anyway, regardless, the standard pattern is to load grub.cfg from the same file as the binary | 23:03 |
| TheJulia | we lean towards BOOTX64.EFI because it is standardized | 23:03 |
| TheJulia | and if there is no loader csv hint file, we would end up with that as the default. | 23:03 |
| genekuo | I don't think chain loading is the issue though, it seems like the efi file is loading grub from a unknown disk | 23:06 |
| genekuo | Here's the different I found in BOOTX64.efi from extracting strings from binary | 23:06 |
| genekuo | https://paste.opendev.org/show/belHDo8rTVax0DgwpdEc/ | 23:06 |
| TheJulia | okay, then it is the stock binary doesn't have what is needed | 23:12 |
| TheJulia | so the incoming image needs to have the appropriate bits then to find it | 23:12 |
| genekuo | stevebaker: Both BOOTX64.efi and grubx64.efi is set to look up grub from /boot/grub | 23:28 |
| genekuo | I'm not sure if this is the behavior when not enabling secure boot | 23:28 |
| genekuo | Other nodes not using software RAID also don't have grub.cfg in /boot/efi/EFI/debian | 23:32 |
| genekuo | They are located in /boot/grub | 23:32 |
| TheJulia | well, at least it is detectable, I guess | 23:37 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 23:39 |
| genekuo | stevebaker: I see, (,gpt3)/boot/grub seems to work without software RAID, but with software RAID it might have to specify mduuid. I reach out to debian community and see if this is the expected behavior | 23:45 |
| TheJulia | interesting... I'm not seeing stevebaker's comments in irccloud | 23:47 |
| genekuo | It might because the message is too long | 23:49 |
| TheJulia | nah, dunno though. | 23:51 |
| TheJulia | I'm out of spoons for the day | 23:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!