| janders | I tried some downstream doco which works a treat but it also produces disk+kernel+ramdisk trio, which doesn't look like whole-disk images | 00:00 |
|---|---|---|
| TheJulia | elements are horribly named | 00:00 |
| stevebaker | TheJulia: ok I'm ready. janders do you want to join on a call too, I'll talk about building images | 00:01 |
| TheJulia | vm grub2 is the apth | 00:01 |
| TheJulia | path | 00:01 |
| janders | stevebaker yes, that would be great | 00:01 |
| TheJulia | start a meet, I'll jump in just going to get something to drink | 00:01 |
| stevebaker | I've sent an invite | 00:02 |
| *** tosky has quit IRC | 00:11 | |
| *** yoctozepto has quit IRC | 00:13 | |
| *** yoctozepto has joined #openstack-ironic | 00:13 | |
| *** gryf has quit IRC | 00:18 | |
| *** gryf has joined #openstack-ironic | 00:21 | |
| *** jamesdenton has quit IRC | 00:46 | |
| *** jamesden_ has joined #openstack-ironic | 00:46 | |
| *** rloo has quit IRC | 00:51 | |
| janders | stevebaker "disk-image-create centos grub2 block-device-efi -o centos-image" produced a whole-disk image with broken GRUB (it boots with GRUB CLI). I was able to boot C8 from the GRUB CLI by manually specifying kernel/rootfs/ramdisk. Is this the issue you were seeing as well? | 01:56 |
| stevebaker | janders: do you mean with UEFI boot it put you into a blank grub prompt? I have to manually run efibootmgr to even get that far | 02:01 |
| janders | stevebaker correct | 02:02 |
| janders | I had to force the node to boot off one of the NVMes instead of PXE before that but this is probably because ironic doesn't have a local_boot flag set on it | 02:04 |
| janders | do you know if I need to do this with true-baremetal on Bifrost (I remember doing this on underclouds) | 02:04 |
| *** k_mouza has joined #openstack-ironic | 02:21 | |
| *** k_mouza has quit IRC | 02:26 | |
| stevebaker | janders: what is supposed to happen is ipa runs efibootmgr to set the boot order for local-boot first, so when it reboots it should boot to your efi setup. However if that entry is broken it will fall through to the next boot option (PXE boot) | 02:27 |
| janders | stevebaker do I need somethinglike boot_mode=local equivalent for that to happen? | 02:28 |
| janders | (calling it equivalent cause I suppose that's a BIOS construct not UEFI) | 02:29 |
| stevebaker | janders: I think ironic switches to local as part of the deploy process | 02:29 |
| janders | stevebaker http://paste.openstack.org/show/803246/ here is the "efibootmgr -v" output from the deployed node | 02:30 |
| janders | I'm not quite sure what the first device on the list is - the node has been cleaned prior to deploy so I'd expect to only see CentOS and PXE entries | 02:31 |
| janders | I will poke around checking what the disks look like | 02:31 |
| stevebaker | janders: all of those were likely auto-discovered by UEFI, the one created by ipa will be called 'ironic1' | 02:32 |
| stevebaker | janders: this is my deploy script http://paste.openstack.org/show/803247/ (note the --property capabilities=boot_mode:uefi) | 02:33 |
| janders | stevebaker hah I was about to ask about this one | 02:33 |
| stevebaker | janders: you'll need that for ipa to run efibootmgr | 02:33 |
| janders | stevebaker right! thanks for that. I will set it and have another go | 02:34 |
| janders | stevebaker so after that is set I should not have to interact with the boot order to have the server boot off local storage, correct? | 02:35 |
| janders | shameless plug: I love the new NVMe cleaning feature for testing things like this :) no need to worry if metadata_erase left something behind | 02:40 |
| janders | (if the node is running NVMes and only NVMes that is) | 02:41 |
| stevebaker | janders: if its working correctly you won't need manual intervention for local boot to be the first option | 02:48 |
| *** irclogbot_0 has quit IRC | 02:52 | |
| janders | stevebaker thank you. That worked. Now I'm back to the GRUB screen. Will hand-boot the node to 1) see if this is repeatable and 2) if it's booted maybe we can get some interesting info out of this | 02:52 |
| janders | yep, it worked the same way as the last time | 02:54 |
| *** irclogbot_2 has joined #openstack-ironic | 02:55 | |
| janders | http://paste.openstack.org/show/803248/ looks like it's missing GRUB config | 03:00 |
| janders | I will copy it in and see if it starts booting normally | 03:00 |
| janders | possibly something missing in the image? | 03:01 |
| janders | copying grub.conf into the path where it is on Fedora didn't help.. will hand-boot it and try reinstalling grub and see if there is any interesting output | 03:03 |
| stevebaker | janders: yeah, I'm working on a dib change to generate that now | 03:03 |
| janders | stevebaker awesome I will stop making noise and leave you to it; when it's ready I'm happy to help with testing :) | 03:04 |
| stevebaker | janders: having a tester will be helpful | 03:04 |
| janders | stevebaker any time | 03:05 |
| *** gyee has quit IRC | 03:08 | |
| janders | stevebaker "grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg" fixed it on my node | 03:12 |
| janders | apparently the config that was sitting around somewhere in /boot/ wasn't right | 03:13 |
| janders | wild guess: maybe something UEFI related prevents something in IPA from running ^^^ (I may be stating the obvious) | 03:13 |
| janders | (or something in DIB) | 03:14 |
| janders | breaking for late lunch now | 03:14 |
| stevebaker | janders: /boot/grub2/grub.cfg is generated by dib, I'm first going to try generating an identical one for /boot/efi/EFI/centos/grub.cfg | 03:15 |
| stevebaker | but possibly they should be different, we'll see | 03:17 |
| *** uzumaki has quit IRC | 03:59 | |
| *** k_mouza has joined #openstack-ironic | 03:59 | |
| *** k_mouza has quit IRC | 04:04 | |
| *** uzumaki has joined #openstack-ironic | 04:23 | |
| *** jamesden_ has quit IRC | 04:25 | |
| *** jamesdenton has joined #openstack-ironic | 04:25 | |
| *** mkrai has joined #openstack-ironic | 04:27 | |
| *** uzumaki has quit IRC | 04:28 | |
| *** uzumaki has joined #openstack-ironic | 04:57 | |
| *** uzumaki has quit IRC | 05:03 | |
| janders | hey stevebaker are you still around (I know it's late in your TZ)? I found something that might be of interest | 05:08 |
| janders | stevebaker this looks weird https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/bootloader/finalise.d/50-bootloader#L163 - first - setting GRUB_OPTS="--target=x86_64-efi" and then running with "--target=i386-pc" anyway... | 05:10 |
| *** rh-jelabarre has quit IRC | 05:26 | |
| stevebaker | janders: that GRUB_OPTS is only used in the second grub2-install call, and that will go away in my next change | 05:47 |
| janders | stevebaker ACK | 05:47 |
| *** openstackgerrit has joined #openstack-ironic | 05:47 | |
| openstackgerrit | sonali bhausaheb borkar proposed openstack/ironic master: DRAC : idrac-redfish inspect updates pxe port https://review.opendev.org/c/openstack/ironic/+/693328 | 05:47 |
| janders | stevebaker I made some other interesting observations... testing, should have more info shortly | 05:48 |
| stevebaker | janders: cool. My current challange is my PC arbitrarily screwing with the boot order. | 05:49 |
| janders | stevebaker I have a node we can test on if you like | 05:50 |
| stevebaker | that'll come in handy later | 05:51 |
| janders | stevebaker I/R/T I deleted all the grub.cfg files anywhere in /boot (or on EFI partition). I then copied the one I re-generated post-deploy to /boot/grub2/grub.cfg. Ended up in GRUB shell post reboot. So it does matter where it goes. Will now move it to /boot/efi/EFI/centos/grub.cfg and see how that goes | 05:51 |
| stevebaker | janders: the original /boot/grub2/grub.cfg would have been generated by dib and will be the one used by legacy boot. Having the same file written to /boot/efi/EFI/centos/grub.cfg is working for me I think | 05:53 |
| janders | stevebaker I'm not sure if the DIB-generated one has the right content. About to test that too. But now, the box stopped booting even with the re-generated config in/boot/efi/EFI/centos/grub.cfg... hmm | 05:54 |
| janders | (first I will make sure I got all the paths right) | 05:55 |
| stevebaker | I'm suspicious of these in the bootloader element: | 05:55 |
| stevebaker | echo "GRUB_DEVICE=LABEL=${DIB_ROOT_LABEL}" | 05:55 |
| stevebaker | echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub | 05:55 |
| stevebaker | for the efi case | 05:55 |
| stevebaker | anyway, I'm off. good evening | 05:57 |
| janders | have a great weekend stevebaker | 05:57 |
| janders | I will poke around a bit more and message you with what I found if I find anything interesting | 05:58 |
| janders | thanks for all your help today stevebaker, I learned a _lot_ from you and TheJulia | 05:58 |
| *** anuradha1904 has joined #openstack-ironic | 05:59 | |
| *** rcernin has quit IRC | 06:00 | |
| *** tzumainn has quit IRC | 06:16 | |
| *** moshiur has joined #openstack-ironic | 06:18 | |
| janders | stevebaker to summarise my testing: http://paste.openstack.org/show/803251/ is a DIB generated /boot/grub2/grub.cfg. With this, the system only boots to GRUB CLI. http://paste.openstack.org/show/803252/ is cat /boot/efi/EFI/centos/grub.cfg generated post-deploy with grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg. This makes the system boot | 06:29 |
| janders | normally. After that is in place, all the other grub config files in /boot can be deleted and the system doesn't care. This seems to be file we need to work with for the EFI installs. | 06:29 |
| janders | wrapping this up as well, we shall continue next week! :) | 06:32 |
| janders | (will likely drop in for the SPUC later :) | 06:32 |
| *** bnemec has quit IRC | 06:43 | |
| janders | stevebaker also: copying content from /boot/grub2/grub.cfg to /boot/efi/EFI/centos/grub.cfg so there is something not right with the content of the file as well it seems. Needs to be in /boot/efi/EFI/centos/grub.cfg AND have something along the lines of http://paste.openstack.org/show/803252/ inside | 06:43 |
| janders | * copying content from /boot/grub2/grub.cfg to /boot/efi/EFI/centos/grub.cfg DOESN'T WORK | 06:44 |
| *** bnemec has joined #openstack-ironic | 06:44 | |
| *** kashifkhan has joined #openstack-ironic | 06:44 | |
| kashifkhan | Good morning, I have a question, if the node reboots in maintenance mood with fastrack enabled, is it supposed to reboot again when we take it out of maintenance mood and do the provision. Is there anyway to avoid this double reboot? | 06:46 |
| *** uzumaki has joined #openstack-ironic | 07:08 | |
| *** uzumaki has quit IRC | 07:10 | |
| openstackgerrit | Moshiur Rahman proposed openstack/ironic-python-agent-builder master: Fix: IPA image building with OpenSuse. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778726 | 07:30 |
| arne_wiebalck | Good morning, ironic! | 07:36 |
| *** mkrai has quit IRC | 07:43 | |
| *** frigo has joined #openstack-ironic | 07:46 | |
| *** alexmcleod has quit IRC | 07:52 | |
| *** zzzeek has quit IRC | 07:52 | |
| *** zzzeek has joined #openstack-ironic | 07:53 | |
| *** mkrai has joined #openstack-ironic | 07:56 | |
| *** zzzeek has quit IRC | 08:08 | |
| *** zzzeek has joined #openstack-ironic | 08:09 | |
| *** mkrai has quit IRC | 08:18 | |
| *** rpittau|afk is now known as rpittau | 08:21 | |
| rpittau | good morning ironic! o/ | 08:21 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Do not build iso for ussuri ipa https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778620 | 08:23 |
| *** zzzeek has quit IRC | 08:40 | |
| *** zzzeek has joined #openstack-ironic | 08:41 | |
| *** dking has left #openstack-ironic | 08:41 | |
| *** zzzeek has quit IRC | 08:42 | |
| *** zzzeek has joined #openstack-ironic | 08:43 | |
| *** ociuhandu has joined #openstack-ironic | 08:44 | |
| *** dougsz has joined #openstack-ironic | 08:51 | |
| *** frigo has quit IRC | 09:00 | |
| rpittau | if anyone has a minute please check https://review.opendev.org/c/openstack/ironic-lib/+/778714 , the test looks fine https://review.opendev.org/c/openstack/ironic/+/778718 | 09:01 |
| *** lucasagomes has joined #openstack-ironic | 09:04 | |
| janders | good morning arne_wiebalck rpittau | 09:07 |
| rpittau | hey janders :) | 09:07 |
| janders | arne_wiebalck rpittau do you guys have much experience with baremetal UEFI wholedisk ironic images (a mouthful I know)? | 09:08 |
| *** derekh has joined #openstack-ironic | 09:14 | |
| *** tosky has joined #openstack-ironic | 09:23 | |
| rpittau | janders: anything in particular on that ? | 09:33 |
| janders | rpittau GRUB2 issues - DIB seems to build images with half broken GRUB (at least for CentOS8) | 09:34 |
| janders | symptoms: instance deploys okay after which it gets stuck on GRUB CLI and if you know the magic (key) words you can semi-manually boot the OS from there | 09:35 |
| janders | looks like 1) GRUB config isn't quite right and 2) it's in the wrong place in the first place | 09:35 |
| rpittau | janders: ah so image building issue, maybe we can check the dib code and see how that is done ? could be a bug there or simply not up to date | 09:35 |
| janders | rpittau - very possible; stevebaker is looking into this; but was wondering if any of us has successfully worked around that (might make the fix easier) | 09:36 |
| kashifkhan | Does anyone have any clue on the question I posted earlier today? | 09:42 |
| *** k_mouza has joined #openstack-ironic | 09:42 | |
| openstackgerrit | Moshiur Rahman proposed openstack/ironic-python-agent-builder master: Fix: IPA image building with OpenSuse. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778726 | 09:44 |
| *** eric_lei has joined #openstack-ironic | 09:50 | |
| eric_lei | hi | 09:52 |
| ajya | janders: regarding GRUB - we just noticed it this week - that virtual media boot does not work anymore - stuck at GRUB CLI | 09:56 |
| ajya | janders: but that's in devstack environment that has some changes to efiboot script, reverting those changes help, but haven't looked further | 09:57 |
| *** eric_lei has quit IRC | 09:58 | |
| janders | thanks ajya | 10:01 |
| janders | SPUC anyone? | 10:01 |
| janders | (/me is looking up link) | 10:01 |
| janders | https://redhat.bluejeans.com/772893798 | 10:02 |
| ajya | janders: that's also at booting IPA, so different, no idea if related, coincidence maybe :) | 10:02 |
| janders | ajya hmm same issue with IPA... didn't see this one coming. UEFI works (or does not work) in mysterious ways... :) | 10:05 |
| janders | rpittau arne_wiebalck feel like dropping into the SPUC? | 10:06 |
| rpittau | janders: sorry I'm a bit busy this morning, I don't think I will be able to make it :/ | 10:06 |
| janders | rpittau no worries :) | 10:07 |
| *** ericlei has joined #openstack-ironic | 10:12 | |
| ericlei | hi, has anyone provision a baremetal use ipxe when neutron integrate with ovn? | 10:14 |
| arne_wiebalck | janders: sorry, in meetings | 10:19 |
| openstackgerrit | Rotan proposed openstack/ironic-inspector master: [WIP]Support AEP, sst at introspection data https://review.opendev.org/c/openstack/ironic-inspector/+/768138 | 10:27 |
| arne_wiebalck | janders: for the UEFI WDIs: yes, this is what we use here now | 10:29 |
| arne_wiebalck | janders: I would not be able to list the correct location, I remember there are some links involved :-S | 10:32 |
| janders | arne_wiebalck do you have any publicly shareable doco? :) it would be handy to see if we're hitting the same issues | 10:32 |
| arne_wiebalck | janders: I am scanning what I wrote in https://techblog.web.cern.ch/techblog/post/bios_uefi_cloud_image/ | 10:33 |
| arne_wiebalck | janders: there is some GRUB magic we needed to do | 10:34 |
| arne_wiebalck | ericlei: I have not, maybe TheJulia can help out (she will probably not be online before a couple of hours, though) | 10:36 |
| janders | arne_wiebalck thank you, this is awesome! I know what I'll be reading Monday morning | 10:36 |
| arne_wiebalck | janders: :) | 10:37 |
| ericlei | arne_wiebalck: thx | 10:37 |
| *** alexmcleod has joined #openstack-ironic | 10:39 | |
| arne_wiebalck | kashifkhan: also for this, TheJulia is the expert (she implemented this feature), but can you already say in which state you start and what triggers the first reboot? | 10:39 |
| janders | see you tomorrow Ironic o/ | 11:01 |
| janders | have a good weekend everyone | 11:01 |
| arne_wiebalck | bye janders o/ | 11:06 |
| *** k_mouza has quit IRC | 11:10 | |
| iurygregory | good morning Ironic o/ | 11:10 |
| iurygregory | bye janders o/ | 11:10 |
| kashifkhan | arne_wiebalck: First I put the node in maintenance mode. Then I do a reboot manually. Then when I unset the maintenance flag and try a provision, ironic reboots the node again. I was hoping to have some flag which would detect a reboot isnt required. | 11:10 |
| *** ociuhandu has quit IRC | 11:16 | |
| *** ociuhandu has joined #openstack-ironic | 11:17 | |
| *** ericlei has left #openstack-ironic | 11:17 | |
| *** ociuhandu has quit IRC | 11:17 | |
| *** ociuhandu has joined #openstack-ironic | 11:18 | |
| *** k_mouza has joined #openstack-ironic | 11:22 | |
| openstackgerrit | sonali bhausaheb borkar proposed openstack/ironic master: DRAC : idrac-redfish inspect updates pxe port https://review.opendev.org/c/openstack/ironic/+/693328 | 11:23 |
| *** lucasagomes has quit IRC | 11:24 | |
| *** lucasagomes has joined #openstack-ironic | 11:28 | |
| *** ociuhandu has quit IRC | 11:29 | |
| *** ociuhandu has joined #openstack-ironic | 11:31 | |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Revert "Update iDRAC doc with missing interfaces" https://review.opendev.org/c/openstack/ironic/+/778835 | 11:35 |
| *** frigo has joined #openstack-ironic | 11:36 | |
| frigo | helllo Ironic!! just to report, Im using DIB to build IPA ramdisks and mount them with bifrost as virtual media, on UEFI systems. It just works:) for the base image I'm trying to customize the rhel qcow2 and see if I can put all the drivers I need inside | 11:39 |
| frigo | (in fact I am not trying anymore, the whole thing just works out of the box) | 11:41 |
| iurygregory | great news =D | 11:42 |
| frigo | haha yes, thanks for the great product :) | 11:42 |
| iurygregory | rpittau, https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778620 =O green | 11:44 |
| iurygregory | when: "'ussuri' not in ipa_branch_path" (nice approach!) | 11:45 |
| arne_wiebalck | kashifkhan: in which state is the node at the very beginning? | 11:46 |
| kashifkhan | arne_wiebalck: I was simulating a use case scenario. The nodes maintenance state was unset. It was in manageable state. I set the maintenance flag manually | 11:49 |
| arne_wiebalck | kashifkhan: ok, so the node starts in 'manageable' | 11:50 |
| arne_wiebalck | kashifkhan: and the end state you'd like to have it in is 'available' or 'active' (i.e. with an instance)? | 11:51 |
| arne_wiebalck | kashifkhan: oh, and the node is off at the beginning, correct? | 11:53 |
| *** ociuhandu has quit IRC | 11:54 | |
| *** ociuhandu has joined #openstack-ironic | 11:56 | |
| kashifkhan | arne_wiebalck: the fast track is enabled | 11:56 |
| arne_wiebalck | kashifkhan: fasttrack allows to avoid reboots between provisioning steps, so inspection/cleaning/deploy can be done without a reboot | 11:58 |
| arne_wiebalck | kashifkhan: without a reboot in between | 11:59 |
| arne_wiebalck | kashifkhan: there is still a reboot at the very beginning (to boot into the IPA image) and at the very end (to boot the user image) | 12:00 |
| *** ociuhandu has quit IRC | 12:00 | |
| kashifkhan | arne_wiebalck: ok I guess I didnt explain my use case better, the node was inspected and in ready state with fasttrack enabled, the I set the maintenance flag, I do a manual reboot of the node, then I unset the maintenance flag, start provisioning and then I experience another reboot. I want to avoid the second reboot if its possible | 12:01 |
| kashifkhan | arne_wiebalck: I want an unexpected reboot when the node is in "available" to not trigger a second reboot when we start provisioning. | 12:02 |
| arne_wiebalck | kashifkhan: ok, so at the beginning the node is powered on, in 'manageable' state, has just been inspected, is running an image with the IPA and you would like it to move to 'available' with rebooting ... does this describe it correctly? | 12:05 |
| arne_wiebalck | without rebooting of course | 12:06 |
| kashifkhan | arne_wiebalck: This is what I am experiencing: provisioned -> cleaning -> available -> reboot (outside of Ironic's control) -> available -> reboot -> provisioning -> reboot -> active | 12:07 |
| kashifkhan | arne_wiebalck: This is what I am expecting: provisioned -> cleaning -> available -> reboot (outside of Ironic's control) -> available -> provisioning -> reboot -> active | 12:08 |
| kashifkhan | I want to know if its possible and if yes how should I configure | 12:09 |
| arne_wiebalck | kashifkhan: "provisioned" at the beginning is equal to "active" at the end, right? | 12:09 |
| arne_wiebalck | kashifkhan: there is no way to avoid the reboot right before "active": the node needs to boot into the image you just deployed | 12:10 |
| arne_wiebalck | kashifkhan: ok, so basically you would like to deploy right after cleaning | 12:11 |
| kashifkhan | <arne_wiebalck> : sorry provisioned= active and no I do not want to avoid the reboot before active. Please see the expectation. I want to avoid the reboot before provisioning. | 12:11 |
| arne_wiebalck | kashifkhan: yep, this is what I just concluded :) | 12:12 |
| arne_wiebalck | kashifkhan: after cleaning == before deployment (provisioning) | 12:13 |
| *** derekh has quit IRC | 12:13 | |
| arne_wiebalck | kashifkhan: I don't think this is possible. I have suggested/asked this as well :) | 12:14 |
| arne_wiebalck | kashifkhan: My use case is an immediate re-use by the same user. | 12:14 |
| arne_wiebalck | kashifkhan: At the moment, the user needs to wait for the cleaning to complete, then the node goes down, then we need to wait for the allocation candidate to appear (which the user cannot see), only then we can redeploy. | 12:15 |
| kashifkhan | arne_wiebalck: any idea if IPA is already running, why do we need to reboot IPA again | 12:16 |
| *** k_mouza has quit IRC | 12:16 | |
| arne_wiebalck | kashifkhan: I don't think this is technically necessary. | 12:17 |
| arne_wiebalck | kashifkhan: Just as it is not necessary after inspection in fasttrack. | 12:17 |
| arne_wiebalck | kashifkhan: However, the entry point is different. | 12:17 |
| kashifkhan | arne_wiebalck: I understand, so the conclusion is its not possible at this moment. | 12:18 |
| arne_wiebalck | kashifkhan: What you describe (and I would like as well) is re-deployment after auto-cleaning. | 12:18 |
| arne_wiebalck | kashifkhan: What fasttrack does is deployment after inspection/cleaning, at least from what I understand. | 12:18 |
| arne_wiebalck | kashifkhan: I think it is not, and I think I asked exactly this when fasttrack was added. But we should check with TheJulia to be sure. | 12:19 |
| openstackgerrit | Rotan proposed openstack/ironic-inspector master: [WIP]Support AEP, sst at introspection data https://review.opendev.org/c/openstack/ironic-inspector/+/768138 | 12:20 |
| arne_wiebalck | kashifkhan: Apologies for taking a moment to understand your case :) | 12:20 |
| kashifkhan | arne_wiebalck: No no, its good that you asked and now that its clear my question would be if it would be possible to add/implement this as well for the sake of making the fasttrack work more efficiently. | 12:22 |
| kashifkhan | arne_wiebalck: Thanks a lot for your answers | 12:23 |
| arne_wiebalck | kashifkhan: I think there was a reason (which I cannot remember) why this is not there ... TheJulia to the rescue! | 12:23 |
| TheJulia | kashifkhan: so the first in maintenance reboot causes the token to be lost, so heartbeats stop getting recorded, and then thus the machine reboots. It iis because your first reboot is unknown to ironic and breaks the security mechanism around the agent token | 12:29 |
| * TheJulia goes and makes coffee | 12:29 | |
| kashifkhan | TheJulia: Thanks. if I wouldn't have done the reboot in maintenance, would it work then? | 12:31 |
| TheJulia | It should | 12:32 |
| TheJulia | I would consider it a bug if it didn’t | 12:33 |
| kashifkhan | TheJulia: did I understand you correctly, it should, even if Ironic is not aware of the reboot at all ? | 12:34 |
| * TheJulia listens to the coffee maker while the feline and corgi overlords continue to sleep | 12:34 | |
| *** tkajinam has quit IRC | 12:35 | |
| *** tkajinam has joined #openstack-ironic | 12:35 | |
| TheJulia | So if you reboot a host with a running agent by other means, then the agent token cannot be retrieved again on reboot | 12:35 |
| TheJulia | That effectively locks out communication with the agent | 12:35 |
| TheJulia | Forcing ironic on deploy to reboot the machine again | 12:36 |
| TheJulia | Tokens can be only gotten once | 12:36 |
| TheJulia | Well, only once unless your using virtual media | 12:37 |
| TheJulia | It that is the exception | 12:37 |
| openstackgerrit | Merged openstack/ironic stable/victoria: Always retry locking when performing task handoff https://review.opendev.org/c/openstack/ironic/+/778241 | 12:38 |
| kashifkhan | TheJulia: thanks a lot. That was clear. is there anything we can do to make it so that Ironic is ready to re-negociate the token? | 12:38 |
| TheJulia | 1 not put the machine in maintenance mode | 12:39 |
| TheJulia | 2) if you reboot it manually via Ironic’s api, that should wipe the token and allow regeneration | 12:40 |
| TheJulia | Note I said should. I seem to remember someone patched that last year | 12:40 |
| kashifkhan | TheJulia: Thanks a lot. I ll do some tests again specially avoiding the maintenance mode and come back again if something is unclear. Thanks again | 12:41 |
| TheJulia | Good luck! | 12:42 |
| iurygregory | good morning TheJulia | 12:45 |
| TheJulia | Good morning | 12:46 |
| *** rloo has joined #openstack-ironic | 12:48 | |
| TheJulia | Mmmm coffee | 12:50 |
| iurygregory | brains with coffee | 12:51 |
| iurygregory | :D | 12:51 |
| iurygregory | coffee brains (beans) :D | 12:52 |
| *** k_mouza has joined #openstack-ironic | 12:52 | |
| *** k_mouza has quit IRC | 12:52 | |
| TheJulia | Heh | 12:52 |
| *** k_mouza has joined #openstack-ironic | 12:53 | |
| arne_wiebalck | Do we have a mechanism to run sth right before or after inspection, on the host? Use case is registration of a auto-discovering node with some external service. | 12:57 |
| iurygregory | not that I'm aware .-., the use case sounds interesting =) | 13:02 |
| openstackgerrit | Merged openstack/ironic master: Port/Portgroup project scoped access https://review.opendev.org/c/openstack/ironic/+/775465 | 13:05 |
| *** ociuhandu has joined #openstack-ironic | 13:06 | |
| rpittau | iurygregory: re https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778620 yep hopefully will work on ussuri as it does on master :) | 13:07 |
| rpittau | not sure how to test that, like a depends-on probably doesn't work | 13:07 |
| *** jamesdenton has quit IRC | 13:18 | |
| *** jamesden_ has joined #openstack-ironic | 13:18 | |
| openstackgerrit | Merged openstack/ironic-lib master: Fixes parsing blkid output with iSCSI devices https://review.opendev.org/c/openstack/ironic-lib/+/778714 | 13:23 |
| *** frigo has quit IRC | 13:24 | |
| *** rh-jelabarre has joined #openstack-ironic | 13:25 | |
| *** ociuhandu has quit IRC | 13:26 | |
| TheJulia | arne_wiebalck: also not that i'm aware of but it could be a small plugin in inspector I think (well, something which could live in tree | 13:26 |
| *** ociuhandu has joined #openstack-ironic | 13:26 | |
| arne_wiebalck | TheJulia: the other option would be to put it into the node_not_found_hook | 13:26 |
| arne_wiebalck | TheJulia: but this is already more specific | 13:26 |
| arne_wiebalck | TheJulia: I put it on the list | 13:27 |
| *** moshiur has quit IRC | 13:28 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Enforce autospec in test_portgroup https://review.opendev.org/c/openstack/ironic/+/778914 | 13:30 |
| arne_wiebalck | TheJulia: do we have examples of such a "execute this script" hook ? | 13:30 |
| TheJulia | arne_wiebalck: I was thinking just put it in to the node_not_found_hook | 13:31 |
| TheJulia | or make a similar node_not_found_enroll_execute_command | 13:31 |
| *** ociuhandu has quit IRC | 13:31 | |
| arne_wiebalck | TheJulia: this would prevent it from running when the node is known ... not sure this is needed, but it seems less generic | 13:32 |
| *** ayoung has quit IRC | 13:32 | |
| arne_wiebalck | TheJulia: also, all the info I need for the call would need to be transported from the node to the hook | 13:33 |
| arne_wiebalck | TheJulia: also also, there might be calls which are not allowed on behalf of the node (due to some reverse lookup or similar) | 13:34 |
| TheJulia | what do you mean might not be allowed on behalf of the node? | 13:39 |
| iurygregory | rpittau, TheJulia we will probably need 4.6.1 in https://opendev.org/openstack/releases/src/branch/master/deliverables/wallaby/ironic-lib.yaml | 14:00 |
| iurygregory | I'm going to push a patch since we merged https://review.opendev.org/c/openstack/ironic-lib/+/778714 | 14:01 |
| TheJulia | iurygregory: ack | 14:11 |
| TheJulia | yes please | 14:12 |
| TheJulia | also note that it is a release to fix it | 14:12 |
| iurygregory | yeah | 14:13 |
| iurygregory | from 4.6.0 it will go to 4.6.1 | 14:13 |
| iurygregory | not sure if we will need to blacklist 4.6.0 in requirements... | 14:14 |
| *** ociuhandu has joined #openstack-ironic | 14:16 | |
| iurygregory | https://review.opendev.org/c/openstack/releases/+/778934 done | 14:17 |
| arne_wiebalck | TheJulia: for instance, the server side of the external service may not allow the controller to register the node (as the request needs to come from the same IP as the one you would like to register) | 14:17 |
| TheJulia | iurygregory: likely, if you can propose a patch to requirements that would be good | 14:18 |
| iurygregory | TheJulia, doing now =) | 14:18 |
| TheJulia | \o/ | 14:19 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Centralize variables https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778942 | 14:29 |
| *** ociuhandu has quit IRC | 14:33 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Increment API version for Secure RBAC https://review.opendev.org/c/openstack/ironic/+/778947 | 14:36 |
| *** ociuhandu has joined #openstack-ironic | 14:39 | |
| *** tzumainn has joined #openstack-ironic | 14:51 | |
| *** frigo has joined #openstack-ironic | 14:59 | |
| *** jamesden_ is now known as jamesdenton | 15:03 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP: Scoped RBAC testing https://review.opendev.org/c/openstack/ironic/+/778957 | 15:03 |
| rpittau | bye everyone! have a great weekend! o/ | 15:03 |
| *** rpittau is now known as rpittau|afk | 15:03 | |
| TheJulia | o/ | 15:04 |
| *** mkrai has joined #openstack-ironic | 15:12 | |
| *** ociuhandu has quit IRC | 15:29 | |
| *** ociuhandu has joined #openstack-ironic | 15:33 | |
| TheJulia | tzumainn: I think https://review.opendev.org/c/openstack/ironic/+/775681 needs a release note | 15:33 |
| tzumainn | TheJulia, ah, good point - I'll add one in! | 15:36 |
| *** ociuhandu has quit IRC | 15:36 | |
| *** ociuhandu has joined #openstack-ironic | 15:36 | |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/777434 just needs a quick doc/source/admin/somethingsomething entry | 15:39 |
| TheJulia | If anyone would like to see old vif style attachments go away, please review https://review.opendev.org/c/openstack/ironic/+/767432 | 15:41 |
| *** anuradha1904 has quit IRC | 15:55 | |
| *** frigo has quit IRC | 15:58 | |
| *** frigo has joined #openstack-ironic | 16:06 | |
| arne_wiebalck | bye everyone, have a good week-end o/ | 16:08 |
| *** frigo has quit IRC | 16:11 | |
| *** mkrai has quit IRC | 16:29 | |
| *** mkrai_ has joined #openstack-ironic | 16:29 | |
| *** ociuhandu has quit IRC | 16:29 | |
| TheJulia | \o | 16:30 |
| *** ociuhandu has joined #openstack-ironic | 16:32 | |
| *** ociuhandu has quit IRC | 16:36 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP: Scoped RBAC testing https://review.opendev.org/c/openstack/ironic/+/778957 | 16:36 |
| *** ociuhandu has joined #openstack-ironic | 16:42 | |
| *** ayoung has joined #openstack-ironic | 16:43 | |
| *** mkrai_ has quit IRC | 16:47 | |
| *** lucasagomes has quit IRC | 16:59 | |
| *** mjturek has quit IRC | 17:23 | |
| *** k_mouza has quit IRC | 17:24 | |
| *** dougsz has quit IRC | 17:32 | |
| openstackgerrit | Tzu-Mainn Chen proposed openstack/ironic master: Allow support for multipath volumes https://review.opendev.org/c/openstack/ironic/+/775681 | 17:34 |
| *** ociuhandu_ has joined #openstack-ironic | 17:39 | |
| *** ociuhandu_ has quit IRC | 17:40 | |
| *** ociuhandu has quit IRC | 17:40 | |
| *** ociuhandu has joined #openstack-ironic | 17:41 | |
| *** jawad_axd has joined #openstack-ironic | 17:50 | |
| openstackgerrit | Merged openstack/ironic master: Lazy-load node details from the DB https://review.opendev.org/c/openstack/ironic/+/776930 | 17:51 |
| *** mjturek has joined #openstack-ironic | 17:55 | |
| iurygregory | TheJulia, regarding the problem to detect if a partition is read only (https://storyboard.openstack.org/#!/story/2008696 ) do you think it would make sense to verify the stderr wipefs will give to skip that device? | 18:01 |
| *** irclogbot_2 has quit IRC | 18:03 | |
| *** irclogbot_3 has joined #openstack-ironic | 18:04 | |
| *** k_mouza has joined #openstack-ironic | 18:10 | |
| *** k_mouza has quit IRC | 18:15 | |
| *** gyee has joined #openstack-ironic | 18:17 | |
| *** irclogbot_3 has quit IRC | 18:24 | |
| TheJulia | iurygregory: likly, looks like it has no identifier on it, but the base drive does | 18:28 |
| TheJulia | so it makes the code a little more complex I think | 18:28 |
| *** irclogbot_3 has joined #openstack-ironic | 18:28 | |
| iurygregory | yeah, I'm wondering if it's because it's nvme ... | 18:28 |
| TheJulia | umm | 18:29 |
| TheJulia | I don't think so, what causes you to think that? | 18:29 |
| iurygregory | well the error only showed up in the two machines and the device was nvme... | 18:30 |
| TheJulia | well, the underlying device according to the full dmesg output, is listed as a usb mass storage device being presented as a scsi target | 18:32 |
| bfournie | iurygregory: I don't think it was related to those nvme drives | 18:32 |
| iurygregory | bfournie, gotcha | 18:33 |
| TheJulia | nvme is becoming popular for its performance and I guess lack of legacy overhead | 18:33 |
| TheJulia | so odds are for nvme to be present | 18:34 |
| TheJulia | regardless | 18:34 |
| openstackgerrit | Tzu-Mainn Chen proposed openstack/ironic master: Allow instance_info to override node interface https://review.opendev.org/c/openstack/ironic/+/777434 | 18:34 |
| iurygregory | I see | 18:35 |
| bfournie | iurygregory, TheJulia: for that metadata erase issue, its sdc1 that is failing. I see in the ramdisk log that the underlying device is skipped - "Skipping metadata erase of read-only device /dev/sdc". Should we also be skipping sdc1 in this case? | 18:37 |
| iurygregory | bfournie, can happen in any device I would say | 18:38 |
| iurygregory | in my mind we could check if the wipefs stderr contains "probing initialization failed: Read-only file system" | 18:39 |
| iurygregory | if yes we would just skip that device | 18:39 |
| *** ociuhandu has quit IRC | 18:42 | |
| *** jawad_axd has quit IRC | 18:44 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP: Scoped RBAC testing https://review.opendev.org/c/openstack/ironic/+/778957 | 18:47 |
| TheJulia | oh, i didn't catch that in the logs | 18:48 |
| TheJulia | if /dev/sdc is read only, then yeah we should skip the partitions on the entire device | 18:48 |
| TheJulia | iurygregory: we can't incase it is a legitimate device we need to destroy | 18:48 |
| TheJulia | realistically, we need to know it is not failed device or failing device that has gone read-only | 18:48 |
| TheJulia | because if it is, cleaning must fail | 18:49 |
| bfournie | TheJulia: yeah I think in _is_read_only_device for partitions we should check if underlying device is readonly | 18:49 |
| TheJulia | I just don't want to disqualify any read-only device | 18:50 |
| TheJulia | but I think that is what we did with the hpe device, so if we mirror that I'm okay | 18:50 |
| *** ociuhandu has joined #openstack-ironic | 18:54 | |
| iurygregory | yeah we have the other cases like you said .-. | 18:55 |
| *** ociuhandu has quit IRC | 18:58 | |
| TheJulia | Well, while CI runs, I'm going to go run into town and see if I can find a more comfortable office chair | 19:00 |
| iurygregory | good luck | 19:05 |
| *** jawad_axd has joined #openstack-ironic | 19:05 | |
| *** k_mouza has joined #openstack-ironic | 19:25 | |
| *** outbrito has quit IRC | 19:28 | |
| *** k_mouza has quit IRC | 19:30 | |
| *** jawad_axd has quit IRC | 19:52 | |
| *** jawad_axd has joined #openstack-ironic | 19:57 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP: Scoped RBAC testing https://review.opendev.org/c/openstack/ironic/+/778957 | 20:47 |
| *** jamesdenton has quit IRC | 20:49 | |
| *** jamesden_ has joined #openstack-ironic | 20:49 | |
| *** jawad_axd has quit IRC | 20:55 | |
| *** hoonetorg has quit IRC | 20:57 | |
| *** hoonetorg has joined #openstack-ironic | 21:01 | |
| *** k_mouza has joined #openstack-ironic | 21:10 | |
| *** k_mouza has quit IRC | 21:15 | |
| *** jawad_axd has joined #openstack-ironic | 21:17 | |
| *** jawad_axd has quit IRC | 21:31 | |
| * TheJulia fires up a fresh devstack | 21:40 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Revising quickstart doc for Centos8 https://review.opendev.org/c/openstack/ironic/+/779061 | 22:54 |
| *** iurygregory has quit IRC | 22:57 | |
| *** rh-jelabarre has quit IRC | 23:03 | |
| *** iurygregory has joined #openstack-ironic | 23:22 | |
| *** rloo has quit IRC | 23:52 | |
| *** k_mouza has joined #openstack-ironic | 23:52 | |
| *** k_mouza has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!