| janders | good morning Ironic o/ | 00:00 |
|---|---|---|
| janders | my usual IRC setup is down due to scheduled maintenance of the server I run it on, will run on HexChat for a few days, due to this my offline presence may be patchy | 00:01 |
| *** pmannidi is now known as pmannidi|afk|1hr | 02:26 | |
| *** pmannidi|afk|1hr is now known as pmannidi | 05:01 | |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add support for verify steps https://review.opendev.org/c/openstack/ironic/+/800001 | 05:28 |
| arne_wiebalck | Good morning janders and Ironic! | 06:30 |
| iurygregory | good morning janders arne_wiebalck and Ironic | 07:00 |
| arne_wiebalck | hey iurygregory o/ | 07:04 |
| *** rpittau|afk is now known as rpittau | 07:35 | |
| rpittau | good morning ironic! o/ | 07:35 |
| iurygregory | morning rpittau o/ | 07:35 |
| rpittau | hey iurygregory :) | 07:35 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost stable/wallaby: Fix Redeploy Playbook https://review.opendev.org/c/openstack/bifrost/+/800379 | 08:16 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Force selinux presence when testing https://review.opendev.org/c/openstack/bifrost/+/799965 | 08:17 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Force selinux presence when testing https://review.opendev.org/c/openstack/bifrost/+/799965 | 08:18 |
| janders | hey @arne_wiebalck, @iurygregory and @rpittau o/ | 08:23 |
| rpittau | hey janders :) | 08:23 |
| iurygregory | o/ | 08:24 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] Test latest ansible https://review.opendev.org/c/openstack/bifrost/+/800440 | 08:36 |
| opendevreview | Merged openstack/ironic master: Allow reboot to hard disk following iso ramdisk deploy. https://review.opendev.org/c/openstack/ironic/+/800300 | 09:11 |
| opendevreview | ankit proposed openstack/ironic-tempest-plugin master: Adding firmware update standalone gate https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/763340 | 10:46 |
| *** cenn is now known as cenne | 11:21 | |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add support for verify steps https://review.opendev.org/c/openstack/ironic/+/800001 | 12:04 |
| janders | see you tomorrow Ironic o/ | 12:24 |
| rpittau | bye janders o/ | 12:25 |
| cenne | Bye janders | 12:27 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Perform upgrade by default when installing ansible https://review.opendev.org/c/openstack/bifrost/+/800461 | 12:35 |
| rpittau | mmm we don't ugprade ansible in bifrost upgrade jobs, this should handle that ^, it will probably fail first | 12:35 |
| *** needssleep is now known as TheJulia | 12:44 | |
| TheJulia | Good morning everyone | 12:45 |
| rpittau | good morning TheJulia :) | 12:46 |
| * TheJulia needs lots of coffee | 12:53 | |
| opendevreview | Merged openstack/ironic master: Update iDRAC virtual media boot known issue https://review.opendev.org/c/openstack/ironic/+/800073 | 12:58 |
| opendevreview | Merged openstack/ironic master: Add ipxe ramdisk kernel append param test https://review.opendev.org/c/openstack/ironic/+/800298 | 12:58 |
| opendevreview | Riccardo Pittau proposed openstack/ironic stable/ussuri: Remove grenade jobs from old stable branches https://review.opendev.org/c/openstack/ironic/+/800465 | 13:04 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-inspector stable/ussuri: Remove grenade jobs from old stable branches https://review.opendev.org/c/openstack/ironic-inspector/+/800466 | 13:05 |
| opendevreview | ankit proposed openstack/ironic-tempest-plugin master: Adding firmware update standalone gate https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/763340 | 13:06 |
| opendevreview | Riccardo Pittau proposed openstack/networking-generic-switch stable/ussuri: Remove grenade jobs from old stable branches https://review.opendev.org/c/openstack/networking-generic-switch/+/800467 | 13:07 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Perform upgrade by default when installing ansible https://review.opendev.org/c/openstack/bifrost/+/800461 | 13:34 |
| TheJulia | so who is chairing the meeting today? | 13:59 |
| iurygregory | I think we forgot to decide that 2 weeks ago :D | 14:10 |
| rpittau | I can chair the meeting today, just need to review the agenda :) | 14:20 |
| iurygregory | tks rpittau | 14:20 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Perform upgrade by default when installing ansible https://review.opendev.org/c/openstack/bifrost/+/800461 | 14:29 |
| TheJulia | I suspect some of the agenda items are redundant, to be honest | 14:34 |
| TheJulia | maybe they need to be revisited? | 14:34 |
| rpittau | TheJulia: you mean redunndant sections? | 14:41 |
| TheJulia | rpittau: I mean the discussion topics | 14:50 |
| iurygregory | he already updated the topics | 14:54 |
| TheJulia | okay | 14:57 |
| TheJulia | nevermind! | 14:57 |
| rpittau | :) | 14:58 |
| rpittau | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Jul 12 15:00:12 2021 UTC and is due to finish in 60 minutes. The chair is rpittau. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | o/ | 15:00 |
| erbarr | o/ | 15:00 |
| tkot | o/ | 15:00 |
| TheJulia | o/ | 15:00 |
| stendulker | o/ | 15:00 |
| ajya | o/ | 15:00 |
| rpioso | \o | 15:00 |
| rpittau | Hello everyone and welcome to our weekly meeting! | 15:01 |
| rpittau | Our agenda can be found in the wiki | 15:01 |
| rpittau | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting | 15:01 |
| rpittau | mmm I wonder if we have quorum today | 15:02 |
| iurygregory | I think we do | 15:02 |
| iurygregory | 8 people.. | 15:02 |
| iurygregory | time to summon arne_wiebalck :D | 15:02 |
| arne_wiebalck | oops ... o/ | 15:02 |
| rpittau | yep, we should be fine | 15:03 |
| * arne_wiebalck was lost in grub | 15:03 | |
| rpittau | #topic Announcements / Reminders | 15:03 |
| TheJulia | arne_wiebalck: oh noes | 15:03 |
| iurygregory | We have a new contributor, tkot ! He's an intern at Red Hat and will collaborate to ironic upstream =) | 15:04 |
| TheJulia | Welcome tkot! | 15:04 |
| tkot | Hi everyone :) | 15:04 |
| rpittau | welcome tkot :) | 15:04 |
| arne_wiebalck | Hey tkot ! | 15:04 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac master: Fix missing Target@Redfish.AllowableValues https://review.opendev.org/c/x/sushy-oem-idrac/+/800495 | 15:04 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac master: Fix IncludeInExport allowed value validation https://review.opendev.org/c/x/sushy-oem-idrac/+/800496 | 15:04 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac master: Update default IncludeInExport value https://review.opendev.org/c/x/sushy-oem-idrac/+/800497 | 15:04 |
| rpittau | I have only a reminder that we're close to a new bugfix branch | 15:05 |
| TheJulia | ++ | 15:05 |
| rpittau | that should happen late this week or early next week | 15:05 |
| TheJulia | I was thinking about that last week when posting updates for ironicbaremetal.org | 15:05 |
| rpittau | oh yeah, we need to remember to update that after the release | 15:06 |
| iurygregory | I would say to do next week | 15:07 |
| iurygregory | if possible ofc (we won't die because of 1 week :D) | 15:07 |
| TheJulia | ++, it presently lists 17.0.1, I posted an update to 18.0, but was thinking we may also want to discuss noted versions later on in the meeting | 15:07 |
| rpittau | yeah, most likely early next week, depends on how things go with the patches this week | 15:07 |
| rpittau | TheJulia: sure! | 15:08 |
| rpittau | does anyone have anything else to announce/remind us ? | 15:09 |
| rpittau | alright, moving on | 15:09 |
| rpittau | #topic Review action items from previous meeting | 15:09 |
| rpittau | I didn't see any action items from 2 weeks ago | 15:10 |
| rpittau | we can skip forward | 15:10 |
| iurygregory | yeah we can move on =) | 15:10 |
| rpittau | #topic Review subteam status reports | 15:10 |
| rpittau | #link https://etherpad.opendev.org/p/IronicWhiteBoard | 15:10 |
| TheJulia | Are we done with anaconda short of CI at this point? | 15:11 |
| rpittau | mmm last update was from JayF 3 days ago | 15:12 |
| iurygregory | we probably need to ping zer0c00l or JayF to see if they are working on something | 15:12 |
| rpittau | we'll see during the week as JayF update is from Friday | 15:13 |
| TheJulia | I chatted with JayF regarding CI, turns out nobody publishes an anaconda stage2 ramdisk. There *is* an upstream ramdisk that has *everything* but it feels inappropriate to pull like 700MB into a CI run for Ironic on every single change revision. | 15:13 |
| TheJulia | ahh, he likely noted that | 15:14 |
| iurygregory | wow :O | 15:14 |
| iurygregory | maybe we can have a job in the experimental pipeline? | 15:14 |
| iurygregory | so we can run from time to time to see if things are ok? | 15:14 |
| TheJulia | maybe | 15:14 |
| iurygregory | it will only run on check-experimental so maybe "it's fine" | 15:15 |
| rpittau | arne_wiebalck: any update about the node error history ? | 15:15 |
| arne_wiebalck | no, sorry | 15:15 |
| * arne_wiebalck will try to get to it now that burn-in is coming to an end | 15:15 | |
| TheJulia | arne_wiebalck: could you list the latest known patch. I might pick it up as my next free time thing... | 15:16 |
| TheJulia | arne_wiebalck: or we could collaborate :) | 15:16 |
| arne_wiebalck | TheJulia: yes, sounds good | 15:16 |
| rpittau | I think there's an open patch since a while | 15:16 |
| TheJulia | for db model changes | 15:16 |
| arne_wiebalck | yes, that one did not see many reviews yet | 15:16 |
| * MahnoorAsghar Hey guys! I'm sorry I wont be able to attend the meeting today. Had an RFE for review on the agenda (https://storyboard.openstack.org/#!/story/2008866). Can we kindly drop comments on the storyboard? | 15:17 | |
| rpittau | MahnoorAsghar: sure thing, no problem | 15:17 |
| * MahnoorAsghar (Sorry for the interruption, please go on) | 15:17 | |
| * MahnoorAsghar Thanks ^-^ | 15:17 | |
| rpittau | I think we can move on | 15:18 |
| TheJulia | iurygregory: w/r/t virtual media visibility. You note verify steps work. I'm largely unaware of this, is there someplace I can read about it? | 15:19 |
| iurygregory | TheJulia, it was a conversation with janders downstream only | 15:19 |
| iurygregory | If I recall correctly dtantsur pointed to him that could be useful for the vmedia also | 15:19 |
| rpittau | do we maybe need a story for that ? | 15:19 |
| TheJulia | Okay, lets try and upstream that discussion | 15:19 |
| iurygregory | I think we do have for verify steps upstream | 15:20 |
| TheJulia | rpittau: apparently | 15:20 |
| iurygregory | let me see if I can find | 15:20 |
| iurygregory | but I don't think it mentions the vmedia part =) | 15:20 |
| ajya | https://storyboard.openstack.org/#!/story/2009025 is the one janders working on | 15:20 |
| ajya | btw, should that be RFE? | 15:20 |
| rpittau | ajya: thanks, and yes that looks like an RFE | 15:21 |
| rpittau | it's mentioned by janders in the story | 15:21 |
| ajya | yup, could add usual tags | 15:22 |
| rpittau | yeah | 15:22 |
| iurygregory | I've updated the title | 15:23 |
| TheJulia | Okay | 15:23 |
| TheJulia | thanks! | 15:23 |
| rpittau | cool, let's move forward ? | 15:23 |
| iurygregory | ++ | 15:24 |
| rpittau | #topic Deciding on priorities for the coming week | 15:24 |
| rpittau | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:24 |
| rpittau | we have few patches left | 15:24 |
| rpittau | I'm going to add a couple for bifrost | 15:24 |
| ajya | can these be added for sushy https://review.opendev.org/c/openstack/sushy/+/800267 https://review.opendev.org/c/openstack/sushy/+/799504 ? | 15:24 |
| rpittau | we also have the one for the node history db there | 15:25 |
| TheJulia | Has anyone scrubbed through the randomly submitted patches recently to see if some should be reviewed? | 15:25 |
| rpittau | ajya: feel free to add the tag there :) | 15:25 |
| iurygregory | I would like people to re-review the event subscription spec =) | 15:25 |
| arne_wiebalck | The last burn-in step (network) could be added maybe: https://review.opendev.org/c/openstack/ironic-python-agent/+/799687 | 15:26 |
| iurygregory | arne_wiebalck, ++ | 15:26 |
| arne_wiebalck | (the static part, we work on the dynamic version) | 15:26 |
| ajya | rpittau: thanks | 15:27 |
| TheJulia | hmm, the ismount catch should be as well | 15:27 |
| TheJulia | https://review.opendev.org/c/openstack/ironic-python-agent/+/798394 <-- any objections? | 15:27 |
| iurygregory | sounds good to me | 15:27 |
| * arne_wiebalck is all for more bootloader things | 15:28 | |
| rpittau | iurygregory: I'll find the time this week to review the spec | 15:29 |
| TheJulia | arne_wiebalck: I found out the vfs layer would lie. was awesome troubleshooting it :( | 15:29 |
| arne_wiebalck | TheJulia: lying about what is mounted? | 15:30 |
| iurygregory | rpittau, tks! I did a lot of changes after some discussion last week with dtantsur and TheJulia =) | 15:30 |
| TheJulia | Yeah, or to be more specific, the ismount call was returning False and then the handling code would blow up | 15:30 |
| TheJulia | because it was actually already mounted | 15:30 |
| arne_wiebalck | erm ... | 15:30 |
| TheJulia | yeah | 15:31 |
| arne_wiebalck | I have a patch submitted recently where I do check for sth being mounted, since it sometimes was mounted | 15:31 |
| arne_wiebalck | Unclear to me why. | 15:31 |
| arne_wiebalck | Maybe I was lied to! | 15:31 |
| rpittau | looks like we have a good list for the weekly priorities | 15:32 |
| rpittau | anything else to add ? | 15:32 |
| TheJulia | the filesystems that get loaded with ramdisks have really funky vfs calls/interactions. We found a similar issue with pre-made device files from the cpio file when compared to devfs behavior | 15:32 |
| TheJulia | rpittau: not from me | 15:32 |
| rpittau | ok, we don't have any topic for Discussion so we'll skip that | 15:32 |
| rpittau | #topic Baremetal SIG | 15:33 |
| rpittau | #link https://etherpad.opendev.org/p/bare-metal-sig | 15:33 |
| arne_wiebalck | Next meeting is tomorrow 10 July 2021 featuring TheJulia on "Getting started with Ironic: Bifrost". | 15:33 |
| arne_wiebalck | Don't miss out! :) | 15:33 |
| arne_wiebalck | We do not have a topic for August yet. | 15:33 |
| arne_wiebalck | I was wondering if we skip it due to holiday season? | 15:34 |
| TheJulia | could we do an ironic operator's round table? | 15:34 |
| arne_wiebalck | Sure ... how do get operators to join, though? | 15:34 |
| TheJulia | Lots of outreach I guess? | 15:34 |
| arne_wiebalck | I could ask some Ironic operators I know directly. | 15:34 |
| TheJulia | I could do the same | 15:35 |
| TheJulia | I could also tweet some to see if any that follow me may be interested | 15:35 |
| arne_wiebalck | So, we keep the one next month or this is for Sep? | 15:35 |
| iurygregory | post on twitter for RT :D | 15:35 |
| rpittau | I could too, but in August basically everyone is on vacation :D | 15:35 |
| TheJulia | oh, yes, the great august vacation window :) | 15:35 |
| * TheJulia needs a vacation | 15:35 | |
| rpittau | we may have more luck for September | 15:35 |
| TheJulia | Yeah | 15:35 |
| TheJulia | agreed | 15:35 |
| arne_wiebalck | I wold vote for Sep, too. | 15:36 |
| arne_wiebalck | would | 15:36 |
| arne_wiebalck | ok, Sep it is then | 15:36 |
| arne_wiebalck | I think that is it for the SIG for today. | 15:36 |
| rpittau | #agreed no baremetal-sig meeting in August | 15:36 |
| rpittau | thanks arne_wiebalck :) | 15:36 |
| TheJulia | iurygregory: tweet posted, retweet away | 15:37 |
| rpittau | ok, onward! | 15:37 |
| rpittau | we had only one request for RFE review but MahnoorAsghar can't participate to the meeting today | 15:38 |
| rpittau | #topic RFE review | 15:38 |
| rpittau | [RFE] RAID config by Operator using generic disk numbers, and vendor-specific RAID controller names https://storyboard.openstack.org/#!/story/2008866 | 15:38 |
| rpittau | he asked to leave comments directly in the story | 15:39 |
| rpittau | alright I guess we can move on | 15:40 |
| TheJulia | okay | 15:40 |
| rpittau | #topic Open Discussion | 15:40 |
| rpittau | anything to discuss today? | 15:41 |
| TheJulia | So one thing I wanted to raise is ironicbaremetal.org | 15:42 |
| TheJulia | We presently list only one latest release | 15:42 |
| TheJulia | I'm wondering if we should consider noting something like "latest release" and "latest stable branch" | 15:43 |
| rpittau | I think it makes sense to mention both | 15:43 |
| TheJulia | s/branch/release | 15:43 |
| TheJulia | We need a goodbye iscsi blog post as well :) | 15:43 |
| rpittau | do we need to make it clear that the latest-latest is in development ? | 15:44 |
| TheJulia | rpittau: well, in development can be viewed a different ways. We've released the thing, we'll stand by and support the thing, just not for a super long time like we will with a stable branch version | 15:44 |
| rpittau | ok | 15:45 |
| TheJulia | and there will be times where the releases will be the same around stable release cycle times | 15:45 |
| rpittau | yes, of course | 15:46 |
| rpittau | anyway, I'm in favor of that | 15:46 |
| TheJulia | There was something else I was thinking of, but my brain can't recall at the moment | 15:48 |
| rpittau | :) | 15:49 |
| rpittau | if we don't have anything else to discuss, I'll pass to the weekly question of.... | 15:49 |
| TheJulia | surely it was something to facilitate taking over the world | 15:49 |
| rpittau | lol | 15:50 |
| lmcgann | Ill just throw out, sdanni and I are working on a demo for a complete attestation workflow - we've identified a few keylime bugs but we're super close! | 15:50 |
| TheJulia | lmcgann: awesome | 15:50 |
| rpittau | that sounds great lmcgann | 15:50 |
| TheJulia | lmcgann: I was working on the bifrost presentation and started recalling all the bugs we found then... so I know that feeling :) | 15:51 |
| TheJulia | lmcgann: When would be good? or do you guys just want to record a video or??? | 15:51 |
| lmcgann | I think a video might be easiest | 15:53 |
| TheJulia | cool | 15:53 |
| lmcgann | And more easily shareable | 15:53 |
| rpittau | alright then, let's move to the last topic | 15:55 |
| rpittau | #topic Who is going to run the next meeting? | 15:55 |
| iurygregory | I can | 15:55 |
| rpittau | thanks iurygregory :) | 15:55 |
| iurygregory | np! | 15:55 |
| rpittau | and that's all folks! | 15:56 |
| rpittau | #endmeeting | 15:56 |
| opendevmeet | Meeting ended Mon Jul 12 15:56:07 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:56 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-12-15.00.html | 15:56 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-12-15.00.txt | 15:56 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-07-12-15.00.log.html | 15:56 |
| TheJulia | so arne_wiebalck, whats going on with grub? | 15:57 |
| arne_wiebalck | Our grub fun of the week: the operations team had to replace a disk in one of our nodes some weeks ago. The repair followed the usual procedure: partition the disk, add the partitions to the corresponding RAID devices, then install grub on the new disk, and finally compare the MBR on all potential boot disks to make sure all is fine. So far so good. A few days ago then, the node needed to reboot. That failed due to | 15:58 |
| arne_wiebalck | grub issues \o/ | 15:58 |
| * arne_wiebalck hopes that noone is able to spot the issue in less than 2 mins ... | 15:59 | |
| TheJulia | arne_wiebalck: was the machine in uefi or bios boot mode? | 16:00 |
| arne_wiebalck | bios | 16:00 |
| TheJulia | hmm. okay. What state was the machine in upon reboot? | 16:01 |
| arne_wiebalck | stuck in grub | 16:01 |
| arne_wiebalck | (I have not seen it at this stage, though) | 16:01 |
| TheJulia | was this the first disk that failed? | 16:01 |
| arne_wiebalck | yes | 16:01 |
| TheJulia | grubenv and grub.cfg files present on the new disk? | 16:02 |
| arne_wiebalck | the files are on /boot which is on the RAID, so on all disks, so yes | 16:02 |
| arne_wiebalck | first hint: the node was able to boot from the recently replaced disk, but not any of the others | 16:03 |
| arne_wiebalck | so, when it got stuck, it was booting from one of the old disks | 16:04 |
| arne_wiebalck | trying to boot | 16:04 |
| arne_wiebalck | here is what I think happened: grub2-install installs boot.img into the MBR, core.img into the MBR gap, and modules into /boot ... | 16:06 |
| arne_wiebalck | when the disk was replaced, all these 3 areas where updated | 16:06 |
| arne_wiebalck | the MBR part was overwritten with the same as on the other disks | 16:06 |
| arne_wiebalck | so the MBR check passed | 16:07 |
| arne_wiebalck | but the core.img in the gap (grub1.5) and/or the modules where replaced as well | 16:07 |
| arne_wiebalck | when the node was rebooted, it used one of the old disks | 16:07 |
| *** jgwentworth is now known as melwitt | 16:07 | |
| TheJulia | I didn't think the ~530 bytes in the mbr had enough information to say "go look at other disk" | 16:08 |
| arne_wiebalck | grub1.5 (which is on the disk) did not play well with the modules in /boot, so the node could not boot | 16:08 |
| TheJulia | because inconsistent versions? | 16:08 |
| arne_wiebalck | TheJulia: it is 440 bytes - 99 in facr | 16:08 |
| arne_wiebalck | fact | 16:08 |
| arne_wiebalck | the MBR also holds the partition table and the bios parameter block | 16:09 |
| TheJulia | ahh, 440 sounds right, I remembered it was super small enough to point to the next spot | 16:09 |
| arne_wiebalck | yes, MBR --> MBR gap --> /boot | 16:09 |
| arne_wiebalck | so, when grub2-install ran, it overwrote /boot and made it unusable for the other disks | 16:10 |
| TheJulia | yup | 16:10 |
| arne_wiebalck | ok, running grub2-install on all disks fixes this | 16:10 |
| TheJulia | That sounds right and I think I hit something similar years ago with a server that had a software raided boot partition | 16:10 |
| arne_wiebalck | as then the gap is updated everywhere | 16:10 |
| TheJulia | yeah | 16:10 |
| arne_wiebalck | ok, cool, so I ran this, compared the MBR *and* the gap, all good, node boots fine from all disks | 16:11 |
| arne_wiebalck | then I wrote a scanner to check for pending issues from the replacments over the past months | 16:11 |
| arne_wiebalck | to fix all nodes where we replaced disks | 16:12 |
| arne_wiebalck | I found some | 16:12 |
| arne_wiebalck | with different MBRs even | 16:12 |
| arne_wiebalck | I ran grub2-install ... MBRs remain different ... huh? | 16:12 |
| * TheJulia is starting to understand why the grub folks just want everyone using uefi boot | 16:13 | |
| arne_wiebalck | grub2-install /dev/sda ; grub2-install /dev/sdb (and the MBRs are different after) | 16:13 |
| TheJulia | oh no | 16:13 |
| TheJulia | Different pointer addresses for where to start? is the layout identical between the disks? | 16:14 |
| arne_wiebalck | when I say MBR I mean the first 440 bytes, not the first 512 with the disk signature | 16:14 |
| arne_wiebalck | same disk models | 16:14 |
| arne_wiebalck | same everything | 16:14 |
| arne_wiebalck | grub does not install into the first 440 bytes | 16:14 |
| arne_wiebalck | it installs into the first 3 bytes | 16:15 |
| TheJulia | but that 440 bytes, if I remember correctly does contain the address of where to start reading at as interpretted by grub | 16:15 |
| arne_wiebalck | these contain a jump instruction | 16:15 |
| arne_wiebalck | TheJulia: right! | 16:15 |
| TheJulia | is the target location the same | 16:15 |
| TheJulia | ? | 16:15 |
| arne_wiebalck | so, grub writes sth like "eb 63 90" at the beginning which means "jump 99bytes noop" to find the real boot.img | 16:16 |
| arne_wiebalck | the space in between is used by the bios parameter block | 16:16 |
| arne_wiebalck | so grub stage1 is even smaller | 16:16 |
| arne_wiebalck | ok, great, I confirm this by zeroing the first 440 bytes and booting the node | 16:17 |
| arne_wiebalck | zeroing, grub-install, booting | 16:17 |
| arne_wiebalck | works | 16:18 |
| arne_wiebalck | I adapt the scanner to take this into account | 16:18 |
| arne_wiebalck | continue scanning and find nodes which differ in the gap! | 16:18 |
| arne_wiebalck | what? | 16:18 |
| arne_wiebalck | differ in the gap after grub-install on all disks | 16:18 |
| arne_wiebalck | WHAT? | 16:18 |
| arne_wiebalck | took me a moment to realise I was meanwhile on a BIOS node with a GPT partition table :-D | 16:19 |
| arne_wiebalck | so, here core.img is in the grub_bios partition | 16:19 |
| arne_wiebalck | I adapted the scanner to support MBR/dos and GPT | 16:20 |
| arne_wiebalck | and start to scan nodes again ... | 16:20 |
| TheJulia | okay, so it does seem like my memory is telling me the right things that the offset is dynamicly determiend and set accordingly | 16:20 |
| arne_wiebalck | ... and get summoned into the Ironic weekly meeting by iurygregory :-D | 16:20 |
| arne_wiebalck | the offset should be the same on identical disks | 16:21 |
| rpittau | bye everyone, good night! o/ | 16:21 |
| *** rpittau is now known as rpittau|afk | 16:21 | |
| TheJulia | Okay, so you understand what is going on, and the pain of grub shall one day be a tech best seller right?!? | 16:21 |
| TheJulia | i concur, it *should* be the same if they are entirely structurally the same, but thinking it may be necessary to rip apart the bytes to figure out what they are saying | 16:22 |
| arne_wiebalck | this is what I thought: I should write a book on grub, s/w RAID, BIOS/UEFI ... only 8 people would *maybe* buy it :) | 16:22 |
| TheJulia | make that 9, I'd buy a copy for valpertha | 16:23 |
| arne_wiebalck | :-D | 16:23 |
| arne_wiebalck | updating kernels, updating grub software is all fine | 16:23 |
| arne_wiebalck | the moment you re-install grub, you have to do it on all disks at the same time | 16:24 |
| TheJulia | yeah | 16:24 |
| TheJulia | or your replacement disk needs get the contents of the original disk mirrored over to it precisely using dd | 16:24 |
| arne_wiebalck | yes, that should also work! | 16:25 |
| * arne_wiebalck hopes he will never forget "eb 63 90" now | 16:25 | |
| arne_wiebalck | ok, that's it ... sorry everyone for 20mins of grub spam | 16:26 |
| TheJulia | :) | 16:28 |
| TheJulia | Interesting grub spam, the basis for your book on grub | 16:28 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Enable disable_by_file backend of healthcheck middleware by default https://review.opendev.org/c/openstack/ironic/+/800515 | 16:32 |
| arne_wiebalck | TheJulia: :) | 16:34 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Enable disable_by_file backend of healthcheck middleware by default https://review.opendev.org/c/openstack/ironic/+/800515 | 16:37 |
| opendevreview | Verification of a change to openstack/ironic-python-agent failed: Reduce logging verbosity when collecting logs https://review.opendev.org/c/openstack/ironic-python-agent/+/796218 | 17:20 |
| arne_wiebalck | bye everyone, see you tomorrow o/ | 17:25 |
| JayF | TheJulia: rpittau|afk: iurygregory: All the pain points about Anaconda CI setup are documented in a story, I think I linked it in the status report? If not I can dig it up. | 18:51 |
| TheJulia | JayF: I mentioned it before I saw the link on the story | 19:09 |
| TheJulia | err, notes on the story via the link on the whiteboard | 19:09 |
| TheJulia | JayF: its all good | 19:09 |
| JayF | I was thinking (and probably won't have time to execute on this) that perhaps a docker container is an option | 19:10 |
| JayF | if we can run docker in the gate, we can run a container that would have the stuff inside needed to make an image | 19:10 |
| TheJulia | likely. I don't really see why not aside from dockerhub rate limitnig | 20:34 |
| trandles | Bifrost question: When was it a requirement that a port be created to enroll a node when using standalone ironic? | 21:33 |
| trandles | *When did it become a requirement...? | 21:33 |
| trandles | nevermind, I guess I didn't notice it when inspection auto-enrolled my node :P | 21:38 |
| janders | good morning Ironic o/ | 23:08 |
| * janders is looking through the meeting logs | 23:08 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!