Monday, 2023-10-30

rpittaugood morning ironic! o/08:08
masgharGood morning!09:10
opendevreviewMerged openstack/bifrost stable/victoria: CI: Update cached cirros image to 0.5.3  https://review.opendev.org/c/openstack/bifrost/+/88588009:58
iurygregorygood morning Ironic11:08
TheJuliagood morning12:52
iurygregorygood morning TheJulia =)12:59
JayFDid anyone else want in on an Ironic docs meeting? Trying to come up with a plan of work that my downstream might try to get some resources to implement fo rus14:53
JayFWhen I asked before Dmitry was the only one that responded IIRC; just ensuring nobody else is interested before sending invites14:53
iurygregoryJayF, please add me =)14:54
JayFack; I'm assuming early PST (this time of day +2-3 hours?) works for you?14:54
iurygregoryyup14:56
JayF#startmeeting ironic15:00
opendevmeetMeeting started Mon Oct 30 15:00:07 2023 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
JayFHey guess what; another meeting!15:00
JayF#topic Announcements/Reminder15:00
TheJuliao/15:00
JayFStanding reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash15:00
iurygregoryo/15:00
JayF#topic Review action items from previous meeting15:00
JayF#info JayF to backport ngs_save fix for networking_generic_swtich, cut a bugfix-version (not branch) release of it15:01
JayFgoing to be honest, don't remember if I did this, checking15:01
rpittauo/15:01
iurygregoryyou did15:01
JayFdid it land?15:01
iurygregoryjust need to update based on the reivew15:01
rpittauJayF: re ironic docs: I think I replied too, or maybe not :P15:01
JayFrpittau: ack, will add you to the meeting, I thought I missed someone 15:01
JayFrpittau: my IRC "highlights channel" is unreadable after the last week, for some reason :D 15:02
JayFiurygregory: I'll follow up on it15:02
JayF#topic Caracal release Schedule15:02
rpittau:D15:02
JayF#link https://releases.openstack.org/caracal/schedule.html15:02
JayFit's there, take note, I'll call out the next milestone in future meetings but didn't do the prework for that today :)15:02
JayFI doubt there's action for us to take this week for release in October :D 15:03
JayF#topic PTG Review15:03
JayF#link https://etherpad.opendev.org/p/ironic-ptg-october-202315:03
JayFAlright, this is what I was trying to get to. We have a lot of action items from this. The longer term ones/workstreams are going into workstream doc, obviously15:03
JayF I wanna track any "JFDI" level items here 15:03
JayFActually; brakes and reverse: I don't want to take this for granted; thank you all for participating in vPTG last week15:04
JayFGoing to start plugging some action items in here. Just because they're here doesn't mean they're due next week, just means I wanted to have them written somewhere other than the etherpad :)15:05
JayF#action JayF to Merge something to metalsmith readme/release notes to announce it is going into maintenance mode before it's next release15:05
JayF#undo15:05
opendevmeetRemoving item from minutes: #action JayF to Merge something to metalsmith readme/release notes to announce it is going into maintenance mode before it's next release15:05
JayF#action JayF [from PTG] Merge something to metalsmith readme/release notes to announce it is going into maintenance mode before it's next release15:05
JayFthe other item around metalsmith will be tracked in the workstreams doc (I think dtantsur still owes an RFE for that; but there's no need to track it)15:06
JayF#action JayF [from PTG] Submit something to contributor docs, drafting a bug deputy role which triages bugs, checks periodic job status, and runs a bug jam meeting15:06
JayF#action JayF [from PTG] Setup IPA-builder in launchpad15:06
TheJulia... do we have periodics?15:07
JayFall stable jobs run in periodics aiui15:07
JayF#action masghar [from PTG] provide feedback about specific gaps in documentation,  fix if small, provide feedback if more sizable15:08
TheJulia.. That doesn't seem right, but it could still be done and not obvious with the config15:08
JayFmasghar: I'll note, something that's even two or three lines by next week will be super useful; no need to stress over details if you're short on time.15:08
JayFTheJulia: either way; I'll dig as part of doign it15:08
JayF#action JayF [from PTG] Update any written vendor driver policy to indicate it's about coordination/communication, not as much 3rd party CI now.15:09
JayFthat's tuesday LOL15:09
JayF#action JayF [from PTG] email list about IntelIPMIManagement driver usage15:09
JayFI don't see explicit actions bolded out of OVN/Redfish, TheJulia were there any?15:10
JayFlooks like maybe just make 3 bugs are the only actions there?15:10
TheJuliaJayF: None from OVN, it was more information/context setting15:10
TheJuliaRedfish, Let me glance again15:10
JayFI think that's all the "small" actions AFAICT15:11
JayFeverything else is large enough to need a bug or to be in a workstream15:11
JayFto set expectation, nonzero chance many of these will push until next week, but they'll 100% get done early-early next week15:11
TheJuliaOne is too early, network port toggling, and the boot record is JFDI the bug and figure out if we can do a secondary purge15:11
JayFOh, there is one other action not to drop15:12
TheJuliaAlso, RFE for boot status, but nothing major really15:12
TheJuliasmall building blocks15:12
JayF#action JayF Book a meeting with stakeholders; TheJulia JayF and johnthetubaguy minimum, on Nova driver bug fixing/hunting 15:13
JayFIf anyone wants in on that, please ask; only reason we didn't PTG it was so we'd be fresher :)15:13
JayFAlright, anything else as follow-up to PTG before we move on?15:14
JayF#topic Review Ironic CI Status15:14
JayFI haven't seen any random failures, but I don't think we've been hammering the gate the last week by any stretch :D 15:15
rpittauhaven't seen any p[articular recurring or worrying failure since a while15:16
JayF#topic RFE Review15:16
JayFthere's one up for review15:16
JayF#link https://bugs.launchpad.net/ironic/+bug/204023615:16
JayFReading this RFE, I'm not sure I want to approve it as written.15:17
JayFThe bug itself implies that it might add the config and only implement it for iRMC15:17
JayFif we add a config under `[conductor]` to set that path, it needs to be respected throughout15:18
JayFand it raises questions around stuff like, should we have separate settings for CA to verify other services connecting in? What about verifying BMCs? Or agents? Shoudl that path be respected if auto tls is on?15:18
JayFJust seems like there's enough edges there that an RFE bug might not be sufficient.15:18
JayFWDYT?15:18
TheJuliaso... I guess I'm conflicted15:18
TheJuliabecause verify_ca is supposed to take a path out of the box15:19
JayFIf it was like, [irmc]irmc_default_verify_ca_path 15:19
TheJuliaI guess I don't understand *why* from the lp bug15:19
JayFI'll comment in there, saying we were unable to approve because the scope and the "why" is unclear15:20
TheJuliaWhy do we need a second level default15:20
iurygregorysince it's a bit complicated for the reporter to be online (due to Timezone) we probably need to raise the questions we have in the RFE15:20
JayFI won't tag it needs-spec yet, but I'll ask for it to be updated before being readded to meeting agenda15:20
JayFof course :)15:20
iurygregoryand take async15:20
TheJulialooks like they also want to add it as a node field15:20
TheJuliawhich is... semi-problematic15:21
JayFYeah, we can't have that as a per-node override.15:21
TheJulianot insurmountable, but yeah, better understanding of need is needed15:21
TheJuliaThere *is* a case where you might need a different CA package to talk to to the BMC, and that could be reasonable15:22
TheJuliabut then that requires you know internal details like location of files on disk and set/get that via an API15:22
JayF[agent]verify_ca exists15:22
JayFso does [ilo]verify_ca15:22
JayFbut nothing condcuctor-wide15:22
TheJuliayeah, this is driver level for speaking with the remote BMC15:23
JayFOK; so I think we need to needs-spec this then, if it's node level15:23
JayFbecause we need to be very explicit about how we RBAC a setting to change the verify_ca settings per node15:23
JayFDo you think that's overkill?15:24
TheJuliaRBAC wies, it would just need to be adminy user15:24
TheJuliawise15:24
TheJuliaFrom a what is in the field wise, it really can't be a file path15:24
TheJuliawhich makes this further challenging15:25
JayFWhy not? You'd expect the deployment tooling to do key management.15:25
TheJuliaThink user not having filesystem access to the remote server where Ironic is running15:25
TheJuliathey don't *actually* know where the files are for that CA15:25
JayFI don't see another path other than full auto_tls style or something like an API integration with a key manager15:25
TheJuliabest they can supply, realistically, is a URL15:25
JayFoh, CA is not a private key15:25
JayFit's a public key15:25
TheJuliacorrect15:26
TheJuliabut they want to provide paths15:26
JayFbut URL is still not ideal, because you can't trust that value is the same15:26
* drannou has done his homework, when you want to speak about RFE #203967615:26
JayFthis needs to be specced.15:26
TheJuliaJayF: yeah, basically it does need to be written up in a more verbose form15:26
TheJuliaI get their basic intention after skimming the openshift enhancement doc15:26
TheJuliaand after you looked at the config, we just need to find a sensible path forward15:27
JayFcommented, tagged needs-spec15:28
JayFdrannou: that was not on the agenda15:28
JayFdrannou: usually we prefer those be on the agenda first15:28
JayFbut I'm curious :)15:28
drannouno problem, we can discuss about it when you want15:29
JayFYou're going to likely get the same answer from me, after looking over the RFE (I've read it already in the past)15:29
JayFa feature of that bulk is going to need a spec15:29
JayFthat shouldn't be seen as a barrier; I like using the spec template b/c it helps me prevent forgetting things15:30
TheJuliaI'm trying to remember, and maybe this is where a spec would help, why not enable/drive the deployed OS to self-manage SED ?15:30
JayFbut given it'll change deployment flows that'll help us understand exactly how15:30
TheJuliaI think this could just be a specific deploy interface, tbh, but those sorts of questions need to be sorted through15:31
JayFthe why honestly is really important15:31
JayFbceause if we know why we can help more with the how :D 15:32
drannouFor me it's a new driver, that is impacting deploy interface, reboot, etc15:32
drannouWhy activating this feature ?15:32
TheJuliaAlso, a related question, this references MBR. How does UEFI work with these sort of systems/devices because it seems like the model is to unlock, trigger a revisit to the boot sequence15:32
TheJuliadrannou: paint the vision for us, although I do seriously think a spec document is in  your future :)15:32
drannouMBR is not mandaroy, I put it because it's the "normal way to go", but for Ironic I don't think it's a good idea15:33
TheJulia(we do require such when we start talking adding interfaces/drivers, fwiw)15:33
TheJuliaAhh, okay.15:33
drannouNo problem to do it, just if you have examples or documentation on how to do it, that would help me a lot15:33
TheJuliatons, in the opendev.org/openstack/ironic-specs repo, there is also a template to start from15:34
JayFOh, there is a spec template, and it is glorious.15:34
JayFI often pull it down and use it for a template for personal projects, too, just because it's a useful checklist for "have I thought of all this stuff"15:34
drannouWonderfull, I will check that and try to do it tomorrow15:34
JayFthanks!15:35
drannouDo you need more context on why we need SED exactly ?15:35
JayFYes, exactly15:35
JayFwe want the use case15:35
JayFtell us the problem you have, how this solves it15:35
JayFwhy SED managed by the OS doesn't solve it15:35
TheJuliaand the why which drives the pattern of offloading the host OS awareness of SED15:35
JayFhonestly, doesn't even matter so much if we have the use case, think it's valuable, etc15:36
TheJuliaTheme wise, I think we kind of get it, we even have a kexec patch somewhere in gerrit you can take/use, but the why is kind of important overall15:36
drannouI will make a short version here : in EU, and more over in France, there is a new security certitication for wloud providers that is call "SecNumCloud", one of the spec is that, as a cloud provider, you need to give to your customer a way to completely encrypt hard drive15:36
TheJuliaoh15:36
JayFAh, so one of the key requirements here is that the customer holds the key15:36
TheJuliasuch that the machine can't independently boot15:36
TheJuliaugh15:36
drannouexactly15:37
TheJuliadrannou: please include links to the regulation15:37
drannouok15:37
TheJuliaI suspect we have some reading we'll need to do15:37
JayFthis is something that would likely need like15:37
JayFconsole or something like that15:37
JayFdang15:37
drannouSo saying that, you see that there is a tons of implication, one is that if the host is power down, for any reason if the disk "leave" the host, data should be encrypted15:37
JayFhow do you do this without the cloud provider being able to intercept?15:38
TheJuliaLots, and lots of layers of trust to establish15:38
JayFeven if you used console and put in key that way, you can proxy it15:38
JayFThis is a good use case, valuable.15:38
JayFAlso impossible-sounding.15:38
JayFI'm glad it's your use case and not mine drannou ;) 15:38
drannouNo You don't really need the "super root admin with all super Power" have no acces to the key, but that should be "control"15:39
TheJuliaeh, nothing is impossible, just depends on the size of the lever15:39
JayFAight, I think we have a decent idea15:39
JayFdrannou: this all still has to be written down btw15:39
JayFgoing to move on to...15:39
JayF#topic Open Discussion15:40
JayFAnything else?15:40
drannouhttps://review.opendev.org/c/openstack/ironic-python-agent-builder/+/89872315:40
TheJuliaYeah, spec document would be needed with context of the regulation driving the need, and then lets try and wrap our heads around the spec15:40
drannouI have this PR that is waiting for reviews, and may be even a global discussion for your question JayF 15:40
JayFyeah I just refreshed my comment and -1 15:41
TheJuliaIn other news, I have written the first recipe down into a gdoc.15:41
drannou(Question arround DHCP on IPA)15:41
JayFThanks drannou; hopefully folks have a look. I suspect reviews will start back up this week as people recover from PTG.15:42
drannouOk wonderfull thx15:42
JayFlast call before I close the meeting?15:43
TheJuliaJayF: just commented15:43
JayF#endmeeting15:44
opendevmeetMeeting ended Mon Oct 30 15:44:03 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:44
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-30-15.00.html15:44
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-30-15.00.txt15:44
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-10-30-15.00.log.html15:44
TheJuliaI've sort of mixed on that change specifically, but start state config is a hard problem (else this wouldn't be week 3 staring at it)15:44
JayFTheJulia: that is ~what I suspected but didn't want to +2 based on an assumption15:44
masgharVery late to the party, apologies. JayF: I have a small document here with respect to Bifrost docs, if we could have similar added to the docs it would be great: https://docs.google.com/document/d/1BlvJNL-wmqQIhKN1Yn0XHeanY53mOrzDHdSutzxjabI/edit15:45
JayFmasghar: can you make that readable to all?15:47
JayFmasghar: if not will dm you list to open it to15:47
masgharOh of course!15:47
masgharI have MACs and IPs mentioned in the diagrams. Is that alright? (I can change it afterwards if its okay)15:48
JayFI don't mind that; but if it's private informatino you likely want to change it before sending it to me :)15:49
JayFI don't want any secrets15:49
masgharI am not entirely sure, let me try to sanitize it a bit, make sure, and will make it readable to all15:49
rpittaumasghar: probably better do an upstream public version with no info15:49
masgharrpittau, ack15:50
JayFhttps://docs.openstack.org/ironic/latest/admin/drivers/snmp.html#software-requirements needs a fix16:27
JayF> The PySNMP package must be installed, variously referred to as pysnmp or python-pysnmp16:28
JayFwe're using a different ecosystem now, I think.16:28
rpittauJayF: yes, that would be pysnmp-lextudio16:46
JayFrpittau: yeah, probably the better fix it "install driver-requirements.txt"16:46
JayFrpittau: I just wanted to comment it here because Is aw it and couldn't fix it r/n16:46
JayFI'll probably do a quickfix before eod if nobody else does16:47
opendevreviewJay Faulkner proposed openstack/ironic master: Remove outdated pysnmp reference  https://review.opendev.org/c/openstack/ironic/+/89962417:01
rpittaugood night! o/17:03
opendevreviewJay Faulkner proposed openstack/ironic master: Remove outdated pysnmp reference  https://review.opendev.org/c/openstack/ironic/+/89962417:05
TheJuliadtantsur: https://bugs.launchpad.net/ironic/+bug/2041901 as a follow-up from the ptg17:25
dtantsurThanks! I wonder if we should default to removing the UEFI records if the disk is cleaned17:26
dtantsurThese operations are not orthogonal17:27
dtantsur(although I do see value in a separate step as well)17:27
JayFeh, it's weird right17:29
JayFlike, if we wanted to tie it to disk eraasure, that's an interesting idea17:30
JayFbut problematic as well17:30
JayFbecause we don't want operators to have to udpate all custom disk erase steps to be uefi aware17:30
JayFand we'd have to do such a disk-tied implementation as single-disk-aware so we'd only wipe the entries for the disks that were getting wiped17:30
JayFand it gets even *more* complex if you have a software raid root17:30
JayFso I think a separate step just gets us away from all those edges17:31
dtantsurIn the presence of disks that are NOT cleaned, we cannot also clean UEFI records. Who knows why these partitions are not cleaned, it could be OS recovery or some vendor tools.17:31
JayFyes-ish, but I also would propose we won't always have the information we need to make those decisions17:34
dtantsurThat's why I wonder if we need two things: a more aggressive clean step that removes everything we don't like and a more gentle part of the erase_* that only removes entries related to the disks being erased17:36
JayFso like, erase_block_devices becomes a dispatcher that, for each device; do:17:36
JayFerase_uefi_boot_for_block_device17:36
JayFerase_block_device17:36
JayFdevice_smart_status17:37
JayFsomething_else_i_cant_think_ofper_block_device17:37
JayFI still worry about having enough information17:37
JayFIMO; Julia's proposal represents an MVP (even if it would be disabled by default due to your concerns)17:38
JayFyour proposal represents the next enahncement to something like that17:38
dtantsurYeah, it makes sense as an MVP17:38
TheJuliaI was sort of having the same inner discussion as I was writing the bug, and my mind leaned towards "nuke everything" and ask for forgivness later from nvram. I mean, we could likely figure out and map things, but my worry is also devices where we can't17:41
* TheJulia looks at Emulex FC adapters with her best corgi impression17:42
JayFTheJulia: is there any way to back this information up to the EFI partition, I wonder?17:43
JayFThat gets  ... sticky and maybe outta ironic scope a little17:43
JayFor maybe even just log it in enough detail to recreate if you have access to the cleaning logs17:43
TheJuliaI honestly kind of think we could find it happy as-is and backport if we really wanted to, or not. This is one of those things we've gone back and forth on since it is a bug at the end of the day17:43
TheJuliaYeah, we can log it17:44
JayFI think there's a specific use case this is brutal to, in terms of how it's bugged17:44
JayFbut I think it's potentially brutal to the flip side use case17:44
JayFwhere someone is using Ironic to provision to a machine once and leave it alone17:44
JayFand may not expect things like, as dtantsur mentions, vendor tools and such to disappear from EFI menus17:44
TheJuliaArne did the pram stuffs as well, which is semi related, but there is the "there is a OS recovery image in the BMC" aspect which is a whole curve ball, but really that should *not* be UEFI addressed under normal cirumstances17:44
TheJuliaAnd... those special records are like FSVAR labels instead of HD labels17:45
JayFthat is part of what hurts here, it's that the rare cases this could cause pain on are almost the exact opposite set from the folks that benefit17:45
JayFat least in my estimation17:45
TheJuliaso, vendor tools *should not* be on the disks, under normal cirumstances.17:45
TheJuliaThat being said, the image inside the BMC thing, we've seen that with the fsvar label17:46
TheJuliathose we would likely just ignroe17:46
TheJulia... someone, hwoever writes the patch, needs to go dig at the e2dk nvram labels17:46
TheJuliasince there is a meaning there that we can only guess about too17:47
JayFaccess to a handful of different hardwares would remove a lot of unknowns from this17:47
JayFor at least, getting output from a command from them would...17:47
TheJuliaDell is definitely like fsvar(stuff):/path/file17:47
* JayF adds ironic-etd to phone home /s /s /s17:47
TheJuliaAnyone got a freshish ilo machine with the efibootmgr output handy?17:48
JayFWhat command will you need run? I can ask my downstream but it would likely be a long turnaround if they got it17:49
TheJulia"efibootmgr -v"17:49
TheJuliaIt will include some UUIDs and labeles, so maybe a little redaction or modification might be the needful17:50
TheJuliaNetwork devices are typically PciRoot devices too17:50
TheJuliahttps://www.irccloud.com/pastebin/HTgnfpbF/17:51
TheJuliaThat is my desktop17:51
JayFack; I passed on the ask. I know they are crushed right now through EOY and this is low priority, but if they're able to they will.17:52
JayFWe could also email nisha's team to get info for at least the lines they work on.17:52
TheJulia++17:52
TheJuliaNisha did add a sushy bug we could use insight on17:52
JayF(well, not low priority for you, I know, but upstream is lower priority than the work they are doing I mean :D)17:52
TheJuliaapparently a string we're looking for in the bmc response is an internal version id that leaks17:53
TheJuliawell, it might just be the OS version17:53
TheJuliabut yeah.....17:53
JayFI'm out today in about an hour, heads up; gotta take the kitty to the vet18:08
TheJulialk18:09
TheJuliaerr, ok18:09
JayFAlso will not be in IRC at all Friday; will be at SeaGL conference Fri/Sat in Seattle.18:09
JayFI will have socks! Tell people to come to my talk and get some.18:09
TheJuliasadly the demise of twitter has largely disconnected me from most of those folks :(18:17
iurygregoryJayF, can you convert your +1 to +2 in https://review.opendev.org/c/openstack/ironic/+/897989 ? 21:52
iurygregoryand +W =) since we have +2 from dtantsur 21:53
JayFJulia has a comment on that she hasn't ack'd your response to22:29
JayFthat's the sorta thing I'd rather wait on or let another core ack who knows the hardware and has more confidence22:29
TheJuliaI just reviewed it, thanks iurygregory 22:31
JayF\o22:32
JayF  /22:32
iurygregoryoh, tks eveyrone22:51
iurygregorytomorrow will be complicated since I need this downstream for a escalation YAY22:51
opendevreviewMerged openstack/ironic master: Make sure we eject media from DVD when CD is requested  https://review.opendev.org/c/openstack/ironic/+/89798923:34
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic stable/2023.2: Make sure we eject media from DVD when CD is requested  https://review.opendev.org/c/openstack/ironic/+/89933523:44
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic bugfix/22.1: Make sure we eject media from DVD when CD is requested  https://review.opendev.org/c/openstack/ironic/+/89933623:44
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic bugfix/22.0: Make sure we eject media from DVD when CD is requested  https://review.opendev.org/c/openstack/ironic/+/89933723:44
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic stable/2023.1: Make sure we eject media from DVD when CD is requested  https://review.opendev.org/c/openstack/ironic/+/89933823:44
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Add test for dhcp-less vmedia based deployment  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/89800623:56
opendevreviewJulia Kreger proposed openstack/ironic master: WIP/DNM: Advanced vmedia deployment test ops  https://review.opendev.org/c/openstack/ironic/+/89801023:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!