Monday, 2023-12-04

Nisha_Agarwalmorning ironic!!!05:01
masgharMorning Nisha!06:53
masghariurygregory: Thank you for explaining! I meant building an ironic image off if openstack-ironic06:53
masgharTheJulia: Will add that release note. I am trying to fix that the volume name (logical disk name) that ironic takes as input from the bmh spec (spec.raid.hardwareRAIDVolumes.name) is being discarded by its redfish raid 06:58
opendevreviewMohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system  https://review.opendev.org/c/openstack/sushy-tools/+/87536607:06
opendevreviewlikui proposed openstack/networking-baremetal master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/networking-baremetal/+/90254807:12
opendevreviewlikui proposed openstack/networking-generic-switch master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/networking-generic-switch/+/90254907:13
opendevreviewMahnoor Asghar proposed openstack/ironic master: Add volume name to Redfish RAID volumes  https://review.opendev.org/c/openstack/ironic/+/90244610:11
opendevreviewMohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system  https://review.opendev.org/c/openstack/sushy-tools/+/87536610:11
iurygregorygood morning Ironic11:17
zigoTheJulia: The ship sailed because on every Python release, eventlet breaks hard, and it takes forever to have the patches upstream. It's just a real pain to maintain the package ...12:53
opendevreviewDmitry Tantsur proposed openstack/ironic-specs master: Boot configuration API  https://review.opendev.org/c/openstack/ironic-specs/+/90257414:26
dtantsurJayF: ^^14:26
JayF#startmeeting Ironic15:00
opendevmeetMeeting started Mon Dec  4 15:00:39 2023 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
iurygregoryo/15:00
JayF#topic Announcements/Reminder15:00
JayF     Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash15:00
dtantsuro/15:01
JayFNo action items from previous meeting, skipping agenda item15:01
JayF#topic Carcal release schedule15:01
JayF#link https://releases.openstack.org/caracal/schedule.html15:01
JayFNext milestone is R-17, Caracal-2 on Jan 11, take notice :)15:01
JayF#topic Review Ironic CI Status15:01
JayFHow is CI?15:01
iurygregoryI think is in a good shape15:02
dtantsurmy changes have passed recently15:02
iurygregorylast week we were able to merge a few things without problems from what I remember15:02
JayFThat matches my experience, nice to hear, thanks for keeping the gate clean.15:03
iurygregorywe finally moved the snmp job from focal =) tks TheJulia 15:03
JayF#topic Bug deputy15:03
JayFJulia was bug deputy last week, and we did some cleanup of bugtrackers15:03
JayFTheJulia: anything to report from time as bug deputy?15:03
iurygregoryJulia is not available today15:03
dtantsurI think she mentioned she wouldn't be around15:03
JayFack15:03
iurygregoryyeah15:04
JayFHopefully it went well, is someone signed up for enxt week?15:04
iurygregoryafaik nope15:04
JayFAight, I'll take a swing15:04
iurygregoryyou mean this week right?15:04
JayF#action JayF to bug deputy this week15:04
JayFyep15:04
iurygregorygotcha!15:04
JayFIs this bugfix branch topic leftover from last week?15:04
JayF#topic New bugfix branches proposed15:05
iurygregoryI think rpittau was doing something related to this, but he is out today15:05
JayFAck; we'll table the topic15:05
JayF#topic RFE Review15:05
JayFdtantsur has two15:05
iurygregory:O15:05
JayF#link https://bugs.launchpad.net/ironic/+bug/204554815:05
dtantsurI do indeed15:05
JayFdtantsur: is the real only use case for this dual stack ip?15:06
dtantsurJayF: for me - yes15:06
JayFdtantsur: basically I'm a little worried this is going to end up as "Ironic API HA via 'yolo try all the api servers'" rather than properly load balanced15:06
dtantsurIsn't properly the same?15:06
JayFoh, I modeled this backwards15:06
JayFI was mentally thinking of this as "N" callback urls *from* IPA15:07
JayFbut this is "N" callback urls *for* IPA to callback to15:07
dtantsurYeah, so IPA->Ironic. My personal pain is less about HA, more about a dual-stack Ironic with a V6-only node.15:07
* iurygregory hides 15:08
JayFI don't have any objection to this directly; but I also think if rpittau and TheJulia is out that we don't really have strong quorum for RFE approval15:08
iurygregoryI know your pain dtantsur 15:08
JayFif you know one or both of them are +1 or at least +0 to it, I'm OK with it15:08
dtantsurI can ask them later on. Putting it on the meeting means that they can at least see that in the scrollback.15:08
JayF++15:08
dtantsurLet's treat this as a sanity check / announcement of intentions :)15:09
JayF#info Bug 2045548 seems ripe for RFE approval, but no strong quorum of cores at meeting. Intention is to move forward if no objection once more cores look.15:09
dtantsurThe next one is more interesting, I believe15:09
dtantsur#link https://bugs.launchpad.net/ironic/+bug/2045551 Auto-configuration of iPXE script for inspection15:10
JayFSo there's two things that confuse me about this: 15:10
JayF1) is it still unmanaged inspection if we're generating PXE scripts around it in Ironic? hehe15:10
JayF2) when we talk about fallback scripts, is that like, generating an ipxe config which would boot anything if put in the correct spot? what is the correct spot?15:11
dtantsurThe current logic in boot.ipxe is roughly this:15:11
dtantsurfor mac in macs: try_chainload(http://server/$mac/boot.ipxe); chainload(http://server/fallback.ipxe)15:11
dtantsurSo, this last item is where inspection is normally hooked into, at least for standalone deployments15:12
JayFbasically "boot any specific configs we have, boot the generic config for anything else" (which in a default config is nothing/chainload to hdd I would assume?)15:12
dtantsurCorrect15:13
dtantsurSo far, this fallback script has to be hand-written (see examples from metal3; bifrost has the same)15:13
JayFIs there a difference from what we document today as "unmanaged inspection" and "discovery" ?15:13
dtantsurTechnically the same thing, only the intention is slightly different15:14
JayFack15:14
JayFmy only real weirdness about this feature15:15
JayFis we can't really enable it on *all* conductors on a given dhcp domain, right?15:15
JayFhmm. I guess we could, because it's just an ipxe script15:15
JayFit's just available should the dhcp server point at it15:15
dtantsurRight. DHCP is a different story.15:15
JayFdhcp is [magic word] UNMANAGED in that case, right? lol15:16
dtantsurRight :D15:16
JayFSo the feature seems OK15:16
JayFbut I kinda hate the name?15:16
JayFI know it fits in ironic contexts, but I don't think it signals to operators what the behavior change will be15:16
dtantsurwhich exactly name?15:16
JayFso my only thing would be, lets make 100% sure it's clear that if someone flips that switch what they're going tobe getting15:17
JayF>  [pxe]generate_ipxe_inspection_script = True/False (default False)15:17
JayFlike, I feel like we could use a word like ... "universal" 15:18
dtantsurYep. (Also, naming suggestions welcome)15:18
JayFor "fallback"15:18
JayFor "global"15:18
JayFor "all of the things"15:18
JayFI don't have an actual good suggestion but you get the idea :D 15:18
dtantsurThis thing will be inspection-specific15:18
dtantsurIt's not like a fallback to something abstract15:18
JayFthat means ... less in a future world where everything is Ironic-based :D15:18
dtantsur:D15:18
JayFSo I think similar adjudication for this, probably OK, lets get more feedback and maybe pick a stronger name?15:20
iurygregorywe probably need a shared doc with good naming ideas lol15:20
dtantsur:D15:20
dtantsurJayF++15:20
JayF#info Bug 2045551 seems ripe for RFE approval, but no strong quorum of cores at meeting. Intention is to move forward once a better config var name is chosen and if no objection once more cores look.15:20
JayFdtantsur: please update those RFE bugs with the comments from the meeting (link to the meeting logs once we get ti closed up would be great)15:21
JayF#topic Open Discussion15:21
JayFNisha_Agarwal: mallik: You have a topic here15:21
Nisha_AgarwalHi Jayf15:21
Nisha_Agarwalyup15:21
mallikHi JayF15:21
JayFSo first of all, the storyboard issues created; please create them in launchpad. We no longer use storyboard and I'm confused as to how new issues were created there.15:22
JayFhttps://bugs.launchpad.net/ironic15:22
Nisha_AgarwalWe are testing the redfish driver with ilo615:22
Nisha_AgarwalJayF, ok15:22
Nisha_Agarwalthere are two features where we see the issue15:22
Nisha_Agarwalone is Event Scusbscription15:22
Nisha_Agarwalilo6 is on higher redfish schema15:22
Nisha_Agarwalwhile ironic is using the schema where attributes are deprecated15:23
* iurygregory was the one to write the event subscription via vendor passthru15:24
Nisha_AgarwalSo sushy needs to enhance the redfish schema implementation and may be intelligently derive which schema version does the server has....15:24
Nisha_Agarwalilo5/SuperdomeFlex280 works with current implementation of event Subscription15:24
Nisha_Agarwalwhile ilo6 has higher schema implemented15:25
dtantsurSo, ilo6 removes the deprecated fields completely?15:25
iurygregoryI think we would normally add the new attributes to sushy to match the new schema and keep backward compatibility I think15:25
Nisha_AgarwalLooks like15:25
Nisha_Agarwaliurygregory, yeah that should solve the issue....15:25
dtantsurSigh, okay. Patches are welcome if you have cycles (esp. since you can test on both hardware versions)15:25
Nisha_Agarwaldtantsur, yup. We are working on the patches15:26
iurygregoryNisha_Agarwal,  would also need to update the vendor passthru logic in ironic after sushy is ready15:26
Nisha_Agarwaliurygregory, ok. 15:26
Nisha_AgarwalWe will test end-to-end using ironic15:27
iurygregorynice!15:27
Nisha_AgarwalSecondly on out-Of-Band Raid15:27
Nisha_AgarwalIronic implements two parameters "VolumeType" and "Encrypted"15:27
Nisha_Agarwalin addition to other parameters15:28
Nisha_AgarwalVolumeType is deprecated and is not present in ilo615:28
dtantsurmasghar: ^^^15:29
dtantsurNisha_Agarwal: what do you use instead for the RAID level?15:29
Nisha_Agarwal"Encrypted" is not deprecated, but iLO6 has cautiouslu not impelmented it. The Storage Distinguished Technoligist has told that very few vendors have impelmented Encrypted15:30
Nisha_Agarwalcautiously^^^15:30
Nisha_Agarwaldtantsur, it is deprecated in favor of "RAIDType"15:30
dtantsurIt seems that we unconditionally send Encrypted:False. Not sure why, but I suspect we can just remove that.15:30
dtantsurAh. And we already pass RAIDType too. Again, I'm curious why we did that.15:31
Nisha_Agarwalso i believe it should work with that payload. But i need to crosscheck with the engineer who is working on it15:31
masgharRedfish driver, right? (just confirming)15:32
Nisha_AgarwalYes15:32
iurygregorymasghar, correct15:32
dtantsurmasghar: can you check if your Dell machine has RAIDType?15:32
Nisha_Agarwalif we remove these two parameters the creation of RAID works15:32
masgharLet me see15:32
Nisha_AgarwalSo we wante dto know if we could add a condition in redfish driver (sushy) specific to Proliants?15:33
Nisha_Agarwalthat would be allowed right?15:33
iurygregorythis wouldn't be a problem I would say15:34
dtantsurNisha_Agarwal: this is not immediately disallowed, but for what purpose?15:34
dtantsurif VolumeType is deprecated and Encrypted is unused, we just drop them.15:34
JayFYeah, that is kinda a last resort if we can't use non-vendor-specific ways to figure out the right thing to do15:34
Nisha_Agarwaldtantsur, ok.15:35
Nisha_Agarwalthere is one power bug also which was raised on ilo5. But it applies to ilo6 as well15:35
Nisha_Agarwalthat was raised by Gresearch i guess15:36
iurygregoryhttps://bugs.launchpad.net/sushy/+bug/2016307 ?15:36
JayFYeah, it's already been fixed, and the backport is back pretty far already15:36
Nisha_Agarwalthere it requires vendor specific check probably in sushy15:36
Nisha_Agarwaliurygregory, not this15:36
iurygregoryoh ok, I've found this one when looking in launchpad15:36
JayFhttps://opendev.org/openstack/sushy/commit/192d897ce97120db98aca6d6bb840c4243a4b5b3 is the fix I was thinking of15:37
Nisha_Agarwalhttps://bugs.launchpad.net/ironic/+bug/202199515:37
Nisha_Agarwalyes the fix done above was partial15:37
Nisha_Agarwaland doesnt solve the above issue15:37
Nisha_Agarwali had done a workaround fix in proliantutils for ilo515:38
Nisha_Agarwalbut it still requires a small fix in sushy to have that workaround available for ilo6 as well15:38
Nisha_Agarwaland probably for other vendors too15:38
Nisha_Agarwalthis may require vendor specific check is what i wante dto put forward15:39
JayFStuff like that is best evaluated in gerrit on the patch IMO15:39
dtantsurI'd have to see the patch to judge15:39
dtantsuryeah15:39
Nisha_Agarwaldtantsur, ok15:39
JayFIs there anythign else to discuss on this topic?15:40
Nisha_AgarwalJayF, one more point about the mail thread on driver deprecation15:40
JayFsure15:40
Nisha_Agarwalthe spec raised for this release item lists ilo driver under the heading "Drivers to be removed"15:41
Nisha_Agarwalwhich looked incorrect and may discourage customers to use ilo/ilo5 driver for gen9/gen1015:42
JayFso two things, firstly, it doesn't say that15:42
Nisha_AgarwalThis was a concern raised by my management15:42
JayFit says: "Drivers to be marked for removal"15:42
Nisha_Agarwalso just putting forward the concern15:42
JayFalso in the paragraph above, it says: "We do not intend to actively remove any of these drivers until it is clear any hardware they exclusively support has gone end of life. We are primarily taking this action to indicate to operators that they should be provisioning new hardware with Redfish-based drivers."15:42
Nisha_Agarwalyup15:42
JayFSo, the context behind why we worded it this way, and put it so strongly 15:43
JayFand there's willingness to change it as long as it achieves this goal15:43
JayFis we've seen the branded-drivers be very attractive to new operators of ironic15:43
JayFso we'll have users choose to use an ilo driver on their ilo6, or a idrac{-wsman} driver on a newer dell15:43
JayFeven though for those devices, a redfish driver is a better choice, because they've heard of "idrac" or "ilo" but have never heard of "redfish"15:44
Nisha_Agarwalhmm15:44
Nisha_Agarwalok15:44
JayFI'll note that for many drivers, we already have it printing a deprecation message -- we didn't put that in for iLo15:44
Nisha_AgarwalThanks JayF that answers the query :) yes thanks for that15:44
JayFand I thought it'd be worse for you all to leave you off that list entirely; I don't want HPE users to be completely lost on what driver to use15:45
Nisha_Agarwali saw the mail, but still i wanted to put forward the message15:45
JayFso if there's suggestions on how to reword that, I'm happy to take edits to that spec, nothign is ever set in stone15:45
JayFbut the goal was to funnel new ironic operators to redfish driver by default15:45
Nisha_AgarwalJayF, that should be great15:45
Nisha_AgarwalI will check with management and reword if they are not ok with what you explained above15:45
Nisha_AgarwalThanks JayF 15:46
JayFno problem o/15:46
JayFAnything else for open discsusion?15:46
Nisha_Agarwalthats all15:46
iurygregoryI have one15:46
JayFWhat's up iurygregory 15:46
iurygregoryit's more a request than a discussion LOL15:47
iurygregoryrelated to how we could speed-up multipath checking in IPA I've created https://review.opendev.org/c/openstack/ironic-python-agent/+/90201215:47
JayFGotta be honest, I've not looked at that much because I don't have very much experience with multipath IO15:48
JayFso I'm a little scared I'd approve something that'd break the underlying tech unknowingly15:48
iurygregoryis just my initial idea in the code, the commit explains what I'm trying to do to speed-up15:48
iurygregoryyeah, so if anyone is interested please look and provide your thoughts in the patch \o/15:49
JayF#link https://review.opendev.org/c/openstack/ironic-python-agent/+/90201215:49
JayFI'll look, but as I said, take any comments I put on it as advisory15:49
iurygregoryJayF, sure =)15:49
JayFI'm not sure I've ever, once, been on a machine with multipath remote IO15:49
JayFAnything else for Open Discussion?15:50
iurygregoryI don't have one, this is just by looking at ironic logs and looking at our logic for multipath 15:50
JayFmakes sense15:50
iurygregorytks!15:50
JayFLast call?15:51
JayFty all o/15:53
JayF#endmeeting15:53
opendevmeetMeeting ended Mon Dec  4 15:53:06 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:53
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-12-04-15.00.html15:53
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-12-04-15.00.txt15:53
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-12-04-15.00.log.html15:53
JayFdtantsur: re: https://review.opendev.org/c/openstack/ironic/+/900568 I'm unsure how we guard it there16:08
JayFdtantsur: historically, afaict, we have not guarded schema changes for local_link_connection behind a microversion16:08
dtantsurJayF: two schemas depending on the API version16:08
dtantsurI don't we've ever expanded it though?16:08
JayFLOCAL_LINK_SMART_NIC_SCHEMA16:08
JayFwhen we added node.is_smartnic16:09
JayFwe expanded that schema, and we hid some of the other fields16:09
JayFbut none of the port changes16:09
dtantsurSigh16:09
JayFif it needs to be guarded, it can be, but I'm just weirded out by how we guard it16:09
dtantsurWell, what I suggest is what we as the project have committed to16:09
JayFdo we hide the response if it matches schema? (expensive?)16:09
JayFdo we only accept if a new enough microversion (likely doable)16:09
dtantsurIt should look like the old schema is used with old API versions16:10
JayFif we only wanna accept on high enough microversion, that's doable. only readable on high enough microversion is probably not doable16:10
dtantsurand yet, that's how it should work16:10
JayFshould with a project that is over a decade old is a funny word16:10
dtantsurand that's what we do with other resources btw: we yank the fields16:10
JayFyeah the thing that's weird here16:10
JayFis we assert "it's only valid if you have a set of (x,y,z)"16:11
JayFso we can't pull (x) without making it invalid16:11
dtantsurthis is annoying indeed.16:11
JayFso essentially, I think the implementation of this will be "return an empty dict/none if any of the ovn vtep values are set"16:11
dtantsurbut imagine how any node looks like with the new-style drivers16:11
dtantsurand no *_interface fields16:11
dtantsurA node cannot exist with boot_interface=None16:12
JayFnot very useful, I'd imagine :D 16:12
JayFoooh16:12
JayFa port *can* exist with a local_link_connection of None, fwiw16:12
JayFwe do not require the field16:12
JayFI will wanna check for patterns in our code (maybe ngs/nw-bw?) that might get port -> do action, to make sure it won't be too breaky16:13
JayFbut I think we can filter on the way out with that in mind16:13
dtantsurJSON fields with interdependent subfields is where microversioning gets really hairy really quickly16:13
dtantsurI don't have a good suggestion, to be honest. We either treat local_link_information as we treat driver_info and such16:14
dtantsuror we treat it as a fully versioned resource16:14
JayFlocal_link_informaion is driver_info for ports16:14
JayFin terms of how we use it16:14
JayFwe more or less take anything in it and put it in neutron's binding profile16:14
JayFfor neutron-based networks16:14
JayFI like that we schema check it, it seems like an improvement over driver_info16:15
JayFbut it *is* weird in that like, this is a feature we're only writing because this schema exists16:15
JayFotherwise someone just chucks stuff into that field and it works16:15
dtantsuryeah, that would actually be a problem for SDK's not in Python..16:21
opendevreviewMerged openstack/ironic master: Add volume name to Redfish RAID volumes  https://review.opendev.org/c/openstack/ironic/+/90244616:31
opendevreviewDmitry Tantsur proposed openstack/ironic master: First pass at the new in-band inspection docs  https://review.opendev.org/c/openstack/ironic/+/89823717:40
dtantsurJayF: ^^ addressed your comments while also finishing the hooks part17:41
dtantsuryou'll see what I don't want a table there :)17:41
*** awb_ is now known as awb18:14
*** seebaer is now known as seba19:26
*** edebeste0 is now known as edebeste19:26
opendevreviewJakob Meng proposed openstack/sushy-tools master: Use existing nvram path in libvirt driver to support session daemon mode  https://review.opendev.org/c/openstack/sushy-tools/+/90241820:25
opendevreviewJay Faulkner proposed openstack/ironic master: DNM: Testing Ironic against OS SDK changes  https://review.opendev.org/c/openstack/ironic/+/90260423:16

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!