Friday, 2023-03-31

samuelkunkel[m]Good morning06:44
samuelkunkel[m]Would love to contribute to the ARM support / Support in CI - if you are looking for some helping hands06:45
samuelkunkel[m]Just saw the Call in the mailing list06:45
rpittaugood morning ironic! o/08:05
rpittauHappy Friday 08:06
fricklerthis looks like a race condition in unit tests, just mentioning in case someone wants to investigate https://zuul.opendev.org/t/openstack/build/6391a419f35f48efb6890a1118433fcb09:22
dtantsurfrickler: interesting, I haven't seen this one yet10:32
dtantsurI"m a bit puzzled how we even trigger that...10:33
opendevreviewDmitry Tantsur proposed openstack/ironic master: Do not log into the template1 database during test-setup  https://review.opendev.org/c/openstack/ironic/+/87914810:37
dtantsurfrickler: shooting in the darkness really, but ^^^10:37
kaloyankmorning everyone, is there something like a no-op option? I want to see what cleaning steps will be performed on node10:45
dtantsurI don't think so10:46
kaloyank:(10:52
dtantsurfg10:53
dtantsurehhmm10:53
kaloyankfollow up question: is it possible that I tell Ironic to only clean the root disk and leave any other disks alone?11:02
dtantsurI think we implemented that..11:20
dtantsurkaloyank: https://docs.openstack.org/ironic-python-agent/latest/admin/hardware_managers.html#devices-skip-list11:20
dtantsurnot literally what you're looking for, but may help11:21
iurygregorygood morning Ironic11:25
iurygregoryTGIF11:25
kaloyankI´ll give it a try. One more thing. I´ve set ´erase_devices_priority = 0´ and ´erase_devices_metadata_priority = 0´, shouldn´t these two options effectively disable erasing of drives other than the root disk?11:33
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: Report system firmware information in the inventory  https://review.opendev.org/c/openstack/ironic-python-agent/+/87904912:28
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: Add network interface speed to the inventory  https://review.opendev.org/c/openstack/ironic-python-agent/+/87915612:30
dtantsurthis ^^ should be all that BMO needs12:30
TheJuliagood morning13:32
TheJuliadtantsur: if you could take a glance at https://review.opendev.org/c/openstack/ironic/+/879060/2 and let me know if you want to scream and hide in the woods, that would be a good data point13:33
dtantsursure, but a bit later - fighting with hard links here..13:34
TheJuliaoh no worries13:34
TheJuliacould be monday13:34
* TheJulia is still waking up and still fighting her not so fun cold13:35
TheJuliastep wisse, if we're cool with them, I can rip them all out of my specs since they are cross cutting13:38
opendevreviewDmitry Tantsur proposed openstack/ironic master: Always fall back from hard linking to copying files  https://review.opendev.org/c/openstack/ironic/+/87916113:50
dtantsurTheJulia: same question re screaming ^^13:50
rpittaudtantsur: wondering if that could give issues with selinux, hardlink maintains context, copy may not13:53
dtantsuryeah, I've been wondering that (but note that the fallback actually existed before this change)13:54
dtantsurI would expect the target directory context to be used..13:54
rpittauright13:54
rpittauI think we fixed something similar recently, if I remember correctly13:54
TheJuliadtantsur: so the tldr is I think that is what I have wanted for a while on fallback13:56
TheJuliaYeah, copy should be new process rule context13:56
rpittaudtantsur: nvm, it was the other way around, copy should not cause issues as it indeed uses the destination context13:57
opendevreviewRiccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source  https://review.opendev.org/c/openstack/ironic/+/87916214:03
iurygregorydtantsur, do you think it's ok to start prototyping the FirmwareInterface?14:11
TheJuliafrom my pov, I suspect so. I *wouldn't* mind a little more delineation of the goals attempting to be achieved with the spec since we realized there was a whole set of thigns (hey, that is my method for pushing change!)14:16
TheJulia:)14:16
JayFsamuelkunkel[m]: so our needs are basically 2x: we need a CI job running that provisions ARM VMs and validates our agent on them.14:18
JayFsamuelkunkel[m]: the 2nd piece is we'd really, really like third-party CI to prove Ironic works on actual ARM hardware before we say "it works"14:18
JayFsamuelkunkel[m]: but even if you can only help us with the virtualized CI job, that's a huge help, and we can leave the call out for hardware still :D 14:18
opendevreviewRiccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source  https://review.opendev.org/c/openstack/ironic/+/87916214:19
kaloyank_TheJulia: I´m available to continue the discussion from yesterday :)14:20
TheJuliakaloyank_: first my wife wants to go get some breakfast, then I'll come back 14:21
samuelkunkel[m]JayF:  sure, sounds good. I help were I can find time. My PO accepted my request to help here so I am allowed to find some dedicated time. :)14:34
dtantsuriurygregory: frankly, I think we should change our approach to specs. Previously, there was a deliberate element of gatekeeping in them. Now it's purely a way to discuss designs, so there is nothing wrong (IMO!) with patches written and even merged while the spec is still being polished.14:34
dtantsurJayF, TheJulia ^^ opinions?14:34
samuelkunkel[m]As there was a general call for help, how is the process from now on? Will we wait how many people are willing to contribute and then assemble a Team?14:34
JayFdtantsur: iurygregory: Eh-ish? I mean, putting patches up = yes. Merging them before the spec I just have an overwhelming feeling of "what's the rush"?14:35
samuelkunkel[m]<JayF> "samuelkunkel: the 2nd piece is..." <- That is nothing I can say for sure currently. We only have a hand full of HPE RL300 Nodes for our own testing and playing currently14:35
samuelkunkel[m]But I ask If I can buy more ;)14:35
JayFsamuelkunkel[m]: even a contributor saying "I know it works on X physical hardware" is better than we have now (which is vague reports of it working)14:36
dtantsurJayF: I have an overwhelming feeling of why not :D Given how long we take to merge specs...14:36
JayFdtantsur: iurygregory: We should fix the last half of that piece. But like, I don't actually care, as long as any changes merged before the spec are not operator-facing/api-generating14:36
dtantsurWe should fix is not a plan how to fix :-/14:36
dtantsurSo far, we're punishing people for writing specs vs just trying to sneak their changes in14:37
JayFI guess I'm confused, I didn't know we had a big issue with getting specs merged14:37
samuelkunkel[m]JayF: This I can and will test for sure. (for the HPE RL300 we are using) but we are checking some Supermicro stuff out as well.14:37
samuelkunkel[m]Its just: I dont know If I can provide constant node(s) that is only for CI runs from a current standpoint. I need to check that.14:37
dtantsurJayF: for one reason or another, the last actual spec was merged a year ago: https://review.opendev.org/q/project:openstack%252Fironic-specs14:38
JayFsamuelkunkel[m]: third party CI is a pretty big commitment14:38
JayFdtantsur: that is weird, but I wonder if there are any https://review.opendev.org/q/project:openstack/ironic-specs+status:open that you think should be landed?14:39
JayFdtantsur: other than your inspector stuff, which I was actively reviewing until PTG (and am waiting for another contributor to review)14:40
dtantsur(okay, the shard one was merged recently - I stand corrected)14:41
dtantsurbut hey, even the shard one took 2 months to merge, and it's not really complicated14:41
jrosserJayF: i can say ironic is working with the ipmi driver on Supermicro MegaDC with the Ampere CPU in a R12SPD-A system board14:46
iurygregorysorry, got pushing into helping my parents, just got back 14:46
JayFjrosser: can you reply to the list with that specific detail so it's not lost in IRC logs forever?14:46
JayFdtantsur: that's mostly my fault for not being on top of it fwiw14:46
jrossersure14:46
JayFdtantsur: I don't ever feel reviews were a primary concern14:47
iurygregoryI think having the patches upda would help with the spec 14:47
dtantsuryeah, that's another thing14:47
dtantsur(note: I definitely don't suggest we merge API changes before properly discussing them... although I'm guilty of doing exactly that for inventory API)14:47
samuelkunkel[m]I am not finished with my final tests - but once I am I can also provide the specs. What List JayF ? :D14:47
iurygregorybut yeah, normally we take a long time to merge specs 14:47
dtantsurwe need to change this norm for sure14:48
JayFsamuelkunkel[m]: that mailing list thread about ARM support is a good place for things, but if you're actively going to work on setting up that CI this cycle, I'll add it to the workstream doc that I'll create documenting things we discussed in PTG14:48
JayFsamuelkunkel[m]: just grab an etherpad and start making notes, I deem you Ironic ARM subteam captain [taps shoulders with sword] lol14:49
dtantsurwow, fun, we have a spec that is on review since 2016: https://review.opendev.org/c/openstack/ironic-specs/+/18605614:52
samuelkunkel[m]JayF: Not sure If should be happy or scared though14:52
samuelkunkel[m]* sure If I should be14:52
JayFsamuelkunkel[m]: honestly, it's all about what you want to make of it14:53
JayFsamuelkunkel[m]: just know one person can make a huge difference, and getting ARM CI up would be huge14:53
JayFsamuelkunkel[m]: and not just because there'd be another brain I can recruit when the gate goes upside-down in any arch ;)14:53
kubajjdtantsur: if I implemented the python-ironicclient call to show (or whatever we want to do with it) the inventory, then it could be used with the openstack baremetal ... command?14:54
JayFdtantsur: it's ironic that one of my oldest specs is maybe actually going to merge (IPA comms) in some form14:54
dtantsurkubajj: correct. Please check my spec for the proposed format.14:54
dtantsurJayF: lol, true14:54
dtantsurI'd really want some old specs to merge.. https://review.opendev.org/c/openstack/ironic-specs/+/501511 for instance14:55
sschmittSpeaking of specs, is there a good starting point/template if I wanted to submit one for the more granular neutron port binding I brought up last week (i think)14:55
JayFI've wanted that, as an operator, since upstream-cleaning existed14:55
JayFmight have even been a "fast follow" item after we landed it that never got done because of the technical headaches14:56
JayFand at the time, our general unwillingness to accept an 80% OK solution14:56
dtantsuryeah, I've become much more tolerant to "80% OK" things14:57
TheJuliadtantsur: given how many times we’ve gone “let’s debate that in code”, I do agree. The value is ask the questions, get the ideas out. I do think lowering the merge threshold makes sense. Hating on all positive agreement… just elongates the process.  Doing so though means we need to reframe things a little.14:58
dtantsurTheJulia: I'm curious if you're still interested in https://review.opendev.org/c/openstack/ironic-specs/+/56015214:58
JayFI will say, regardless of why it was created, etc, specs are a super useful tool for clarifying thinking about a feature14:58
JayFI've taken our spec template downstream to fill out for non-Ironic things before, just because the structure makes it easy to ensure you're thinking of more edges in your idea14:58
TheJuliadtantsur: me, meh. Others surely yes14:59
TheJuliaJayF: I agree completely, but it is a giant gate keeping mechanism as well. And specs are always a “this was the discussion”, reality might be a little different15:00
JayFI think for specs, we need to get the API interfaces to like, 90-95% 'right', everything else just get in the ballpark and go15:00
TheJuliadtantsur: perfection is the enemy of good…15:01
JayFthe time is long gone where we'd bikeshed over implementations if there weren't material tradeoffs inovlved15:01
JayFlike, lets put a point on this: is there anything, right now, that needs the spec landed b/c it's blocked, or there's a perception it's blocked?15:03
JayFif ^^ that ever happens, please bring it up loudly, on the list, in the meeting, anywhere. I'll make it a huge priority to review it within 24 hours and we can hope other contributors would also give it priority15:03
dtantsurTheJulia: you say, the hold patch works? I'm quite puzzled by it..15:05
TheJuliadtantsur: works for unit tests15:05
dtantsurFeels like it messes up the states somewhat..15:05
dtantsurright15:05
dtantsurTheJulia: you expect DEPLOY WAIT -> DEPLOY HOLD -> DEPLOY WAIT.15:05
dtantsurI bet (and it makes more sense) it's actually DEPLOYING -> DEPLOY HOLD -> DEPLOYING15:06
TheJuliaThat is what I thought at first as well15:06
TheJuliaBut then the methods went “you can’t call this in that state15:06
dtantsurhmm, to make it actually work, probably DEPLOYING -> DEPLOY HOLD -> DEPLOY WAIT15:06
TheJuliaSo deploy wait since it is async15:06
dtantsuryeah, but you're calling process_even('hold') from a place where the node is in an 'ING state15:07
dtantsurlemme better comment15:07
TheJuliaYeah, seemed weird when I was doing it15:08
TheJuliaAnd cold meds15:08
dtantsurTheJulia: yeah, that's tough :( anyway, left some comments. nothing critical, except for this issue.15:11
TheJuliaYeah, i need to go back and look at the state transitions around steps because at first I thought everything would be ING, but then it wouldn’t work as is and I realized the state past an executed step was wait15:12
dtantsuronly if the step handler returns *WAIT15:13
TheJuliaHmm, so both are needed then15:14
TheJuliaI was suspecting that would be the case yesterday15:14
JayF-ING -> HOLD vs WAIT -> HOLD might be the difference between oob step -> hold vs in-band step -> hold15:17
JayFI wonder15:17
TheJuliaIt is, in theory15:18
* TheJulia has consumed breakfast and now wants a nap15:19
dtantsurNot really. If I recall it right, we always return to ING before analysing what is left15:21
JayFif that's behavior, it's newish behavior15:23
dtantsurJayF: https://opendev.org/openstack/ironic/src/branch/master/ironic/conductor/manager.py#L1227-L123615:26
dtantsurwe expect nodes to be in WAIT state while an async step is running, but we run "resume" before further processing15:26
dtantsurthis is not really new15:26
JayFin... Ocata I never observed it pass thru -ING in those cases15:27
JayFbut it was ocata and that was approximately two lifetimes ago15:27
dtantsuryep, newer than ocata is not necessarily new :)15:27
JayFyeah I didn't realize it was that old of an observation until I went to put a release name on it lol15:28
opendevreviewRiccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source  https://review.opendev.org/c/openstack/ironic/+/87916215:47
TheJuliaJayF: how did you navigate https://github.com/openstack/ironic/blob/abbd859b766c339f5de33ff08704a7b9ad045bef/ironic/drivers/modules/agent.py#L580 downstream ?15:57
JayFTheJulia: we were using kickstart driver for all deployments15:58
JayFTheJulia: so basically I didn't have to navigate agent imaging at all, just make sure cleaning worked with hb-only15:58
TheJuliaAhh15:58
TheJuliaRight15:58
TheJuliakaloyank: fyi ^16:01
JayFTheJulia: how many of those places exist, I wonder?16:05
JayFTheJulia: if not many... we could just preemptively send the info as part of lookup?16:05
kaloyankaren´t partitioned disks already deprecated?16:05
TheJuliawe still have to get the IDs for later commands16:05
TheJuliaand partition images are not deprecated16:06
kaloyankO.o16:06
TheJulia16 places where we directly invoke agent client calls in agent_base.py16:06
TheJuliafinalize rescue.. which means credentials have to get downloaded again16:07
TheJuliarefresh_steps16:07
TheJuliaexecution of steps16:07
TheJuliaagent time sync, telling the agent to power off16:08
TheJuliaand bootloader installation16:08
JayFooooof16:12
TheJuliato do it all one way, we would have to intercept each call, build buffer of commands to convey, and return to that point in the sequence16:12
JayFso basically when doing this downstream and only using kickstart driver, I punted the hardest part16:12
TheJuliayeah16:12
JayFTheJulia: or put the logic into the conductor about what steps are next and have it ask in advance16:13
JayFTheJulia: sorta like mis-in-place for cooking16:13
JayFI know that's not possible for all cases, but might be for some16:13
TheJuliaJayF: we already do that, but because steps can cause next steps to change, we have to ask it again16:13
JayFmakes sense16:14
JayFCould we move more of the logic directly into the agent in some of the cases we're doing round trips from conductor?16:18
TheJuliaat least the rescue password is hashed now so there is less concern about allowing the agent to just get it16:18
* JayF just brainstorming feel free to ignore/not engage if you're doing actual-work16:18
JayFI'm TC-PTG distracted16:18
TheJuliaI think we could possibly, but we're going to need to have a compatability window as well16:18
TheJuliaThat overlap is the hard part16:18
JayFwe can require you run min(ironic, ipa) of a given version to get access to hb-only mode16:19
JayFwhich means you could use old logic for old path; new logic for new path, yeah?16:19
JayFI guess if you felt strongly motivated to DRY it could get messy16:19
TheJuliabasically yeah, this is in the someone likely just needs to go off in a corner and hack out territory unfortunately16:19
* TheJulia feels super tired and thanks the cold for this feeling16:22
JayFI am video-meeting'd-out right now. Trying to make it thru one more day then weekend, then one more week then it's vacation time (I'm out all of the week-after-next)16:24
rpittauTheJulia: hope you'll feel better during the weekend :)16:24
rpittauhave a great weekend everyone! o/16:24
TheJuliadtantsur: w/r/t ironic/common/states.py and allowing abort, do you think abort to just abort out of cleaning?16:25
JayF++ hold step is abortable, and aborting it should act like abort does in all other cases (yeah?)16:25
dtantsurTheJulia: we allow aborting CLEAN WAIT, so it feels natural to allow it also for CLEAN HOLD16:25
dtantsuryep16:25
JayFIMO the only steps we ship that should not be abortable are ones that break the machine or leave it in a bad state if we stop (or if we just can't stop)16:25
JayFso pause? abortable. hold? abortable.16:26
TheJuliaack16:27
opendevreviewRiccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source  https://review.opendev.org/c/openstack/ironic/+/87916216:27
TheJuliaokay16:27
* TheJulia lays down for a little while16:29
JayFI just gave a positive RSVP for Ironic for the PTG in Vancouver, with a request for us to have some kind of way to pull in remote folks.16:41
JayFI know we likely won't have a quorum unless we can include remote folks16:41
JayFI (once again) forgot to schedule a vPTG team photo :( sorry folks17:13
dtantsurwith at least 2 people sick, it could be better to skip it actually..17:35
dtantsurhave a nice weekend folks, quick recovery to everyone with cold!17:36
TheJuliaUgh, i was feeling so much better yesterday evening :(18:15
JayFmy wife has had a cold all week too. I'll give you both the same advice: go sit down and rest and don't do work :)18:16
* TheJulia stumbles towards the espresso machine18:18
TheJuliawife intercepted me and has convinced me to lay back down18:29
TheJuliamy inner workahaulic does not like this18:30
JayFShe is a very smart lady :D18:30
samuelkunkel[m]Will there be a Ironic PTG Session in Vancouver?18:44
samuelkunkel[m]Do you say „a“ ironic or „an“ ironic? Both fit ;)18:44
JayFI would probably say "an" but nobody is going to care about minor language nits :) 18:47
samuelkunkel[m]„We have an ironic session“18:47
JayFI am planning for Ironic to have space to discuss at the PTG in Vancouver. It's my expectation that we will not have enough core contributors to get "quorum" on important decisions unless we have a virtual option.18:47
JayFthere will be a separate vPTG at the start of "C" release cycle as well18:48
samuelkunkel[m]Sounds good18:48
samuelkunkel[m]Is there already a timeslot? I assume its after the regular schedule?18:49
samuelkunkel[m]Like „on Friday“18:49
JayFSignups for teams to get into PTG are open until April 5, per the email that hit the list recently18:49
JayFso I would assume the schedule won't be around for a while18:50
JayFI will be in Vancouver for the entire week.18:50
samuelkunkel[m]I think I arrive on Sunday and stay till saturday, so same for me18:50
samuelkunkel[m]If you add 11-12h flight I am also basically the a whole week on tour :/18:51
samuelkunkel[m]* If you add 11-12h flight oneway I am also basically a whole week on tour :/18:51
JayFI'm arriving sometime on Monday. I'm driving up from Tacoma, WA so I have about 4 hours on the road, unless the border checkpoint is backed up18:51
TheJuliaI’m thinking of landing on Sunday and I may end up spending an extra week in yvr if the wifey gets her passport back in time18:54
JayFnice18:56
samuelkunkel[m]Thats cool18:56
JayFy'all are always welcome to swing by on the way back down and I can smoke something tasty18:56
JayF(assuming the new-less-dark bus is how you're getting there?)18:57
* JayF afk for a while18:57
TheJuliaJayF: dunno yet. We really haven't decided19:09
* TheJulia drinks coffee with a straw because... because.19:10
iurygregoryI love FW related bugs .-. people say after they updated to a specific version something is not working, I tested in the same type of machine with the FW they mentioned and couldn't reproduce the issue .-. 19:30
TheJuliaOh those are the best bugs19:36
TheJuliaMy next favorite ones which are obvious kernel issues19:36
iurygregoryI have 3 bugs related to BIOS Settings 2 seem to be related to using new iLO FW that is providing empty description of the attributes :D19:41
TheJuliasweet!19:44
* TheJulia for some reason has forgotten how to add columns to the db19:44
iurygregoryupgrade existing table with new column using alembic?19:46
TheJuliayeah19:50
TheJuliaremembered alembic19:50
TheJuliabut not having luck on unit tests19:50
TheJuliaoooh ahh invalid syntax now19:51
iurygregory\o/19:54
TheJuliaoh, I think the issue was I was trying to use an integer when... really, I should have just used a string due to orm. lets see!20:24
TheJuliagah, it was the way the test is setup20:39
opendevreviewJulia Kreger proposed openstack/ironic master: Add jsonrpc port to database  https://review.opendev.org/c/openstack/ironic/+/87921022:14
opendevreviewJulia Kreger proposed openstack/ironic-lib master: Add jsonrpc client port capability  https://review.opendev.org/c/openstack/ironic-lib/+/87921122:51

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!