Monday, 2023-07-31

opendevreviewMerged openstack/ironic master: Log when a periodic is completed  https://review.opendev.org/c/openstack/ironic/+/88850300:13
opendevreviewMerged openstack/ironic master: DB: Select upon delete for allocations  https://review.opendev.org/c/openstack/ironic/+/88989000:38
opendevreviewlikui proposed openstack/python-ironicclient master: These should be indented 4 spaces to match the other things in this block  https://review.opendev.org/c/openstack/python-ironicclient/+/79469107:03
opendevreviewlikui proposed openstack/python-ironicclient master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/python-ironicclient/+/89005007:16
rpittaugood morning ironic! o/07:16
opendevreviewlikui proposed openstack/ironic master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic/+/89005107:19
opendevreviewlikui proposed openstack/ironic-inspector master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-inspector/+/89005207:25
opendevreviewlikui proposed openstack/ironic-python-agent master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-python-agent/+/89005307:26
opendevreviewlikui proposed openstack/ironic-prometheus-exporter master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/89005407:28
kubajjGood morning Ironic o/07:38
masgharo/08:09
opendevreviewRiccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build tinyipa with tinycore 14.x  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/88775408:11
iurygregorygood morning Ironic11:19
dtantsurOh, how much I love turning off my IRC bouncer during vacation: no pings, no scrollback to read! Looking at you, Slack!!11:20
dtantsurIn other words, good morning folks, I'm back :)11:20
kubajjdtantsur: good morning11:23
dtantsurhey-hey kubajj, long time no see11:23
kubajjdtantsur: Yeah, exams got me a bit busy and then volunteered at a scout camp but now I'm back11:24
dtantsur\o/11:26
opendevreviewJacob Anders proposed openstack/ironic master: [WIP] Wait for secure boot state change if it's not immediate  https://review.opendev.org/c/openstack/ironic/+/86399911:28
kubajjdtantsur: I'll be working on ironic part-time, mostly looking into https://specs.openstack.org/openstack/ironic-specs/priorities/2023-2-workitems.html#improving-deploy-kernel-ramdisk-config but can also help with anything that could help with the inspector11:28
dtantsurI think the kernel/ramdisk is a good and useful action items.11:29
dtantsurI need to wrap my head around the recent Inspector development myself (need to sync with masghar)11:29
opendevreviewMerged openstack/ironic master: Add the initial skeleton of the agent inspect interface  https://review.opendev.org/c/openstack/ironic/+/87781411:51
JayFkubajj: you know, you can loudly state to dtantsur and others that you work with me :D kubajj is part of the GR-OSS team now helping us out for a while14:14
dtantsurw000t!14:14
JayFdtantsur: in the realm of things you might know/care about14:14
JayFdtantsur: I might have an opportunity for an MLH intern (think: outreachy but different brand and coming thru GR-OSS channels, not openstack)14:15
JayFdtantsur: Curious if you had ideas on a project, because I wanted to nail that down before I said yes :D 14:15
opendevreviewDmitry Tantsur proposed openstack/ironic master: Fix minor grammar issues in the help for new inspector options  https://review.opendev.org/c/openstack/ironic/+/89013814:16
dtantsurJayF: ^^^14:16
dtantsurfollow-up you asked for (funnily, these constants are not used until the next patch)14:16
JayFyeah I just didn't want the grammar fix to get lost; +214:16
dtantsurJayF: how ambitious should the project be? I have quite a few ideas (or at least I had before my brain was reset during the vacation lol)14:17
dtantsurall the way up to IPv6 support in Bifrost ;)14:17
JayFdtantsur: you've worked with that style of intern before; I usually try to make the project easily achievable and a gateway to further stuff if they knock it out14:17
dtantsurto rephrase my question: which level will the interns have and how much time (both per week and totally) are they going to spend?14:18
JayFyou know, bifrost is fertile ground14:18
JayFv6 support is probably very fiddly for an intern14:18
JayFmaybe similar for neutron integration14:18
dtantsurBIOS settings / RAID support. Actually, any manual cleaning support in the Bifrost playbooks.14:20
dtantsurSomeone recently complained that we don't support node.network_data (as opposed to configdrive) in Bifrost. Can be a small "try it" work item.14:21
JayFplus I feel less bad tossing someone at a bifrost dev item14:21
JayFbecause the dev env is less painfail14:21
JayFyou know, that's the most hilariously perfect typo of painful to describe devstack14:21
dtantsur:D14:21
dtantsurJayF: something people keep asking about: support for symlinks in root device hints14:22
dtantsurso that they can use /dev/disk/by-whatever/whatever instead of /dev/sda14:22
JayFI was surprised at first we didn't support that14:23
JayFthen I remembered cleaning would change many identifiers there14:23
JayFUUID/PARTUUID/LABEL14:23
dtantsurthe old item: https://storyboard.openstack.org/#!/story/200981414:23
dtantsuryeah, there are caveats14:23
dtantsurhttps://storyboard.openstack.org/#!/story/1718726 needs fixing if not already14:23
JayF2009814 is a good idea, too14:24
JayFI think 2009814  or one of those bifrost enhancements14:24
JayFeither way, thanks for giving me confidence we have enough low-hanging items14:24
dtantsuryeah, I have an enormous TODO list in my personal board under "Ideas and Low Priority" aka stuff I'll never get to :D14:25
dtantsurJayF: a pretty ambitious task would be to move ironicclient to openstacksdk internally14:26
dtantsurI think I should stop at this point :)14:26
JayFyeah I'm looking for not-ambitious14:26
JayFI'm looking for get the dev env to hello world14:26
JayFtake one more step14:26
JayFgreat success14:26
JayFthen we can add ambition14:26
JayFlol14:26
dtantsurOh, last one. My personal pet peeve. Get rid of that stupid API for setting maintenance and make it normally updateable :)14:27
dtantsurit's API work, but it's not super sophisticated14:28
JayFs/make it noramlly updatable/and remove the vague concept which causes more confusion than anything else/14:28
dtantsurheh14:29
JayFif I could go back in time and change anything about original ironic14:29
JayFit'd be maintenance14:29
JayFjust make it online:bool just like the old school printers with an on-line light and button lol14:29
dtantsurI would have changed many things, but yeah. We've overloaded this one too much.14:31
JayFyou only get one, what is it14:31
dtantsurfree-form JSON fields14:32
dtantsur(except for node.extra, of course)14:32
JayFnode.junk_drawer['lol'14:32
JayF:)14:32
dtantsuranother dream of my is to gradually decompose and get rid of them14:32
dtantsuryeah14:32
JayFI think in a weird way, those fields were a strength early on14:33
dtantsurproperties could be a good start14:33
JayFbecause it allowed us to express things in our data model we didn't understand when it was written14:33
dtantsuraka technical debt ;)14:33
JayF#startmeeting ironic15:00
opendevmeetMeeting started Mon Jul 31 15:00:05 2023 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
iurygregoryo/15:00
rpittauo/15:00
kubajjo/15:00
masgharo/15:00
JayFGood morning. Welcome to the Ironic meeting. Reminder this is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct15:00
JayF#topic Announcements/Reminder15:01
JayF#note If you have a patch ready for review; hashtag it with ironic-week-prio; and please use our review dashboard to prioritize patches tagged as such.15:01
JayF#note next PTL is 10/23-2715:01
JayFI'll say generally, Bobcat release timeline is in the agenda. I'll only mention if we have a milestone coming up15:02
JayFnext one is 8/24 for non-client library freeze15:02
JayF#topic Review action items from last meeting15:02
JayFpretty sure there were none; checking15:02
* dtantsur is curious if "next PTL" is a typo or a prediction15:02
JayFconfirmed; no actions last meeting omving on15:02
JayFdtantsur: haha, typo for sure15:03
JayFdtantsur: but I will say I've been nudging rpittau that it's his turn coming up soon ;) 15:03
* rpittau hides15:03
JayF#topic Review Ironic CI Status15:03
dtantsurthe clock is ticking, rpittau!15:03
JayFHow are things? They look OK? At least relative to the horrible place we were a month ago...15:03
rpittauless horrible than usual15:03
iurygregoryneed to agree, I didn't look that much during last week..15:04
JayFKeep an eye on CI. If it gets too outta whack mention in here, and put it in the whiteboard15:04
JayFfor now I assume things are happier15:04
JayF#topic Review ongoing 2023.2 workstreams15:04
JayFAs a reminder if you're newer reading this log or in meeting, we have a doc we publish at the start of the cycle15:05
JayFwhich is entirely hopeful about features we wanna work on15:05
JayFwhen we do work on them, we ask contributors working on this to put a very high levl update here15:05
JayF#link https://etherpad.opendev.org/p/IronicWorkstreams2023.215:05
JayFtbh we've made quite a dent into our priorities list15:06
JayFwe might actually complete most of the things \o/15:06
dtantsur\o/15:07
JayFThere are no RFEs to review, skipping on to...15:07
JayF#topic Open Discussion15:07
JayFFloor is open if anyone has anything15:07
JayFLast call?15:09
rpittauI've struggling a bit understanding why the new tinycore 14.x is failing on standalone jobs on uefi15:09
rpittauit looks like efibootmgr is failing and the issue seems related to efivars moduile not present15:09
rpittauwe're not in a rush with that, but if anyone has ideas on how to fix that or how to troubleshoot further help is appreciated15:10
JayFefivar is a separate binary, not a kernel module though15:10
JayFcan you link the exact err?15:10
JayFWe will troubleshoot this with the logger off :) TY all15:11
JayF]15:11
rpittauefivar is a separate binary, but efivars is indeed a module15:11
JayF#endmeeting15:11
opendevmeetMeeting ended Mon Jul 31 15:11:07 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:11
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-31-15.00.html15:11
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-31-15.00.txt15:11
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-31-15.00.log.html15:11
JayFrpittau: not one I have loaded15:11
JayFrpittau: lemme see if it's built i15:11
JayF*ikn15:11
iurygregorymaybe the module changed the name or we need something extra? <just thinking out loud>15:11
rpittauprobably15:11
JayFCONFIG_EFIVAR_FS=y15:11
JayFlikely all that's needed15:11
JayFor the module that comes out if =m15:11
rpittauit could be a change in the kernel15:12
rpittauI tried with efivars and efivarfs but no luck15:12
rpittauthis is the lat result https://zuul.opendev.org/t/openstack/build/463f1cfac2d0420e9e1a3b7a1c258c7d15:12
rpittauadn I found this https://michael-prokop.at/blog/2021/06/09/efivars-is-gone-with-debian-bullseye-newinbullseye/15:12
rpittauI will give that a try since the kernel seems compatible15:13
iurygregoryinteresting15:13
JayFefivarfs is the module name15:13
JayFyeah that blog looks correct15:13
opendevreviewRiccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build tinyipa with tinycore 14.x  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/88775415:15
opendevreviewMerged openstack/ironic-prometheus-exporter master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/89005415:36
opendevreviewMerged openstack/python-ironicclient master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/python-ironicclient/+/89005015:36
* dtantsur has forgotten that updating his NAS involves bring the IRC bouncer offline15:39
opendevreviewMerged openstack/ironic-python-agent master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-python-agent/+/89005315:39
dtantsuroh, and now I need to update ZNC itself, so offline again :D15:39
dtantsurSo, folks, with virtual media support growing.. WDYT about promoting virtual media attach/detach to the top-level API?15:43
JayFCan you help with a use case?15:44
dtantsurI'm aware of two actually. 1st seems pretty common: custom installers. Ironic is pretty decent in this regard: you can connect a CD, boot from it (aka ramdisk deploy), then set boot device to disk. BUT you end up with the CD still connected.15:45
JayFdtantsur: For that use case; we should genericize the kickstart driver to provide that value to all hardware IMO15:45
dtantsur2nd is something new for me, but people want to be able to just connect/disconnect a thing to/from a machine.15:45
JayF#2 is the case I was thinking of the whole time15:45
dtantsurThink, provide some configuration.15:45
dtantsurI'm not sure how the kickstart drive helps with custom installers..15:46
JayFI'm saying the kickstart driver right now15:46
JayFis boot an install ramdisk + some metadata15:46
JayFcustom installers are: boot an install ramdisk + some metadata15:46
dtantsurhow is it different from the ramdisk deploy?15:47
JayFDoes the kickstart deploy allow us to pass arbitrary stuff to the booted ramdisk?15:47
JayFs/kickstart/ramdisk/15:47
JayFe.g. ks=https://server/my.ks15:47
dtantsurNeither do?15:47
JayFexcept probably more like preseed=https://server/my.preseed15:48
dtantsurit's not arbitrary15:48
JayFI'm saying if we take ramdisk interface + some of the bits added on top for kickstart driver15:48
dtantsurnot everything is done through kernel parameters, but the ramdisk deploy does support kernel parameters15:48
opendevreviewMerged openstack/ironic master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic/+/89005115:48
opendevreviewMerged openstack/ironic stable/victoria: [stable-only] Cap virtualenv/setuptools  https://review.opendev.org/c/openstack/ironic/+/88870115:48
JayFOK; so we're basically there then15:48
opendevreviewMerged openstack/ironic-inspector master: Add python3.10 support in testing runtime  https://review.opendev.org/c/openstack/ironic-inspector/+/89005215:48
JayFyou just have to self-host your preseed file15:48
dtantsurJayF: it does not anyhow remove the need for disconnecting the ISO15:48
JayFthat's... true15:49
JayFsame with kickstart, actually15:49
JayFheh15:49
JayFcustom installer deploy steps can now end with hold15:49
JayFand you can call resume on the node when the install is done 15:49
JayFto get the cleanup actions done, like removing the disk15:49
JayFattach ISO -> change boot device -> boot an installer -> hold -> change boot device to local -> detach iso15:49
JayF(BTW; I'm sold on the API, this is just a nice thought experiment to walk down)15:50
JayFuse cases for julia's new hold step15:50
dtantsurI totally agree, and I see value in both15:50
dtantsurthe trigger for my question were people with "hey, can we use metal3/ironic to connect a CD with configuration"15:51
dtantsurI'm trying to move our people away from the ramdisk deploy :D15:51
JayFI suspected as mnuch, which is why I started asking more use case questions15:51
dtantsur(so, case #2 more than #1 for me personally)15:51
JayFI feel like a "just attach a thing and boot a server" might have people use Ironic more like a mechanical turk than a full orchestrator15:51
dtantsur(but I do know that case #1 is popular, also outside of Red Hat)15:51
dtantsurwell.. remember we discussed the dual nature of our API before my PTO?15:51
JayFlike, we can't both be the primary orchestration AND give all the hooks to the outside world for us to be the target of orchestration15:51
JayF...can we?15:52
dtantsuryou literally argued that we can :D15:52
dtantsurit was about having stuff like set_boot_device in the API15:52
JayFhmm15:52
JayFI just realized you might be the only person who would get onboard with my insaneo idea15:52
JayFI tried to sell Julia on it IRL and got furrowed brows back as a response15:52
JayFIronic could potentially add a lot of value to the bare metal ecosystem15:53
JayFif we split our service from the library pieces in the back15:53
JayFso that other tools could use our *_interfaces directly as a libraryu15:53
dtantsur"Can we have Ironic as a library" was roughly the 1st question we got when Metal3 started15:53
dtantsurThe fatal roadblock was Python15:53
JayFI mean, yeah, it'd be in python15:54
JayFI'm not saying just for metal3 use case, to be clear15:54
JayFcan you imagine the value of 15:54
JayFironit --redfish=http://myserverbmc power on15:54
dtantsurThat being said, yes, I can see value of that, and I've been toying with this idea in my head for at least 2 years15:54
JayF(yes, iron - it, not ironic)15:54
dtantsurThe amount of work is enormous. We now don't distinguish between "hardware interfaces" (power, management) and "workflow interfaces" (deploy, inspect, your future clean).15:56
dtantsurStuff like BIOS/RAID is theoretically "hardware only" but in practice relies on rebooting into IPA15:56
dtantsuretc etc15:56
JayFyeah, factoring agent out is probably a huge % of the difficulty15:56
JayFjust a thought though15:57
dtantsurand TaskManager15:57
JayFI want to make sure that even if ... Ironic's view of the world doesn't win over the long term15:57
JayFthat the work we've done to make these things easier to provision is in a place where others can use it and not have to start over from scratch15:57
JayFthat's the real motivation; for me personally; behind this idea15:57
dtantsurSushy is part of this effort15:57
dtantsurMaybe we should move more fishy logic there15:58
dtantsurMaybe we should finish the late Ilya's sushy-cli15:58
JayFsushy-cli15:58
JayFplus moving some of our old IPMI magic into a library15:58
JayFwould probably be 50% of the value with 10% of the work15:58
dtantsurPossibly, but I'd focus on Redfish since 1) it's the future, 2) it requires much more hacking (who could imagine!)15:59
JayFthe problem I have is that while this is really cool16:00
JayFzero of this use case crosses over with GR 16:00
JayFor is even in the general direction they care about16:00
JayFthat doesn't mean I can't work on it; but it is a canary that maybe this is work that'd be fun for us and not useful to anyone16:00
dtantsurI know janders will work on improving the logic around picking the vmedia device. We cannot move it to sushy right away (backports!), but maybe we should encourage him to follow-up with moving this logic to sushy.16:01
dtantsurOtherwise, I hear ya. But then, I'm stubbornly pushing the Inspector merger for years now, while nobody seems to care much :D16:02
JayFI think two things:16:02
rpittausee ya tomorrow folks! o/16:02
JayF1) I care about Inspector merge; it's good and it makes our model cleaner16:02
JayFBUT that's all mitigated by16:03
JayF2) Deployment automation has improved to where an average operator DGAF it takes an extra service to do a thing16:03
JayFso like, good work, it's needed, but most of the pain it resolves is hidden from opers16:03
dtantsurIt's not just installing a service. It's performance overhead on the ops side and a lot of maintenance overhead on our side.16:06
JayFI agree with that from our, deeper perspective16:07
JayFbut it's hidden from operators just looking at it from a surface level16:08
opendevreviewMerged openstack/ironic-python-agent-builder stable/2023.1: Add libaio engine for fio on Centos/Rhel9  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/88648816:29
JayFdtantsur: iurygregory: Isn't one of you on the RH Openshift team?16:47
dtantsurJayF: both (and Riccardo too)16:47
JayFDM16:47
dtantsurand Mahnoor, and Derek, and Jacob :)16:47
JayFheh, was telling dtantsur to be nice to my friend starting in openshift16:51
JayFbut apparently you all are a small army16:51
dtantsur:D16:51
dtantsurspeaking of inspector, could I get a 2nd +2 on https://review.opendev.org/c/openstack/ironic/+/887202 and https://review.opendev.org/c/openstack/ironic/+/885450 please?17:13
opendevreviewVerification of a change to openstack/bifrost master failed: Fix ipaddr deprecation warning  https://review.opendev.org/c/openstack/bifrost/+/88919518:08
JayFlots and lots of TIMED_OUT on that one18:09
opendevreviewMerged openstack/ironic master: Correct two mistakes in the /continue_inspection API  https://review.opendev.org/c/openstack/ironic/+/88720218:43
opendevreviewMerged openstack/ironic master: Very basic in-band inspection with the "agent" interface  https://review.opendev.org/c/openstack/ironic/+/88545018:51
opendevreviewJay Faulkner proposed openstack/ironic master: Fix typo in deploy_templates docs  https://review.opendev.org/c/openstack/ironic/+/89016321:35
JayFExtremely trivial; spotted it while drafting a spec21:36
opendevreviewJay Faulkner proposed openstack/ironic-specs master: Self-Service via Templates  https://review.opendev.org/c/openstack/ironic-specs/+/89016421:56
* JayF notes that 890164 was partially drafted by ChatGPT 4.0 + link reader plugin21:56
JayFIt did ... okay. Not great, but OK. Gave me a starting point for almost everywhere when fed our spec template, code repo, and the bug with the feature summary I already wrote21:57
JayF(The posted copy is heavily edited by my own hand; that was just a starting point)21:57
opendevreviewJay Faulkner proposed openstack/ironic-specs master: Self-Service via Templates  https://review.opendev.org/c/openstack/ironic-specs/+/89016422:14
JayFI have a friend from another IRC channel I was pointing at bifrost; I was telling them about discovery but aside from the release note ( https://github.com/openstack/bifrost/blob/de39802cdb75e5eeba0895824f83e6aba7420ffd/releasenotes/notes/enable-inspector-discovery-77aa6f5c1f1e42c6.yaml ) I don't see any docs on it in bifrost23:04
JayFis there any magic to it?23:04
JayFIf awb joins, that's the friend, help 'em out :D 23:04

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!