Monday, 2022-08-22

kubajjgood morning ironic!06:49
opendevreviewAija Jauntēva proposed openstack/sushy master: Capture requests errors  https://review.opendev.org/c/openstack/sushy/+/85320907:04
*** RamonaRautenberg[m] is now known as RamonaBeermann[m]07:55
dtantsurhappy monday folks09:14
kubajjhey dtantsur09:19
ftarasenkogm ironic! have question for you about mdraid that is created on server by client (not by IPA). When client decommission this server, it goes to cleaning, but IPA sees mdraid and tries to clean it instead of physical disks. Can delete_configuration for mdraid be explicitly enabled while automated cleaning? arne_wiebalck: do you clean md devices with automated clean or you have different workflow?09:29
iurygregorygood morning Ironic10:43
dtantsurmorning iurygregory 10:58
TheJuliagood morning13:10
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Changes to support Anaconda CI jobs  https://review.opendev.org/c/openstack/ironic/+/84958713:10
iurygregorygood morning TheJulia 13:14
TheJuliaDid everyone have a good weekend?13:15
TheJuliaftarasenko: can it today... no. With the model of raid configuration, I'm not sure we can run explicit delete_configuration, but maybe it would be okay if the raidset is torn down if the agent knows nothing of it based upon the configuration from ironic?!?13:19
iurygregoryI did13:27
iurygregoryre - good weekend13:27
arne_wiebalckftarasenko: we do it during auto-cleaning13:28
arne_wiebalckftarasenko: we have our own h/w manager to do this13:28
arne_wiebalckftarasenko: but I think you could crank up the prio of this step to be included as well13:28
* arne_wiebalck *thought* you could do this, but has now read TheJulia's comment, so is maybe wrong13:30
TheJuliaarne_wiebalck: I didn't think about resetting/overriding the priorities13:30
TheJuliaI mean... it is not something any of us have tested...13:30
arne_wiebalckTheJulia: I never tried either, but this was my understanding of what you could do with clean step prios.13:31
opendevreviewMerged openstack/sushy master: Capture requests errors  https://review.opendev.org/c/openstack/sushy/+/85320913:32
TheJulia\o/13:32
arne_wiebalckanother topic: 'node history' ... this is unfinished work from xena ... I guess we could pick this up w/o interfering with anyone working on this, right? kaifeng was working on this at the time, but I don't think he got round to do much about it lately ... does anyone know what the status is?13:33
TheJuliaI *think* it works13:34
TheJuliaI've seen the tests pass13:35
arne_wiebalckoh, yeah?13:39
arne_wiebalckseems like xena added sth on the API yes13:41
TheJuliaugh13:42
TheJuliawell, this is fun. Google now prefers mitaka and newton docs13:43
TheJuliahttps://docs.openstack.org/python-ironicclient/latest/cli/osc/v1/index.html#baremetal-node-history-list13:44
TheJuliathe db side code journals, you cannot delete entries for obvious reasons... the conductor will purge them. I don't remember what the defaults were13:47
TheJuliaspeaking of kaifeng, has anyone heard from him in say the last year?13:57
dtantsurnot me14:04
dtantsurgood morning TheJulia 14:04
iurygregorynope =(14:04
iurygregoryI think he only joined one time after we moved to OFTC...14:04
TheJuliahttps://www.stackalytics.io/?release=all&metric=commits&user_id=kaifeng14:09
TheJuliaI suspect they have moved on, they haven't responded to any emails for quite some time if I'm remembering correctly14:10
arne_wiebalckis the node history fully done then?14:13
arne_wiebalckor is there sth left to do?14:13
iurygregoryTheJulia, yeah14:14
TheJuliaarne_wiebalck: I think the work is done, although to be totally honest, I've not really "kicked the tires"14:19
arne_wiebalckTheJulia: ok, we will try to check ...14:20
arne_wiebalckTheJulia: thanks!14:21
TheJuliaiurygregory: that is a great email, thanks!14:53
iurygregoryTheJulia, dtantsur also helped a lot with the email14:54
TheJuliagreat job both of you!14:54
JayF:( I'm sorry to hear about etingof. Condolances to those of you who got to know them better than I did.14:56
* TheJulia sighs14:57
iurygregory#startmeeting ironic15:00
opendevmeetMeeting started Mon Aug 22 15:00:16 2022 UTC and is due to finish in 60 minutes.  The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
iurygregoryHello everyone, welcome to our weekly meeting15:00
JayFo/15:00
erbarro/15:00
matfechnero/15:00
kubajjo/15:00
TheJuliao/15:00
rlooo/15:00
dtantsuro/15:01
iurygregoryyou can find the agenda for the meeting in the wiki15:01
iurygregory#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting15:01
ajyao/15:01
iurygregory#topic Announcements / Reminder 15:01
kamlesh6808co/15:01
rpiosoo/15:02
iurygregorythe first announcement is a sad one15:02
iurygregory#info Ilya Etingof Passed Away. Goodbye, etingof!15:03
iurygregory#link https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030062.html15:03
ajyaSorry to hear that, my condolences.15:03
iurygregoryI would like to share with our community, some of us knew him a lot15:04
arne_wiebalcko/15:04
rlooOH no, I'm so sorry.15:05
rpiosoI am very saddened to learn of etingof's passing. My condolences to the ironic community, Red Hat, and Ilya's family and friends.15:07
iurygregory#info This week we will release our non-client libraries15:07
ajyathat's sushy also?15:07
iurygregoryajya, correct15:08
dtantsursushy, ironic-lib, metalsmith15:08
iurygregoryyeah15:08
dtantsurwe need to check for outstanding patches (I have one, has some comments)15:08
iurygregoryso we will focus on reviewing this 3 to make sure we have included what we want in Zed =)15:08
iurygregorydtantsur, yeah15:08
ajyacan I get 2nd reviewer for this https://review.opendev.org/c/openstack/sushy/+/850899 and include that in release?15:09
iurygregoryajya, sure we will try to include the open patches =)15:09
JayFI'll look; I haven't traditionally worked much on sushy but should proabbly ramp it up.15:09
ajyathanks15:09
iurygregory#info Antelope PTG etherpad15:09
iurygregory#link https://etherpad.opendev.org/p/ironic-antelope-ptg15:10
iurygregoryjust a reminder that our etherpad for the PTG is this one =)15:10
iurygregory#info PTG registration 15:11
iurygregory#link https://openinfra-ptg.eventbrite.com/15:11
iurygregorydon't forget to register for the PTG15:11
TheJuliaPlease register for the PTG so the foundation knows how many attendees plan to actively engage. This allows them to have information for future planning as well, so everyone attending registering would help them a lot.15:12
iurygregory#info ironic-ui is fixed =)15:13
* TheJulia suspects we all need to dance now15:13
dtantsuryay!15:14
iurygregoryI don't know the irc handle of Vishal, tks for the help!15:14
iurygregory#link https://review.opendev.org/c/openstack/ironic-ui/+/85270215:14
iurygregoryno action items from previous meeting, skipping15:15
iurygregory#topic Review subteam status reports15:15
iurygregory#link https://etherpad.openstack.org/p/IronicWhiteBoard15:15
iurygregorystarting around L9015:15
JayFAre there even meaningful updates there to review?15:18
TheJuliaI guess the one w/r/t anaconda15:18
iurygregoryAnaconda CI 15:18
iurygregoryyup =)15:18
TheJuliaUhh... so... tl;dr is I cannot use opendev's mirror system without hacking in another feature (maybe) into the interface to explicitly delineate package repositories versus all the other artifacts15:19
iurygregory=(15:19
TheJuliain essence, the mirror can't take on a more stuff without there being an increasingly negative impact, and the guidance is to just use public mirrors for folks doing Rocky linux15:20
JayFHave we looked at if we can make the install lighter/faster in any way to help get past timeouts? 15:20
TheJuliaso.. I *think* the net effect is I just need to get the timing right15:20
JayFI'm mainly curious if we can pass flags to anaconda, disable some of the setup steps to get a thinner test15:20
TheJuliait is fairly minimal, downloads can just take a ton of time15:20
JayFbut I haven't looked at a breakdown of what it's spending most of the time in15:20
JayFexcept that one were we saw it was taking ~5 minutes for all the packages15:20
TheJuliapossibly, although again, I'm thinking we're talking borderline featurey things15:21
TheJuliamaybe those are okay since they would just be jinja2 parsing15:21
JayFbut really ... 5 minutes is small in context of an hour+ job15:21
TheJuliayeah, latest run got to configuring the kernel post-install at 1hr. I think we're also having the lack of paravirt kill us too15:21
TheJuliasince it is a lot of CPU overhead on uncompressing15:21
TheJuliaI'm going to push forward, already working to make it it's own single job15:22
TheJuliajust wanted folks to generally be aware15:22
JayFif you want to pair on this at any point, or just get a second set of eyes, feel free to ping me and/or we can even set aside some time15:22
TheJuliaThe "evil" option in my brain is just look to see if anaconda checked in with ironic, and then abort the deployu15:22
TheJuliawhich could be valid too...15:22
TheJuliaMaybe we should discuss that instead15:23
JayFthat's exactly the kind of short circuiting I'm looking for15:23
TheJuliaUltimately, it all comes down to the template and it's contents, it is hyper customizable15:23
JayFironic does all the orchestration up front; do we want the CI to "catch" anaconda breakages, or just Ironic's ability to set the table for anaconda?15:23
TheJuliaIt wouldn't catch the close out of the deploy... but that should be fairly clear if it breaks based upon reports/issues15:24
TheJuliabut starting/template processing/data handling, it would catch any issues there15:24
JayFwe could even intentionally have a deploy break and return an error to ironic15:24
JayFtesting the unhappy path is arguably more important than the happy one15:24
TheJuliadepending on what, yeah15:24
JayFget anaconda going far enough for it to be able to do the err callback to ironic; then we know at least we setup anaconda as expected15:25
TheJuliaAnyway, I could use opinions here, I'm a bit tired of fighting this :)15:25
JayFyeah, use me as the help for that, I'm somewhat guilty for helping upstream that sans-CI 15:25
TheJuliaThat might be a really good separate test, fwiw15:25
JayFI am proposing it potentially as the only test, as we could probably skip all the package installs15:25
TheJuliado one that aborts, do one that errors, call it a day15:25
JayFthen you get something working sooner, then worry about the happy/aborted path15:25
TheJuliaoh, error would get called before package installs I believe15:25
JayFthat's what I'm saying15:26
TheJuliaI think you just need to feed it invalid stuffs and it calls %onerror15:26
rloowhat exactly are we trying to test wrt anaconda interface?15:26
rloominimum test I guess...15:26
JayFHeh. You could go an even step further. Write test code in anaconda template, run it, use the returned err as an indication that things were setup as expected (or not)15:26
TheJuliaI'd like to know end to end it works15:26
TheJulia*but* there is a lot of overhead to make it happy in our CI15:27
rlooend-to-end == actually install an OS image?15:27
TheJuliawell, in this case, install from a repository15:27
JayFrloo: yeah TheJulia has it working with an actual install, it just times out in the last few %15:27
TheJulialike last 10% it looks like15:27
JayFusing the support added for repo-based deploys instead of liveimg based deploys15:27
TheJuliabut it is quite variable based upon the mirrors15:27
rloogad. hmm... and it needs everything from the mirrors?15:28
rlooand where does the time out come from? can we increase it?15:29
TheJulianot *everything* only like 320 rpms15:29
TheJuliainfra cannot carry the stage2/install image15:29
TheJuliaso... no local mirrors15:29
iurygregoryonly...15:29
TheJuliawe should move on15:30
TheJuliaand continue in open discussion15:30
iurygregoryI was about to say this =)15:30
iurygregory#topic Deciding on priorities for the coming week15:30
iurygregory#link https://review.opendev.org/q/status:open+hashtag:ironic-week-prio15:30
JayFI'm working on trying to get several Nova-Ironic driver patches backported, probably wouldn't hurt to get more Ironic +1s on them if folks want to add them to their review list as well. I'm tracking it here: https://etherpad.opendev.org/p/NovaPatchesFromJay15:31
JayF(I don't think we can put weekly-prio tag on nova patches)15:31
kamlesh6808cCan you please help to add this patch to week priority list : https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/85362115:32
TheJuliaI think we can hashtag it...15:32
iurygregoryTheJulia, normally only the owner of the patch can do that if I recall15:32
TheJuliaahh, yeah!15:32
iurygregoryalso depends on the config for the project15:32
iurygregoryfor example https://review.opendev.org/c/openstack/nova/+/813897 has the hashtag15:33
iurygregoryso JayF you can probably try to add the hashtag (I think it should work...)15:33
iurygregorykamlesh6808c, added15:34
kamlesh6808cthanks !15:34
iurygregoryI'm adding dtantsur's patch https://review.opendev.org/c/openstack/sushy/+/851023 also15:34
JayFiurygregory, all: I updated the hashtag on those nova stable patches owned by me (many are owned by others and I'm just playing frontman to get them merged lol)15:35
iurygregoryJayF, no worries!15:35
iurygregorytks!15:35
iurygregorynot sure if Eric Lei is around to update https://review.opendev.org/c/openstack/ironic-lib/+/84466615:36
iurygregoryI'll push an edit later today so we can merge =)15:36
iurygregorymetalsmith doesn't seem to have patches we would need to review 15:38
iurygregorymoving on o/15:38
iurygregory#topic Baremetal SIG 15:38
TheJuliagiven the time, I think one of us should jsut make the change15:38
arne_wiebalckNTR for the SIG15:39
iurygregorytks arne_wiebalck =)15:39
iurygregory#topic RFE review15:39
iurygregoryI'm a bit puzzled if the topic from open discussion would be rfe review... =)15:40
arne_wiebalckyep, could be15:40
iurygregory#info Discussion of the software RAID story15:40
iurygregory#link https://storyboard.openstack.org/#!/story/201023315:40
iurygregorykubajj, o/15:41
arne_wiebalckkubajj has been working on extending the disk protection to s/w RAID devices15:41
arne_wiebalckone question we ran into is what to do with create_configuration15:41
arne_wiebalcki.e. when the devices are re-created15:42
JayFCan I ask a question a step behind that?15:42
arne_wiebalcksure15:42
kubajjSure15:42
JayFWhy do we need the ability to explicitly skip disks that hold RAID partitions15:42
JayFif the operator already has the (thanks to kubajj) ability to skip disks based on device hints?15:42
kubajjBecause RAIDs are skipped by default anyway. They are handled in a different function15:43
JayFAh, and you just want to add software raids to those that are skipped.15:43
JayFWe have to be careful how we implement this to prevent a malicious actor from putting something that looks like a raid superblock on a disk to prevent being cleaning15:43
kubajjYeah, the goal is just to extend the functionality.15:43
JayF**cleaned15:43
arne_wiebalckJayF: everything but the partitions which form the RAID are cleaned15:44
arne_wiebalckwell, almost everything :-D15:45
JayFYeah, I'm not saying we shouldn't do it, I'm saying we should be careful and make sure there's an opt-out for anyone with a higher security bar15:45
arne_wiebalckJayF: sure, unless you explicit say on the node that you would like to skip sth, all will be cleaned15:46
arne_wiebalcklike before15:46
JayFawesome15:46
arne_wiebalckthis is about the special case where you have multiple s/w RAID devices 15:46
JayFI apologize, some of this stuff, I don't know what happened when I wasn't looking so I appreciate you filling in the context15:46
arne_wiebalckand you would like to skip cleaning *some*15:46
kubajjexactly, the plan is to use the volume_name as mentioned in the story and include it in the skip_block_devices list in the properties section like with normal disks15:47
arne_wiebalckthe volume name is the name you give the device itself (not the block device file), it is md device metadata15:48
kubajjhttps://review.opendev.org/c/openstack/ironic-python-agent/+/853182 enables actually creating logical disks with volume name enabled15:49
kubajjI have tested it out on our testing node and it works15:49
arne_wiebalckand the question was if there is an obvious problem with this ... I think dtantsur mentioned the inspector as one potential source of problems15:49
arne_wiebalckotherwise we go ahead and see where it gets us :)15:50
iurygregoryI liked the idea, just trying to understand the inspector problem ...15:52
arne_wiebalckthe inspector adds the root device to the inspector data (I think)15:52
JayFAre we not worried at all about the ability for whoever got that device provisioned to them being able to change that volume name?15:53
JayFLike, if that's not a case we're worried about; awesome... but it's trivial for that volume name to change15:53
JayFI was going to suggest PARTUUID but pretty much any unique identifier is changable from the system :( 15:54
* TheJulia wonders if we're scope creeping to cover all possibilities as opposed to trying to cover 90%15:54
arne_wiebalckif you change the volume name, the next cleaning would erase your data15:54
JayFTheJulia: that's why I asked if we were worried about it :D 15:54
TheJulia(not to say everything is good!, but obviously we need to start somewhere)15:54
JayFarne_wiebalck: of course, so it fails safe15:54
JayFaight, sounds like fun :) I look forward to reviewing it15:54
arne_wiebalckheh15:54
arne_wiebalckok, we can check with dtantsur directly once more, seems he is not here atm15:55
dtantsurI'm kinda here, not following the discussion tho15:55
JayFI have a small item for open discussion, if we've talked this one through15:55
iurygregoryJayF, go ahead15:56
JayFSo, most of you know I've been working a new job, with 20% time dedicated generally to openstack and 80% to the other project I work on, Armada15:56
dtantsuriurygregory: the problem with inspector is that it does not have access to any symlinks on the node15:56
JayFover the next few weeks, those percentages will be swapping and you'll be seeing more of me around15:56
JayFI'm going to focus, until I get into a good cycle of my own work, on upstreaming some stuff from downstream here, stable maintenance, and reviews15:57
JayFbut feel free to nerd snipe me in for bigger stuff as I will have the time to give15:57
iurygregoryJayF, nice15:57
arne_wiebalckdtantsur: I fail to see how that is a problem15:57
dtantsurthe discussion was around resolving /dev/md/<something>15:58
JayFI know we don't have a formal role for it anymore, but generally I am going to try to be the grand-poo-bah of stable branches, getting stuff backported (and starting with heralding our long-outstanding patches to our nova collegues)15:58
dtantsurI still haven't read the full scrollback15:58
arne_wiebalckdtantsur: we can do this tomorrow or so15:58
dtantsurJayF: I feel bad asking someone to look at our bug list.. but someone needs to look at our bug list15:58
JayFThat /dev/md/<something> is going away in newer mdadm aiui15:58
JayFwhen you said volume name, I assumed that was shorthand for the partition label15:59
TheJuliacool!15:59
JayFdtantsur: when I get ramped up, i'll specifically cut aside some time to work on bugs15:59
TheJuliaJayF: is it hiding in under the lvm interface?15:59
arne_wiebalckJayF:  no, what we mean is the name of the md device (which is sth you can set on an md device)15:59
JayFarne_wiebalck: that is going away in newer mdadm/kernel combinations15:59
JayFarne_wiebalck: people were complaining about it being gone in gentoo channels the other day15:59
dtantsurarne_wiebalck: without /dev/md/<something>, volume name seems useless16:00
TheJuliaoh noes16:00
kubajjJayF: you mean the --name option?16:00
arne_wiebalckdtantsur: why?16:00
dtantsurarne_wiebalck: how else do you access it?16:00
JayFkubajj:  I *think* so? I'd have to look in depth to remember exactly16:00
arne_wiebalckmdadm --detail16:00
dtantsurarne_wiebalck: okay, here is the trick: how are you going to do it on the inspector side? :)16:00
dtantsurassuming you want it as part of the root device hints16:00
arne_wiebalckdtantsur: why do I need to do it on the inspector side?16:01
dtantsurarne_wiebalck: because we already do16:01
iurygregorywe are at the top of the hour for the meeting , going to close and we can continue talking :D16:01
dtantsurwe parse and process root device hints in inspector16:01
iurygregory#endmeeting16:01
opendevmeetMeeting ended Mon Aug 22 16:01:24 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-08-22-15.00.html16:01
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-08-22-15.00.txt16:01
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-08-22-15.00.log.html16:01
dtantsurnow, you could argue we should just deprecate and throw away this from inspector, and I'll likely agree :)16:01
arne_wiebalckthanks iurygregory 16:01
dtantsurbut it's there, and we cannot just ignore it16:01
arne_wiebalckI most likely do not understand how that is used16:02
arne_wiebalckthe inspector gets the hints, and populates the inspection data with root_device ... and then?16:02
arne_wiebalckwhat is that used for ?16:02
TheJuliatarget of where to write the image16:02
arne_wiebalckroot_device in the inspection data I mean16:02
dtantsurarne_wiebalck: largely, local_gb detection16:02
arne_wiebalckthis read from the node and not redone upon deployment?16:03
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Changes to support Anaconda CI jobs  https://review.opendev.org/c/openstack/ironic/+/84958716:04
kubajjdtantsur: but could we pass the volume names to inspector somehow?16:04
kubajjI have to admit that I am not really sure when the inspector gets called though16:05
arne_wiebalckkubajj: we could add them to the inspection data, but not if there is no reason to do it16:05
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Initial tempest test idea anaconda deploy  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/85403116:06
* TheJulia wonders if we should instead just be working to peel back some of local_gb16:07
* TheJulia is just thinking outloud16:07
dtantsuryeah, that's what I mean: this code can probably be killed. but with deprecation sooo...16:07
TheJuliayeah16:07
arne_wiebalckok ... we will let code speak :)16:14
TheJuliaJayF: I think if my latest changes don't result in a passing job, I'll just head on the "oh, we got to deploy wait, abort!"16:14
TheJuliaBut if it gains conciousness....16:15
arne_wiebalckTheJulia: the code? hmm ... 16:15
* arne_wiebalck goes home now to think about that ...16:16
TheJulialol16:16
arne_wiebalckbye everyone o/16:16
TheJuliagoodnight16:20
*** tkajinam is now known as tkajinam|off16:35
*** rcastillo|rover is now known as rcastillo18:22
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Initial tempest test idea anaconda deploy  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/85403118:38
TheJuliadtantsur: so I read in the mboot source after I remembered iso booting drops the rest of the iso contents from memory... and cried even more because they do things....19:49
* TheJulia largely eradicates the doc items he wrote up19:54
TheJuliashe19:54
opendevreviewMerged openstack/sushy master: Add new Storage controllers  https://review.opendev.org/c/openstack/sushy/+/85089920:33
opendevreviewJulia Kreger proposed openstack/ironic master: Add docs for VMware deployment  https://review.opendev.org/c/openstack/ironic/+/85354921:27
TheJuliaJayF: well... https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/854031/ (i have no idea how it is now a new change set....) but it passed it seems21:31
TheJuliaJayF: I don't feel too horrible about a selector in the config file...21:36
JayFaight, looking21:53
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Changes to support Anaconda CI jobs  https://review.opendev.org/c/openstack/ironic/+/84958723:21
opendevreviewJulia Kreger proposed openstack/ironic master: Docs: Add considerations to anaconda docs  https://review.opendev.org/c/openstack/ironic/+/85404523:21
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: Tempest test for anaconda deploy  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/85403123:21

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!