Monday, 2022-04-25

rpittaugood morning ironic! o/07:28
*** tkajinam is now known as tkajinam|away08:33
dtantsurgood morning folks09:03
dtantsurlost scrollback over the weekend, please repeat any pings09:03
* dtantsur is trying Stream 9 for his bifrost VMs now that his lab machine got restarted anyway09:18
rpittauI'm not alone in the lab restarting madness \o/10:07
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532410:58
iurygregorygood morning Ironic 11:12
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532412:28
TheJuliagood morning12:58
TheJuliaiurygregory: any word on multipath?13:05
iurygregoryTheJulia, a lot ....13:05
iurygregorybut they haven't tested yet..13:05
iurygregorybecause of many different problems13:05
TheJuliaoh joy13:06
TheJuliaI guess my only concern is it is CR-2 at present13:06
iurygregoryI can give some details in the downstream channel13:06
TheJuliaAnd.. I'm not sure it is not a generally good chagne, but agree we need to be sure about it13:06
rpiosoGood morning, ironic :)13:07
TheJuliaThe only major difference though is we use device mapper names when detected, I think.13:08
TheJuliaBut I guess to "truly" test it in CI, we need multipath-tools/device-mapper-multipath installed13:08
* TheJulia wonders if there *is* any way to do multipath simulation in CI13:08
hjensasTheJulia: https://sharkcz.livejournal.com/12846.html13:11
TheJuliamuahahahahaha13:12
TheJuliaI was thinking of trying exactly that13:12
TheJuliaMaybe something to do once I try to get my new devstack VM working13:12
TheJuliaSilly... new ubuntu version13:12
TheJuliagood morning tzumainn 13:14
tzumainnTheJulia, hi!13:14
TheJuliadevstack's random IO this morning is thrashing one of my disks :(13:44
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532413:48
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532414:17
*** tkajinam|away is now known as tkajinam14:42
iurygregoryTheJulia, summary from the PTG on the ML14:43
iurygregory\o/14:43
iurygregoryreally sorry for the delay (the downstream case was driving all my energy in the last two weeks...)14:43
TheJuliano worries14:48
opendevreviewMerged openstack/ironic master: Fixes log formatiing string.  https://review.opendev.org/c/openstack/ironic/+/83913214:50
iurygregory#startmeeting ironic15:00
opendevmeetMeeting started Mon Apr 25 15:00:00 2022 UTC and is due to finish in 60 minutes.  The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
iurygregoryHello ironicers!15:00
erbarro/15:00
rpiosoo/15:00
iurygregoryWelcome to our weekly meeting!15:00
dtantsuro/15:00
hjensaso/15:00
TheJuliao/15:00
ajyao/15:00
rpittauo/15:00
iurygregorythe agenda for our meeting can be found in the wiki15:00
iurygregory#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meetin15:00
stendulkero/15:01
iurygregoryops wrong link 15:01
iurygregory#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting15:01
iurygregory#topic Announcements / Reminder15:01
iurygregory#info Zed PTG Summary is the ML15:02
iurygregory#link http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028293.html15:02
rlooo/15:02
kamlesh6808co/15:02
iurygregorysorry about the delay on this and also on the patch with the priorities for the cycle, last two weeks were a bit complicated downstream15:03
iurygregory#info First bugfix branch to be created next week15:04
TheJuliaI guess I'm curious, first bugfix like 4 weeks into the new cycle?15:04
rloothx for the PTG summary iurygregory!15:04
iurygregoryTheJulia, yeah...15:04
rloowhat, there's a bug? :D15:04
TheJuliaThere are always bugs15:04
TheJuliaThey shall rule the world one day!15:04
rpiosoMore, now that it's spring :)15:04
TheJuliarpioso: exactly!15:04
TheJuliaiurygregory: just every six week timing from the last?15:05
iurygregorysome information to help on that front.. downstream we consume bugfix and stable branches, and we had some changes in the calendar15:06
rpittauI think we're in between 5 and 7 weeks, so we should be good? Next week is 5 weeks from the branch cut15:06
iurygregoryyeah we will be 5 weeks I think15:06
TheJuliaOkay15:06
iurygregoryit would help so we don't have to do downstream backports from a feature we had included in *zed* to *yoga* downstream15:07
TheJuliaugh15:07
TheJuliafun!15:07
TheJuliaAnyway, onward15:08
iurygregoryok o/15:08
iurygregory#topic Review action items from previous meeting15:08
iurygregorywell I don't think I added as action item, but it's one.. I had to push the summary + the priorities patch so people cold review15:09
iurygregoryI've done the summary with some delay, the patch with priorities will be up after my lunch today15:09
iurygregory#Review subteam status reports15:10
iurygregorywe will likely skip this week and get back to it next week, thoughts?15:10
iurygregoryok, moving on =)15:12
TheJuliayeah, move on15:12
iurygregory#topic Deciding on priorities for the coming week15:12
iurygregory#link https://review.opendev.org/q/status:open+hashtag:ironic-week-prio15:12
TheJuliaI'm thinking of putting multipath patches up... if I do so, any objection if I add them to the prio review list?15:13
iurygregoryDoes anyone have topics that we should review? we only have 4, I know the tempest-plugin one have been open for a while (I will take a look, I was quite busy past 3 weeks)15:13
iurygregoryTheJulia, ++ to adding15:13
iurygregoryrpittau, since you have a -2 on it, what are your thoughts?15:14
TheJuliaSo looks like I could add the auto-add lessee id field feature. two minor things to fix and it shoudl be good to go15:14
TheJuliaI'm thinking so we actually have a disk representing it, although it might break our RAID testing. Would be interesting to see!15:15
rpittauiurygregory: let's add that to priority, my -2 was related to the testing part15:15
TheJuliaI'm also focused on trying to get our v6 job sorted15:15
iurygregoryi would just try to keep as simple as possible so we can backport without many problems TheJulia =D15:15
TheJuliabut I think the issue is in devstack at the moment15:15
iurygregoryrpittau, ack =)15:16
TheJuliaiurygregory: well, at least on master we can likely start testing it fairly easily 15:16
TheJuliaI'll add related patches once they are up15:16
iurygregorymakes sense to me15:16
iurygregoryI will also add the grenade skip one after looking at the feedback (tks TheJulia and dtantsur )15:18
iurygregoryand ofc the one with the priorities I will push =D15:18
iurygregoryok, moving on15:19
iurygregory#topic Open discussion 15:19
iurygregorywe have one topic today \o/15:19
iurygregory#info Transition Victoria to EM15:20
TheJuliaskipping rfe review?15:20
iurygregoryrfe is after sig 15:20
TheJuliaahh, open discussion should be at the end15:20
iurygregoryat least in the order I see in the agenda...15:20
TheJuliaSorry15:20
TheJuliasince open ended discussions can end the meeting15:20
TheJuliaAnyway, ignore me15:20
iurygregoryno worries!15:20
TheJuliaI think i figured out why our v6 job is broken15:20
iurygregory#link https://review.opendev.org/c/openstack/releases/+/83793715:21
iurygregoryI'm wondering if we want to push some releases before tagging victoria-em15:22
TheJuliaseems reasonable if they are out of date15:22
iurygregory#action iury to check if we have releases in victoria before moving to em15:22
iurygregory#topic Baremetal SIG15:23
iurygregory#link https://etherpad.opendev.org/p/bare-metal-sig15:23
iurygregory#info Recording of the last SIG meeting - Manuel Holtgrewe on "Bare Metal for Health - Using OpenStack Ironic for HPC at Berlin Institute of Health" 15:24
iurygregory#link https://youtu.be/5yJvXFOqzSI15:24
iurygregoryarne_wiebalck, anything to add for the SIG?15:24
iurygregoryok, I think Arne is not around today =)15:26
iurygregorymoving on15:26
iurygregory#topic RFE review 15:26
iurygregory#info Prevent mass instance deletions/rebuilds15:27
iurygregory#link https://storyboard.openstack.org/#!/story/201000715:27
iurygregoryTheJulia, o/15:27
TheJuliaSo!15:27
TheJuliaI proposed two RFE's based upon discussions during the PTG15:27
TheJuliaThe first is a basic mechanism to allow preventing mass deletions of nodes15:28
TheJuliaThe idea being lightweight, and simple to implement, and I believe fairly easy for us to wire in. I'd appreciate any feedback.15:28
TheJuliaThe latter is regarding Agent power savings15:29
TheJulia#link  https://storyboard.openstack.org/#!/story/201000815:29
TheJuliaI've posted a WIP of this and did some local experimentation, just to keep it simple15:29
TheJulia#link https://review.opendev.org/c/openstack/ironic-python-agent/+/83909515:29
TheJuliabasically, just changes the governor and tries to invoke intel's internal pstate selector if present15:30
TheJuliaPlease let me know if there are any thoughts or concerns, otherwise I'll proceed with as I've started15:30
iurygregorythis is related to the safeguards we talked at the PTG right?15:31
TheJuliayes15:31
iurygregoryok, in my mind it was only for cleaning, but the idea does make a lot of sense after reading what you wrote in the RFE15:32
rpittausorry, I need to drop, I'll check the backlog later o/15:32
iurygregorybye rpittau =)15:32
TheJuliaI'm thinking upstream can carry a reasonable default, and folks like Arne can tune the setting down to match their environment's usage15:33
iurygregory++ yeah15:33
TheJuliaand actually, that reasonable default *could* just default on startup to number of conductors * threads too15:34
TheJuliamaybe that is not reasonable15:34
TheJuliaAnyway, just thoughts15:34
TheJuliaoh, we should add https://review.opendev.org/c/openstack/ironic/+/818299 to the list for reviews15:34
iurygregoryinteresting idea (# conductors * threads)15:35
TheJuliaIn similar vain of protections, I posted another WIP to get an idea out of my head15:35
TheJuliahttps://review.opendev.org/c/openstack/ironic-python-agent/+/83908415:35
TheJuliaWhich also kind of starts building a groundwork an intern can carry forward15:35
iurygregoryTheJulia, I just noticed you mention in the RFE *This spec proposes*15:36
iurygregoryin the end you think it requires a spec?15:36
TheJuliaoh, old habits maybe?15:36
iurygregorymaybe =)15:36
iurygregoryI just wanted to double check, because after reading I don't think it would require one...15:37
iurygregorybut maybe is just me :D we need more feedback to make a final decision 15:37
iurygregoryto me it does make sense, no objections15:38
TheJuliaI did go through and look for common shared disk/lun block filesystems15:38
TheJuliaand added the 3 to that patch that made sense15:38
dtantsurI don't think we should add all filesystems15:38
TheJuliamaybe not for GPFS, but I found a shared block device example in IBM's docs15:38
dtantsuronly that magical ones that can wipe the cluster via iBFT or whatever15:38
TheJuliayeah, and those are these filesystems15:38
dtantsurI highly doubt GFS2 is one15:38
TheJuliaAccording to docs, it is, but I'm happy to remove it. It also doesn't use partitions or a table, which caused me to raise an eyebrow some15:39
TheJulias/table/mbr or gpt table/15:39
dtantsurI mean... there is a legitimate case of wiping members of distributed file systems15:39
dtantsurthink, decomissioning of a storage node15:39
TheJuliaabsolutely!15:39
dtantsurI don't disagree that people should do a clean-up first, but it may be impossible e.g. if the OS has died15:39
TheJuliaand distributed filesystems in network distributed mode, absolutely15:39
TheJuliaagreed, which is why I've also though to fan easy node and global level knob15:40
TheJulia"smash it all, I don't care" sort of button15:40
TheJuliabut maybe not a button15:40
TheJuliaTBD15:40
hjensasGFS2 is for shared access to the same device. So I think it is something we want to protect?15:40
dtantsurbeing a distributed fs is not enough15:40
TheJuliabut shared block to same device is kind of it, Distributed as in Ceph doesn't use shared block devices15:41
dtantsurwe're talking about some $magic that will let the cluster notice (and get broken) even if the cluster software is no longer running15:41
TheJuliaindeed, and GFS2 is one of those15:41
dtantsurso, some ring -1 level magic15:41
TheJuliait is not magic actually15:41
TheJuliait is range locking instead of whole device locking15:41
TheJuliafor IO at least15:41
dtantsurrange locking that survives reboot?15:42
TheJuliaDepends on the SAN!15:42
* TheJulia has had to reboot a SAN controller due to it keeping a range lock in RAM15:42
TheJuliaWith VMFS actually15:42
TheJuliaI couldn't vmotion the VMs off the dead hypervisor15:42
dtantsurisn't GFS a software thing?15:42
TheJuliabecause the block ranges were locked15:42
TheJuliaNo, its a kernel module15:42
dtantsurstill, you're talking about a SAN15:43
TheJuliaGFS as in Google Filesystem.. .is unrelated and is software AIUI15:43
TheJuliagfs2 is like any other filesystem, it just does range locking instead of whole device locking15:43
TheJuliaspecifically Red Hat GFS215:44
TheJuliaAnyway, we can move on15:44
TheJuliagot our v6 devstack scripting past Neutron \o/15:44
iurygregoryyay15:45
iurygregoryok, next topic o/15:45
iurygregory#topic Who is going to run the next meeting?15:45
iurygregorydo we have any volunteers?15:45
iurygregoryI will run the next meeting, thanks everyone!15:46
iurygregory#endmeeting15:46
opendevmeetMeeting ended Mon Apr 25 15:46:48 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:46
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-04-25-15.00.html15:46
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-04-25-15.00.txt15:46
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-04-25-15.00.log.html15:46
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: v6/grenade multinode jobs  https://review.opendev.org/c/openstack/ironic/+/83908615:52
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: v6/grenade multinode jobs  https://review.opendev.org/c/openstack/ironic/+/83908615:59
TheJuliaOdds are high that shoudl work15:59
ftarasenkoHi, Ironic!16:02
ftarasenkoajya: Hi! Regarding race condition while cleaning with RAID creation, I have conductor logs uploaded here https://drive.google.com/file/d/1gF7IKTjVP7skO1HQpQuy-S3pr6X5gEdg/view?usp=sharing I think that IPA logs might not help you cause nothing actually happens on IPA. Also I've noticed that after changing to idrac-redfish raid interface additional 50% of nodes succeeded to create RAID. But I still have 2-316:02
ftarasenko nodes in each deployment which do not clean with any RAID interface. I can try to create debug logs if required.16:02
ajyaftarasenko: thanks, will take a look tomorrow and see if I can spot anything unusual16:04
ftarasenkoajya: Sure. Thanks. I have couple of deployments to run additional test at any time if required. Also want to notice that today it was R340 Server and last week it were R650.16:06
TheJuliaiurygregory: so looks like I can get the v6 job working again, looks like Neutron's agent has some issue right now or on the newest ubuntu16:25
TheJuliarpittau: Just fyi, got past enrolling baremetal nodes, and getting networking up with one of the minor devstack fixes under discussion, so things should be fairly good. At least using sushy-tools. Virtualbmc on the other hand...16:25
dtantsurwe don't *have* to use IPMI everywhere16:46
iurygregoryTheJulia, ack16:47
TheJuliadtantsur: agree 100000%16:47
TheJuliaunfortunately we'll still want at least one ipmi job16:47
dtantsurwell, but not necessarily with v6, right?16:48
TheJuliadtantsur: the issue is python 3.10 incompatability with virtualbmc16:50
dtantsur....16:50
dtantsurcause pyghmi, right?16:50
dtantsuras a workaround, add dead snakes, use venv :)16:50
TheJuliaactually, at a glance it was not pyghmi16:51
dtantsur(if someone does not know what dead snakes is: https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa)16:52
TheJulianeat16:52
dtantsurnew ubuntu = jammy?16:53
TheJuliayup16:54
* TheJulia thinks she sees why the multitenacy tests are unhappy16:54
TheJuliaewww, and we run it twice by accident16:57
* dtantsur has just updated his bifrost VMs to CS916:58
dtantsursee you tomorrow folks o/16:58
TheJuliao/17:13
TheJuliahmm.... more than one issue on multinode17:32
rpiosoTheJulia: Regarding https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/826646/2..3/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_cleaning.py#b175, are you suggesting the following?17:56
rpiosoTheJulia: credentials = ['primary', 'system_admin']17:57
TheJuliaso 'primary', 'admin', 'system_admin' I think17:58
TheJuliaat least, until I strip the legacy auth stuff out in aa cycle17:58
rpiosoTheJulia: Isn't that already done by its base class, BaremetalStandaloneManager -- https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/ironic_tempest_plugin/tests/scenario/baremetal_standalone_manager.py#L34?18:01
TheJuliayup18:02
TheJuliain which case, because the test is derived from that, it doesn't need to be defined on that test18:02
rpiosoTheJulia: Thank you for clarifying :)18:02
rpiosoTheJulia: In contrast, do we prefer the min_microversion be explicitly set in the derived class?18:03
TheJuliaThe test should be mapped to a version, where appropriate of course, most aligning the minimum version of ironic which supports the functionality18:04
TheJuliayou don't want to invoke the test and fail when the system doesn't actually support it18:04
TheJuliahopefully that makes sense18:04
rpiosoTheJulia: Got that :)18:05
rpiosoTheJulia:  Similar to credentials, min_microversion is set to the derived test's needed value by its base class -- https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/ironic_tempest_plugin/tests/scenario/baremetal_standalone_manager.py#L3718:06
TheJuliaYes, however if your clean step is exercising something that didn't exist until say, 1.46 internally, you'll want to set that as the minimum version18:16
TheJuliasince the tempest plugin is branchless18:16
rpiosoTheJulia: s/min_microversion/api_microversion/18:20
TheJuliamin microversion is the minimum version the test can be successful at18:20
rpiosoTheJulia: What's api_microversion?18:22
TheJuliaAre you asking what is an API microversion?18:27
rpiosoTheJulia: I'm trying to understand how the test variables min_microversion and api_microversion differ?18:29
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532418:33
TheJuliaI'm not sure there is an explicit api_microversion setting18:33
TheJuliahmm th ere is18:34
TheJuliarpioso: it sets the explicit version to use18:35
TheJuliawhich is a bad habit of ours18:35
TheJuliawe should be using min and max18:35
*** rcastillo_ is now known as rcastillo19:02
* rpioso was in a downstream meeting. Out now :)19:04
rpiosoTheJulia: The new class BaremetalIdracManagementCleaning's base classes set those.19:05
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: v6/grenade multinode jobs  https://review.opendev.org/c/openstack/ironic/+/83908619:06
rpiosoTheJulia: BaremetalStandaloneManager: min_microversion = '1.28' and BaremetalStandaloneScenarioTest: api_microversion = '1.28'.19:07
TheJuliaThat is really an antipattern, tbh19:07
TheJuliamin/max microversion allow client negotiation to occur19:07
rpiosoTheJulia: Should BaremetalIdracManagementCleaning explicitly set either of them?19:08
TheJuliajust min_microversion I think19:08
rpiosoTheJulia: ack re: antipattern and negotiation.19:08
TheJuliaOne more meeting and then I think I'm going to lay down for a little while19:09
admiyoWhen last we left out heroes...I was able to boot using bifrost and the snponly.efi on AARCH 64.  I want to craft this into a patch for Bifrost, but I am not certain when I would need to go snponly (which works for me) and when to go for something else.19:09
admiyoFun fact.  SNP stands for Simple Network Protocol.  It is not a Network Protocol.  It is an API for talking to network cards.  I've been told that this is a Microsoftism.19:10
rpiosoTheJulia: The already existing cleaning standalone tests explicitly set only api_microversion: https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_cleaning.py.19:10
TheJuliaadmiyo: so I think it is part of the UEFI standard, but maybe not by that exact name19:11
TheJuliarpioso: :(19:11
admiyoPretty sure it is that exact name19:11
admiyoI looked in the EUFI standard19:11
admiyoUEFI.  Man, I need to stop assuming I can edit my chats.19:12
TheJuliaadmiyo: So.... we've basically found that for UEFI, snp only is the only way19:12
TheJuliaat least, in more recent firmware19:12
TheJuliaand by that... new in the last 4-5 years19:12
TheJulias/UEFI/EFI/19:12
TheJuliait is not universal19:12
admiyoOK, so I can make snponle.efi the default, at lease for arm64?19:15
admiyoTheJulia, the other change I would need to make is that for aarch6, get the firmware from a subdir: https://boot.ipxe.org/arm64-efi/  so a little switch logic based on arch19:17
admiyoother than that it should be minimal.  Which makes for faster patch reviews and acceptance.  19:18
TheJuliayeah, that should be reasonable19:18
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532419:25
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: v6/grenade multinode jobs  https://review.opendev.org/c/openstack/ironic/+/83908619:44
rpiosoTheJulia: Thank you for your assistance. It was very helpful. I summarized my understanding in comments I posted to the review.19:47
opendevreviewHarald Jensås proposed openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532419:48
TheJuliarpioso: happy to help20:24

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!