Wednesday, 2020-09-30

*** Goneri has quit IRC00:09
*** Goneri has joined #openstack-ironic00:15
*** Goneri has quit IRC00:29
*** yolanda has quit IRC00:38
*** yolanda has joined #openstack-ironic00:39
openstackgerritXuan Yandong proposed openstack/networking-baremetal master: Remove Six  https://review.opendev.org/75514800:54
*** gyee has quit IRC01:08
*** damiandabrowski has quit IRC01:23
openstackgerritMerged openstack/ironic master: Fix a race condition in the hash ring code  https://review.opendev.org/75436201:33
*** QianbiaoNG has joined #openstack-ironic01:38
*** yolanda has quit IRC02:06
*** zzzeek has quit IRC02:07
*** zzzeek has joined #openstack-ironic02:10
*** yolanda has joined #openstack-ironic02:13
*** arne_wiebalck has quit IRC02:14
*** rajinir has quit IRC02:14
*** vkmc has quit IRC02:14
*** phrobb has quit IRC02:14
*** dking has quit IRC02:14
*** tkajinam has quit IRC02:14
*** tkajinam_ has joined #openstack-ironic02:15
*** vkmc has joined #openstack-ironic02:15
*** phrobb has joined #openstack-ironic02:15
*** arne_wiebalck has joined #openstack-ironic02:15
*** QianbiaoNG has quit IRC02:18
*** rajinir has joined #openstack-ironic02:20
*** dking has joined #openstack-ironic02:20
*** QianbiaoNG has joined #openstack-ironic02:31
*** prometheanfire has left #openstack-ironic02:34
*** rcernin has quit IRC02:55
*** rcernin_ has joined #openstack-ironic02:56
openstackgerritSteve Baker proposed openstack/ironic master: Utility functions for REST API JSON handling  https://review.opendev.org/75116003:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert allocations endpoint to plain JSON  https://review.opendev.org/74367903:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert bios endpoint to plain JSON  https://review.opendev.org/74547503:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert chassis endpoint to plain JSON  https://review.opendev.org/74547603:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert conductors endpoint to plain JSON  https://review.opendev.org/74577903:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert deploy_templates endpoint to plain JSON  https://review.opendev.org/74578003:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert drivers endpoint to plain JSON  https://review.opendev.org/74186503:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert event endpoint to plain JSON  https://review.opendev.org/74662303:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert nodes endpoint to plain JSON  https://review.opendev.org/74782003:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert ports endpoint to plain JSON  https://review.opendev.org/75012003:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert portgroups endpoint to plain JSON  https://review.opendev.org/75086403:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert ramdisk endpoint to plain JSON  https://review.opendev.org/75085703:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume endpoint to plain JSON  https://review.opendev.org/75085803:00
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume/connectors endpoint to plain JSON  https://review.opendev.org/75085903:01
openstackgerritSteve Baker proposed openstack/ironic master: Convert volume/targets endpoint to plain JSON  https://review.opendev.org/75086003:01
openstackgerritSteve Baker proposed openstack/ironic master: Expunge the internal version of WSME  https://review.opendev.org/75086103:01
*** rh-jelabarre has quit IRC04:10
*** Lucas_Gray has joined #openstack-ironic04:31
*** abdysn has joined #openstack-ironic05:22
*** Lucas_Gray has quit IRC05:32
*** Lucas_Gray has joined #openstack-ironic05:38
*** jtomasek has joined #openstack-ironic05:57
*** yolanda has quit IRC06:10
*** tzumainn has quit IRC06:19
*** portdirect has quit IRC06:20
*** portdirect has joined #openstack-ironic06:20
openstackgerritAnkit Kumar proposed openstack/ironic master: Adds config parameter kernel_append_param for iLO  https://review.opendev.org/75518906:34
arne_wiebalckGood morning, ironic!06:38
ftarasenkoGm, Ironic! dtantsur: https://review.opendev.org/754362 worked for me in Train.06:51
patchbotpatch 754362 - ironic - Fix a race condition in the hash ring code (MERGED) - 9 patch sets06:51
*** yolanda has joined #openstack-ironic07:05
*** rcernin_ has quit IRC07:06
iurygregorygood morning arne_wiebalck ftarasenko and Ironic07:17
*** rcernin_ has joined #openstack-ironic07:17
*** rcernin_ has quit IRC07:20
*** rcernin has joined #openstack-ironic07:20
*** jtomasek has quit IRC07:30
*** tosky has joined #openstack-ironic07:30
*** Lucas_Gray has quit IRC07:32
*** jtomasek has joined #openstack-ironic07:33
*** ociuhandu has joined #openstack-ironic07:50
*** ociuhandu has quit IRC07:54
SuiongNg|ITRI_TWHi everyone! I recently stumbled upon a problem that prevents me from entering the rescue mode. And I think wiping the agent token on power_on event can solve it. Unfortunately, I'm not familiar with the opendev workflow. I would be happy if anyone could take a look at https://review.opendev.org/754260/ and help me out.08:03
patchbotpatch 754260 - ironic - wipe token on power_on event. - 1 patch set08:03
*** ociuhandu has joined #openstack-ironic08:09
*** lucasagomes has joined #openstack-ironic08:11
iurygregoryI'm not 100% sure that by only adding POWER_ON it will work, it may work ofc, but I think it should be checked if it's during rescue....08:12
mgoddardmorning. Seeing an odd issue in kayobe CI on our ussuri -> master upgrade job. We register some nodes with the iscsi deploy interface on ussuri, then after upgrade they have switched to direct. A fresh install on master does not see this. default_deploy_interface is iscsi in all cases08:13
mgoddarddtantsur|afk:, maybe you know? ^08:13
SuiongNg|ITRI_TWiurygregory: thanks. I second your opinion that this needs to be reviewed by someone familiar with the conductor. I'm not sure if this is the best way to fix the problem or it had been fixed recently. I've only tested it against the victoria pre-release version in ubuntu's cloud archive. It works for me but might not be the case for everyone else.08:21
iurygregoryyeah, I have the feeling this may shoot us on the foot (in other scenarios)08:23
*** k_mouza has joined #openstack-ironic08:28
QianbiaoNGHi ironic.08:41
iurygregoryhello QianbiaoNG08:41
QianbiaoNGGood morning, iurygregory08:42
QianbiaoNGmorning arne_wiebalck o/08:42
arne_wiebalckHey iurygregory and QianbiaoNG o/08:44
iurygregoryarne_wiebalck, o/08:44
*** derekh has joined #openstack-ironic08:45
openstackgerritMerged openstack/sushy stable/train: Fix #Bios.ResetBios for HTTP 400 Bad request error  https://review.opendev.org/75465508:46
openstackgerritMerged openstack/sushy stable/victoria: Fix #Bios.ResetBios for HTTP 400 Bad request error  https://review.opendev.org/75444308:46
openstackgerritMerged openstack/sushy stable/ussuri: Fix #Bios.ResetBios for HTTP 400 Bad request error  https://review.opendev.org/75465408:48
*** rcernin has quit IRC08:48
*** abdysn has quit IRC08:59
openstackgerritMerged openstack/ironic master: Fix handling OctetString for pysnmp  https://review.opendev.org/75040109:06
openstackgerritMerged openstack/ironic master: Minor agent version code cleanup  https://review.opendev.org/74955209:06
*** derekh has quit IRC09:32
*** derekh has joined #openstack-ironic09:33
*** mraineri has quit IRC09:36
*** Lucas_Gray has joined #openstack-ironic10:31
*** dansmith has quit IRC10:53
*** dansmith has joined #openstack-ironic10:55
*** Lucas_Gray has quit IRC11:03
*** dtantsur|afk is now known as dtantsur11:11
dtantsurmorning ironic11:12
dtantsurftarasenko: thanks for confirming, I'll backport it11:12
dtantsurmgoddard: yep, there is an upgrade procedure that does precisely that11:12
mgoddardthanks dtantsur, I managed to find it in the commit history eventually.11:27
* dtantsur is glad it works, less glad that it breaks people11:27
mgoddarddtantsur: it's not ideal in the case where the direct interface is enabled but not correctly configured11:27
dtantsuryeah, I tried having some safeguards, but maybe not enough. any suggestions?11:28
mgoddardyou could argue our config is wrong, but we generally rely on the default interface11:28
dtantsurmmm, maybe we should just default_deploy_interface!=iscsi..11:29
dtantsurI usually recommend people avoid default_* options, but I cannot really claim it's wrong11:29
mgoddardthat might help11:30
mgoddardat least in this case11:30
dtantsurmgoddard: wanna propose a patch or should I? it may take me a bit of time11:35
dtantsurjanders: re whole disk images: you want to chat right now or may I grab a tea first?11:37
dtantsurI know it's a bit late for you11:37
* dtantsur declares tea time11:39
jandersdtantsur go ahead and grab your tea :)11:39
jandersI will work on this with iurygregory for another 90mins at least11:39
janderswe need to get this done11:39
iurygregoryIPE is answering \o/ now I just need to enroll the nodes \o/11:40
iurygregoryfingers crossed11:40
* janders is looking11:40
* janders is enrolling the first node11:41
janderslets see if it starts getting power info11:41
*** rcernin has joined #openstack-ironic11:50
*** rh-jelabarre has joined #openstack-ironic11:51
dtantsurit didn't take 90 mins :)11:56
jandersso far we figured that janders can't spell "Administartor"11:58
jandersgood I never pursued career as a Windows guy11:58
jandersI can spell root, this is as far as it goes11:58
dtantsurnow we know why they picked "root"12:05
dtantsuralso yes, administrator is a difficult word to spell12:06
openstackgerritDmitry Tantsur proposed openstack/ironic stable/ussuri: Fix a race condition in the hash ring code  https://review.opendev.org/75526712:07
openstackgerritDmitry Tantsur proposed openstack/ironic stable/train: Fix a race condition in the hash ring code  https://review.opendev.org/75526812:07
dtantsurSuiongNg|ITRI_TW: hi, do you have https://review.opendev.org/#/c/744655/ in your environment?12:08
patchbotpatch 744655 - ironic - Wipe agent token and URL on rescue and unrescue (MERGED) - 1 patch set12:08
dtantsuriurygregory: if you could spare a minute: https://review.opendev.org/#/c/755001/12:09
patchbotpatch 755001 - ironic - Trivial: fix formatting in the prelude - 1 patch set12:09
iurygregorydtantsur, sure12:09
iurygregory+W12:09
dtantsurhttps://review.opendev.org/#/c/754884/ is also trivial12:09
patchbotpatch 754884 - ironic - Handle patching node /protected value with None - 2 patch sets12:09
dtantsurI guess I should have asked for a unit test.. maybe stevebaker could follow-up with one12:10
iurygregoryI was about to say =)12:11
iurygregoryI will +W and ask for the test12:11
dtantsur++12:12
dtantsurnow, https://review.opendev.org/#/c/726483/ would be awesome to have for the release.. but requires A LOT of brainpower12:12
patchbotpatch 726483 - ironic - Guard conductor from consuming all of the ram - 17 patch sets12:12
dtantsurarne_wiebalck: I think you've started reviewing ^^ already, do you think you could check it?12:12
mgoddarddtantsur: I can pick it up. Might be tomorrow12:12
iurygregoryI think arne_wiebalck was trying to test12:12
iurygregoryI would look a bit more, but my brain power today is for trivial things and my demo XD12:13
dtantsurmgoddard: ack12:13
arne_wiebalckdtantsur: my promised review is to test this with real nodes at scale12:13
dtantsuriurygregory: don't worry then. I may ping you for my minor stuff.12:13
dtantsurarne_wiebalck: the thing is, we should cut stable/victoria ASAP. so if doesn't get merged soon, it's pushed to Wallaby12:13
arne_wiebalckdtantsur: still working on this ...12:13
arne_wiebalckdtantsur: yes12:14
arne_wiebalckdtantsur: do you know if others saw this?12:14
arne_wiebalckdtantsur: this is mostly for iscsi deploy (and direct with http download)12:14
dtantsurarne_wiebalck: I gave a +2, there have been a few other reviews previously12:15
dtantsuryes12:15
*** rcernin has quit IRC12:16
arne_wiebalckdtantsur: my point is, would it be a problem to push it? ASAP does not  sound like I am going to make it :)12:16
arne_wiebalckdtantsur: or we merge without testing at scale12:16
dtantsurarne_wiebalck: I don't think so. but we need a 2nd review from someone.12:16
arne_wiebalckdtantsur: yes, but only if we want to have it for V12:17
arne_wiebalckdtantsur: what's the timeline12:17
dtantsurah, that's what you mean. it's not terrible to push it to W, but TheJulia might be upset12:17
arne_wiebalck?12:17
iurygregorydtantsur, if things go fast I may have more time12:17
iurygregory\o/12:17
*** Goneri has joined #openstack-ironic12:17
iurygregory1 node collecting metrics12:18
iurygregory1 to go12:18
iurygregoryand I can record my demo12:18
iurygregory\o/12:18
arne_wiebalckdtantsur: yes re TheJulia12:18
dtantsurarne_wiebalck: we should have made stable/victoria by last Friday :)12:18
arne_wiebalckdtantsur: oops12:18
dtantsurso ASAP is pretty literally ASAP12:18
dtantsurmeanwhile, does anyone have anything for bifrost victoria? if no, does anyone want to propose the final release?12:19
arne_wiebalckdtantsur: ok, no way I can test it (too much downstream in parallel atm), and there is still the issue that 5-10% of the RAIDs are not created properly12:19
dtantsurarne_wiebalck: okay, let's wait for TheJulia to decide if she's comfortable for it to go in without testing12:20
arne_wiebalckdtantsur: so, we will need to decide to merge without testing (from me)12:20
arne_wiebalckdtantsur: ++12:20
dtantsurmoving on, inspector is blocked by https://review.opendev.org/#/c/753435/. it's not trivial, but not overly complex either. any volunteers for a 2nd review?12:21
patchbotpatch 753435 - ironic-inspector - pxe filter - option to always block unknown hosts - 6 patch sets12:21
arne_wiebalckdtantsur: I can have a look.12:24
dtantsurthx12:24
dtantsuriurygregory: do you have plans to update https://review.opendev.org/#/c/752042/ today?12:36
patchbotpatch 752042 - bifrost - Add ipe installation to bifrost - 7 patch sets12:36
iurygregoryif the demo goes well yes12:36
iurygregory=)12:36
dtantsur:)12:36
dtantsurit's the last item before we can release12:36
iurygregoryyeah I'm trying to figure out why it fails to download the ipe repo...12:36
iurygregoryon centos812:37
dtantsuriurygregory: if you have something messed up around source_install, hold on12:39
dtantsurs/if//12:39
iurygregoryhummm12:39
dtantsuriurygregory: yep, commented12:40
iurygregorydtantsur, tks12:40
jandersdtantsur sorry for the delay, got bogged in all sorts of BMC issues12:43
jandersdtantsur on whole-disk-images...12:43
janders1) do I still need them for true-baremetals?12:43
jandersI used to do it this way but maybe it's a 2018 thing...12:43
dtantsurjanders: we still support both types of images12:44
dtantsurdifferent operators have completely different feelings about which image type is better12:44
janderswhat's the easiest way for ironic/standalone in your opinion?12:44
dtantsurjanders: whole disk, I would say12:44
jandersokay!12:44
janderswhats the easiest way to get a wholedisk image?12:44
dtantsurjanders: GenericCloud from https://cloud.centos.org/centos/8/x86_64/images/ may just work, to be honest12:45
dtantsurat least CentOS 7 image worked without modifications12:45
dtantsurif you need any customization, diskimage-builder is your friend (hahaha)12:45
dtantsurI mean, it can be terrible at times, but it works12:45
dtantsurbifrost has an ability to build images for you12:45
jandersI hope stock images should be enough for what we need12:47
janderswhat kind of checksum is best?12:47
jandersI see CentOS mirrors use sha25612:47
jandersbut Ive seen shorter ones in /httpboot12:47
dtantsurwe default to md512:47
jandersok12:47
dtantsuryou can use sha256, but they go into non-default fields12:47
dtantsurhold on, I'll pull the docs12:47
dtantsurjanders: https://docs.openstack.org/ironic/latest/install/standalone.html is a bit messy, but it does have the information12:48
dtantsursee https://docs.openstack.org/ironic/latest/install/standalone.html#populating-instance-info12:48
*** ricolin_ has joined #openstack-ironic12:48
jandersgreat, thank you dtantsur12:49
*** rloo has joined #openstack-ironic12:49
jandersI will try CentOS8 generic12:49
dtantsurjanders: please put any findings to https://docs.openstack.org/ironic/latest/install/creating-images.html if they're not there12:50
jandersah another thing - the kit we are testing on has heaps of disks12:51
jandersare root disk hints a thing in standalone?12:51
dtantsuryep12:51
dtantsurwork exactly the same. you can put them to properties (permanent) or instance_info (per instance)12:51
jandersgot it12:51
jandersi think I can see it in the same doco page you linked12:51
openstackgerritMerged openstack/ironic master: Trivial: fix formatting in the prelude  https://review.opendev.org/75500112:53
*** derekh has quit IRC13:01
*** derekh has joined #openstack-ironic13:03
openstackgerritVerification of a change to openstack/ironic failed: Handle conductor_affinity earlier in the deployment process  https://review.opendev.org/75437513:07
* dtantsur rolls eyes13:09
TheJuliaCI hates everyone13:09
dtantsurmorning TheJulia, this is true13:09
dtantsurI'm trying to wrap up releases today13:09
dtantsurwe're looking good, but all projects have at least one outstanding patch13:10
dtantsurTheJulia: FYI I'll be out tomorrow and on Friday13:10
* dtantsur launching an internal spoon recollection routine13:10
*** ricolin_ has quit IRC13:11
TheJuliaYeah, I was hoping we would be about there today13:11
TheJuliaspoon collection ++13:11
TheJuliaLike Java GC, it is13:11
dtantsurexactly13:12
*** tkajinam_ has quit IRC13:12
dtantsurno mountains any soon, so need to reclaim whatever is possible13:12
TheJuliaarne_wiebalck: replied to https://review.opendev.org/#/c/75230813:17
patchbotpatch 752308 - ironic-python-agent - Don't update the grubenv - 5 patch sets13:17
TheJuliaarne_wiebalck: I guess the point is there seems to be no fix for grub, even without raid it launches os-prober apparently and then the grub-mkconfig command goes out to lunch13:19
arne_wiebalckTheJulia: But only with --dont-update-grubenv?13:20
*** Lucas_Gray has joined #openstack-ironic13:20
arne_wiebalckTheJulia: The bugzilla was not really clear to me as the even a command with that option took 2 mins to complete.13:21
TheJulia--no-grubenv-update  is the option I found in another bug that was bascially the same issue13:22
TheJuliait is of course, undocumented \o/13:22
arne_wiebalckYeah, I realised :-D13:22
TheJuliaYeah, there do seem to be lots of reports of issues with newer grubs hanging and taking forever and they all seem to revolve around os-prober firing up13:23
arne_wiebalckSo the other bug said using the option fixed the timing?13:23
TheJuliaI don't remember seeing that13:23
TheJuliaoh you mean executed without a crazy delay13:23
arne_wiebalckyes13:24
* arne_wiebalck is looking for the logical connection13:24
TheJuliayeah, apparently from what I've gathered os-prober is not executed then13:24
*** priteau has joined #openstack-ironic13:24
arne_wiebalckthat is what I mean :)13:24
TheJuliano worries13:24
* TheJulia goes and gets coffee since words are hard this morning13:24
arne_wiebalckThe referenced bug still has bad timing.13:24
arne_wiebalckWIth the option.13:24
TheJuliaI think yeah, 20 seconds as opposed to 20+ minutes from what my brain processed13:26
*** Lucas_Gray has quit IRC13:30
*** ricolin_ has joined #openstack-ironic13:31
openstackgerritJulia Kreger proposed openstack/ironic stable/ussuri: CI: Additional minor job changes to improve CI reliability  https://review.opendev.org/75510013:31
TheJuliaany core reviewers have a chance to look at https://review.opendev.org/#/c/754717/13:34
patchbotpatch 754717 - ironic-python-agent - Software RAID: Get component devices by md UUID - 3 patch sets13:34
arne_wiebalckTheJulia: Hmm ... the 19 mins and the 2 mins are from different users. And the 2 mins is compared with 2 secs depending on what is on the disk, not which options are used: https://bugzilla.redhat.com/show_bug.cgi?id=174469313:36
openstackbugzilla.redhat.com bug 1744693 in grub2 "grub2-mkconfig does not work correctly with os-prober" [High,Closed: eol] - Assigned to pjones13:36
*** tzumainn has joined #openstack-ironic13:37
openstackgerritMerged openstack/ironic master: Update 'idrac' driver requirements for Victoria  https://review.opendev.org/75486813:38
TheJuliaarne_wiebalck: I may have copied the wrong bz down since when I was searching originally I think I ran across 8 filings or something that made my had hurt13:39
TheJuliamy head13:39
arne_wiebalckTheJulia: ah!13:40
TheJuliaInteresting, disabling it in grub.cfg made it run long13:40
TheJuliawtf13:40
arne_wiebalckLOL13:40
arne_wiebalckbtw, I checked with our Linux folks here, but we have not run into this yet, it seems13:41
arne_wiebalckTheJulia: the other thing dtantsur and I were discussing earlier is the RAM guard patch13:42
arne_wiebalckTheJulia: I am still working on doing scale tests, but after the cleaning issues, I am now looking into the UEFI RAID creation issue13:42
arne_wiebalckTheJulia: so, it seems unlikely I can reproduce the OOM issue and then test the patch IRL13:43
TheJuliarealistically your not the only person who has oomed the conductor though :)13:43
arne_wiebalckTheJulia: we were wondering if you would like the  patch to go in for Victoria without these tests, or if you prefer to wait13:43
TheJuliawe've actually partially seen it in CI too13:43
arne_wiebalckTheJulia: ok13:44
arne_wiebalckTheJulia: I was missing this bit13:44
TheJuliaof course, now that is even worse with only 1gb of swap13:44
TheJuliaeh, in ci it just creates a random variable13:44
TheJuliain terms of how long the job may take becuase overall system memory pressure13:44
arne_wiebalckTheJulia: I was wondering if others suffered which would mean it should go in early13:44
arne_wiebalckTheJulia: s/early/now/13:44
arne_wiebalckTheJulia: for V13:44
TheJuliaI can't think of anyone who has been really vocal, I just don't want to have to wait until the end of W13:45
arne_wiebalckTheJulia: ok13:45
TheJuliaso after branching seems reasonable to me if we want to take a wait and see approach13:45
arne_wiebalckTheJulia: sorry for not being able to give feedback earlier, but machines arrived later than expected and then I ran into all these other issues ...13:46
TheJuliasuch is life :)13:46
arne_wiebalckTheJulia: is backporting an option in case we decide to wait?13:46
TheJuliaOff hand, I don't see why not13:47
dtantsurmmm, because it's potentially breaking?13:47
TheJuliawell, yeah, overall behavior wise but for good reasons13:47
dtantsurwe may be missing a case where ironic barely works now but will be broken after this patch13:47
dtantsurcan we backport it, but with the check disabled? i.e. required_memory==0?13:48
dtantsuror like 256?13:48
TheJuliabut that was likely an already unstable case13:48
dtantsurbut don't like unstable cases turning into broken cases :)13:48
TheJuliayeah, I guess we could13:48
TheJuliawell, unstable turning into broken at least forces it to be fixed13:48
dtantsurwe assume that image conversion takes up to 1G. if their images are tiny (cirros-alike), much less may be needed13:49
dtantsureven worse, they may start hitting retries and not realizing it at first13:49
TheJuliaWe could at least log a "well, your memory is not in a good state" warning and hope people pickup on it13:49
dtantsursounds nice. "it will become an error in the Wallaby release"13:50
TheJuliasounds reasonable to me13:50
dtantsurhmm, but we still need a way to opt into a full error with retries13:51
TheJuliathat way we can backport it just leave it disabled, cern can enable it and hopefully not kill their ironic-conductor processes :)13:51
TheJuliai think if set to 0, then it just logs a warning if below 102413:51
TheJuliaif set to any value then it would be enabled and engage the logic13:52
TheJuliawe know 1024M makes sense because of the ironic-lib default13:52
TheJuliaI could make the patch do that and we could merge it to W, and backport it to V, and then in W just change the default13:53
TheJuliaanyone want to review https://review.opendev.org/#/c/745598/13:55
patchbotpatch 745598 - ironic-python-agent - Add basic retries for inspection - 4 patch sets13:55
openstackgerritMudit proposed openstack/ironic master: Add GPU reporting to idrac-wsman inspect interface  https://review.opendev.org/75100613:57
*** lmcgann_ has joined #openstack-ironic13:57
dtantsurTheJulia: sounds like a plan13:59
*** k_mouza has quit IRC13:59
openstackgerritMark Goddard proposed openstack/ironic master: Don't migrate away from iscsi if it is the default  https://review.opendev.org/75529514:06
openstackgerritDmitry Tantsur proposed openstack/sushy-tools master: Handle race conditions when creating a directory for PersistentDict  https://review.opendev.org/75529714:10
dtantsurstbenjam: this ^^ should get you going14:10
dtantsurmy comment about this code needing some cleansing fire stands though14:12
*** ricolin_ has quit IRC14:14
TheJuliaWould that be cleansing fire with klingon opera playing in the background?14:14
dtantsurlikely14:15
openstackgerritVerification of a change to openstack/ironic failed: Handle patching node /protected value with None  https://review.opendev.org/75488414:20
*** bhagyashri|rover is now known as bhagyashri14:20
mgoddarddtantsur, TheJulia thanks for the fast reviews :)14:27
TheJuliahmmmm the ibmc devstack change looks okay but I'm guessing downstream merge conflicts are tanking all of the 3rd party ci jobs14:28
TheJuliaalso the failure in our gate seems same old fun issues around rescue14:28
*** k_mouza has joined #openstack-ironic14:34
*** cdearborn has joined #openstack-ironic14:36
*** k_mouza has quit IRC14:39
openstackgerritMudit proposed openstack/ironic master: Add GPU reporting to idrac-wsman inspect interface  https://review.opendev.org/75100614:41
*** rajinir has quit IRC14:45
*** rajinir has joined #openstack-ironic14:46
janderssee you tomorrow Ironic o/14:48
jandersbaremetal node power off janders14:48
openstackgerritJulia Kreger proposed openstack/ironic stable/ussuri: CI: Additional minor job changes to improve CI reliability  https://review.opendev.org/75510014:56
arne_wiebalckjanders: :D14:58
* arne_wiebalck thinks that most of the ironic devs are on fasttrack 14:59
*** gyee has joined #openstack-ironic14:59
iurygregorybifrost enroll --15:03
iurygregoryironic cli enroll ++15:03
openstackgerritAnkit Kumar proposed openstack/ironic master: Adds config parameter kernel_append_param for iLO  https://review.opendev.org/75518915:06
*** k_mouza has joined #openstack-ironic15:17
openstackgerritManuel Schönlaub proposed openstack/sushy master: Add oem_vendors from manager links  https://review.opendev.org/75532315:23
*** QianbiaoNG has quit IRC15:26
openstackgerritMerged openstack/ironic-inspector master: pxe filter - option to always block unknown hosts  https://review.opendev.org/75343515:32
*** belmoreira has joined #openstack-ironic15:33
dtantsurTheJulia: a very reasonable complaint from stbenjam: deploying only one node in parallel in the CI does not allow us to catch races15:33
dtantsurI have no idea what to do about it. Ideally we should have at least something with high concurrency15:33
TheJuliadtantsur: multinode jobs have a deploy everything job that runs if memory serves15:34
dtantsurnope, no longer15:35
TheJuliaoh really? shoot15:35
dtantsurwe disabled that test because it caused too many sporadic failures15:35
dtantsurI don't know how to balance the desire to test things in parallel with the fact that it increases random failures quite substantially15:35
TheJuliaWe can't... not with our CI resources15:36
TheJulia8 GB of ram, and the amount of data we have to ship to do even a multinode deployment is just a ton of potential risk for sporatic failures due to underlying infrastucture15:36
dtantsuryep :(15:37
dtantsurmaybe one day we'll see a 3rd party CI from OCP15:37
dtantsurmeanwhile, I feel like we're ready to release ironic-inspector. anyone wants to add anything or review the renos?15:37
* TheJulia goes and looks15:38
TheJuliaLGTM15:40
TheJulia10.4.0 ?15:40
dtantsurlemme dbl-check15:45
dtantsurTheJulia: yep, a nice and small release15:46
dtantsurwanna propose it?15:46
openstackgerritDhuldev Valekar proposed openstack/ironic master: DRAC: Added redfish management clean steps  https://review.opendev.org/72159315:46
*** openstackgerrit has quit IRC15:46
TheJuliadtantsur: sure15:47
*** k_mouza_ has joined #openstack-ironic16:03
*** k_mouza has quit IRC16:06
*** lucasagomes has quit IRC16:06
dtantsurcould someone please review https://review.opendev.org/#/c/755297/ to make metal3 folks a bit happier?16:10
patchbotpatch 755297 - sushy-tools - Handle race conditions when creating a directory f... - 1 patch set16:10
iurygregorywoot16:12
iurygregoryfunny16:13
iurygregory+216:13
iurygregoryif no other core show up I will +W after finishing my videos16:14
dtantsurI have interesting feelings every time I dive into sushy-tools code :)16:14
dtantsurthanks!16:14
TheJuliaapproved16:16
dtantsur\o/16:18
* dtantsur starts talking to people about a 3rd party CI16:18
*** tosky has quit IRC16:19
TheJuliaoooh ahh16:24
*** thiagop has joined #openstack-ironic16:47
TheJuliafor ironic, is it just the hash ring patch we're thinking at this point?16:48
dtantsurthe hash ring patch has merged to master, I think there was something else..16:49
* dtantsur checks open patches16:49
TheJulia754375 ?16:50
TheJuliaplus mgoddard's chagne16:50
dtantsurhttps://review.opendev.org/755295 https://review.opendev.org/75488416:50
patchbotpatch 755295 - ironic - Don't migrate away from iscsi if it is the default - 1 patch set16:50
patchbotpatch 754884 - ironic - Handle patching node /protected value with None - 2 patch sets16:50
dtantsurhttps://review.opendev.org/#/c/754375/ is not blocking16:50
patchbotpatch 754375 - ironic - Handle conductor_affinity earlier in the deploymen... - 4 patch sets16:50
TheJuliaagreed16:50
TheJuliaok, well if there are no objections and those merge today I can go ahead and tag later today, what about IPA, I think that seems settled since the bugs we can just backport16:51
dtantsurTheJulia: wanna me check the IPA uuid patch quickly?16:54
dtantsurit's the closest to approval16:55
* TheJulia pull sup IPA16:55
dtantsurI've approved it16:56
TheJuliak16:56
TheJuliathe other raid patch is also a fix16:56
*** ociuhandu has quit IRC16:57
TheJuliahttps://review.opendev.org/#/c/755062 looks good to me16:57
patchbotpatch 755062 - ironic-python-agent (stable/ussuri) - Software RAID: Don't delete partitions too early - 1 patch set16:57
TheJuliaand just pushes back the wipefs until later16:57
*** derekh has quit IRC17:00
dtantsurmm, I suspect we can release ironic-ui if it has anything to release. at least branch it.17:11
*** openstackgerrit has joined #openstack-ironic17:18
openstackgerritVerification of a change to openstack/sushy-tools failed: Handle race conditions when creating a directory for PersistentDict  https://review.opendev.org/75529717:18
* dtantsur rolls eyes17:19
JayFthat's the second DB-related failure in unit tests I've seen17:20
dtantsuroh, interesting, it's a race in related unit tests17:20
JayFis there an infra issue, or actual race?17:20
JayFyeah :D17:20
dtantsuran actual race17:20
dtantsurI won't comment on sushy-tools code and design. I simply refuse.17:20
dtantsurit doesn't help that I cannot reproduce it locally17:22
dtantsuranyway, I'm out for a short PTO. have a great weekend and see you on Monday.17:22
*** dtantsur is now known as dtantsur|afk17:23
iurygregoryJayF, DB failures on tests are the best!17:32
iurygregorywhen you want to merge something very fast they will happen =D17:32
*** k_mouza_ has quit IRC17:40
*** thiagop has quit IRC17:56
*** thiagop has joined #openstack-ironic18:08
TheJuliahmmm... I thought ironic-ui already had been18:11
* TheJulia will look after som elunch18:11
openstackgerritMerged openstack/ironic master: Don't migrate away from iscsi if it is the default  https://review.opendev.org/75529518:20
* TheJulia signs due to ussuri failures18:28
openstackgerritMerged openstack/ironic master: Handle patching node /protected value with None  https://review.opendev.org/75488418:30
openstackgerritMerged openstack/ironic-python-agent master: Software RAID: Get component devices by md UUID  https://review.opendev.org/75471718:30
*** thiagop has quit IRC18:31
*** thiagop has joined #openstack-ironic18:35
openstackgerritMerged openstack/ironic-ui master: Add npm tox env  https://review.opendev.org/75443918:43
*** bdodd has quit IRC18:52
*** priteau has quit IRC18:56
*** ociuhandu has joined #openstack-ironic19:02
*** belmoreira has quit IRC19:05
*** ociuhandu has quit IRC19:06
*** bdodd has joined #openstack-ironic19:11
*** thiagop has quit IRC19:30
openstackgerritMerged openstack/sushy-tools master: Handle race conditions when creating a directory for PersistentDict  https://review.opendev.org/75529719:31
*** tosky has joined #openstack-ironic19:32
mushrushuhey! not sure if it's the right place to ask but is there any special procedure to change ip address of an ironic host deployed with bifrost? I've changed the ip in every config that I could find (including OS_ENDPOINT) but am still getting errors like "unable to establish connection to http://OLD_IP:6385/v1/nodes19:36
*** bdodd has quit IRC19:39
iurygregorydid you re-run the playbooks (it should just override the new configs and put the services to run again19:46
iurygregoryif you have nodes that are trying to talk with the old ironic and you want to update them this will probably be a bit different...19:47
*** bdodd has joined #openstack-ironic19:51
mushrushu@iurygregory you mean like deployed barametal nodes?19:53
iurygregoryyeah19:53
mushrushuhuh, i had a test one, but i've deleted it in mysql. could it be the cause?19:55
iurygregoryyou only deleted in mysql?19:55
mushrushuyup, i was not able to perform any openstack baremetal command, so i've deviceded that should be a good idea :)19:56
iurygregoryif you do a curl http://new_ip:6385/v1/nodes19:58
iurygregoryyou get a valid response right?19:58
TheJuliatbh, I'd just re-install wish the ip address in the configuration, the templates will be updated with the new address and it will restart services... It sounds like clouds.yaml might not have been updated though19:59
iurygregorywhat I understood is that the error messages are on logs even after updating to use the new ip (this shouldn't happen afaik)20:02
mushrushu@iurygregory I haven't tried. the environment is on remote site so i can not check it right now.20:02
TheJuliamushrushu: was one of the files you edited clouds.yaml?20:03
mushrushu@TheJulia no, the thing is i've seen it being mentioned in the docs but couldn't find it on my vm even with find20:04
mushrushui've definitely edited ironic.conf inspector.conf dnsmasq.conf /httpboot/pxelinux.cfg/default and openrc20:07
iurygregoryif you edited manualy and you are trying "baremetal node list"20:12
iurygregoryit won't work because the clouds.yaml will point to the old ip20:12
openstackgerritMerged openstack/ironic-python-agent stable/ussuri: Software RAID: Don't delete partitions too early  https://review.opendev.org/75506220:12
mushrushuwhere should that file be? Is it like a mandatory one? find / -name "cloud.yaml" gives me nothing20:14
*** jtomasek has quit IRC20:15
mushrushuoh20:16
TheJuliatry looking at ~/.config/openstack/clouds.yaml20:16
mushrushui see a typo, sorry20:16
TheJuliaoh, okay20:16
mushrushui was looking for cloud.yaml the whole time not clouds20:17
mushrushuthanks for your help and patience20:19
openstackgerritIury Gregory Melo Ferreira proposed openstack/bifrost master: Add ipe installation to bifrost  https://review.opendev.org/75204220:19
stevebakermorning20:20
iurygregorymorning stevebaker o/20:29
openstackgerritMerged openstack/ironic master: Handle conductor_affinity earlier in the deployment process  https://review.opendev.org/75437520:50
TheJuliarpioso: rechecking stable/ussuri is not going to help right now21:03
rpiosoTheJulia: Ack21:04
TheJuliaI've got a patch where I'm trying to wrap everything together to fix CI in general due to the infra changes but I've just not had any luck yet21:04
TheJuliaany last calls before I tag releases?21:09
* TheJulia makes some fresh coffee21:17
openstackgerritIury Gregory Melo Ferreira proposed openstack/bifrost master: Add ipe installation to bifrost  https://review.opendev.org/75204221:21
TheJuliaiurygregory: prometheus exporter release?21:47
iurygregorypushing21:47
TheJulialooks like bifrost still needs to be released21:48
iurygregorywe are *waitting*21:48
TheJuliahuh?21:48
iurygregoryI'm trying to add the support for ipe on bifrost =X21:48
iurygregoryworks locally but CI seems unhappy21:48
TheJuliahmm21:48
iurygregorywe have 2.1 for IPE so 2.2 should be good right?21:52
rpiosoTheJulia: We're trying to land https://review.opendev.org/#/c/751006.21:52
patchbotpatch 751006 - ironic - Add GPU reporting to idrac-wsman inspect interface - 9 patch sets21:52
iurygregorycan be also 2.1.1 since the only addition was a fix for node_name...21:52
iurygregorywdyt?21:52
TheJuliarpioso:  the committed/published date of the community was the 28th...21:55
TheJuliaiurygregory: can you takea  quick look at rpioso's patch21:57
rpiosoTheJulia: Understood. I thought you made a last call :-)21:57
rpiosoiurygregory, TheJulia: I can quickly make any needed changes.21:57
iurygregorydell ci is testing this or you had some downstream tests rpioso ?21:59
iurygregoryjsut me or this would need a bump in requirements?22:00
TheJuliaiurygregory: w/r/t bifrost, I _think_ if you try to combine the conditional on one line it should work just fine. There is some funkiness at times around lists of whens and I suspect the bool casting is throwing it since it is really for humans overriding it with a string22:00
iurygregorysince you need a new version of the python-dracclient...22:01
iurygregoryTheJulia, I will give a try =)22:01
rpiosoiurygregory: It's a new feature, so the Dell 3d Party CI is not testing it. However, we have extensively integration tested it.22:02
TheJuliayeah, minimum needs to bumb22:02
iurygregoryI would +2 https://review.opendev.org/#/c/751006 but I think it needs to update the driver-requirements.txt22:02
patchbotpatch 751006 - ironic - Add GPU reporting to idrac-wsman inspect interface - 9 patch sets22:02
iurygregory=)22:02
rpiosoiurygregory, TheJulia: It's been bumped.22:03
TheJuliayeah, minimum needs to be bumped in driver-requirements.txt since there is no handling of attributeerror due to the method call22:03
TheJuliarpioso: not quite22:03
iurygregoryrpioso, I don't see driver-requirements.txt updated22:03
iurygregoryhttps://review.opendev.org/#/c/751006/22:03
patchbotpatch 751006 - ironic - Add GPU reporting to idrac-wsman inspect interface - 9 patch sets22:03
iurygregory=)22:04
TheJuliait is defined as >=3.1.0,<6.0.022:04
TheJuliabut anyone that hits that code with <5.0.0, i.e. 4.0, will have an exception22:04
rpiosoiurygregory, TheJulia: Then that lower bound has been broken for a long time. Our impression is that represents the version necessary for the minimum, required ironic support -- power and management.22:06
iurygregoryomg22:06
rpiosoI communicated with cdearborn earlier this week. He recalls receiving that guidance from dtantsur.22:06
TheJulianot quite, inspection is totally non-functional as it is not handled otherwise, so the minimum should be what is compatible and generally worsk22:06
rpiosoTheJulia: Agent inspection, no?22:07
TheJuliaso adding the code is going to blow it up if anyone calls inspect today or tomorrow22:07
TheJuliaat least, oob inspection22:07
rpiosoiurygregory, TheJulia: Should it be >=5.1.0,<6.0.0? If so, I'll make that change, along with any others.22:09
TheJuliarpioso: based on the proposed code, yes22:09
iurygregoryyeah22:09
* TheJulia takes a quick break and emails winnebago :(22:09
*** rcernin has joined #openstack-ironic22:12
rpiosoiurygregory, TheJulia: The owner of that change had a question about the release note. May we drop the issue subsection from it?22:12
* iurygregory looks22:12
TheJuliaI'd prefer it dropped and clarified in documentation22:13
TheJuliabut that doesn't need to happen now()22:13
TheJuliabut yeah22:13
iurygregoryyeah a note in doc/source/admin/drivers/idrac.rst would make more sense22:14
rpiosoWe need to further analyze the 13G observation. It may be caused by GPU configuration, iDRAC support or lack thereof, or smth else. How about we follow up the change with a bug fix and/or documentation update after we have more info?22:15
iurygregoryand docs we can backport without problems22:15
TheJuliarpioso: perfect!22:15
* rpioso modifies driver-requirements.txt and removes the issue from the release note22:15
TheJuliawithout clarity, I'd prefer it just be in docs once ther eis clarity22:15
TheJulia++22:15
iurygregoryyeah =)22:16
rpiosoiurygregory, TheJulia: <322:16
iurygregoryTheJulia, I'm pusing IPE 2.1.1 since we only have a bugfix ok?22:16
rpiosoiurygregory, TheJulia: Anything else?22:16
iurygregoryrpioso, not from my side =)22:16
TheJuliaiurygregory: sounds good22:16
TheJuliayeah, that is all from my pov22:17
openstackgerritRichard G. Pioso proposed openstack/ironic master: Add GPU reporting to idrac-wsman inspect interface  https://review.opendev.org/75100622:18
iurygregoryrpioso, I think you forgot git add ...22:20
iurygregoryreleasenote is the same and no req change =)22:20
rpiosoiurygregory: I rebased, because driver-requirements.txt change would conflict. Now making the changes.22:21
iurygregoryI see22:21
*** rloo has quit IRC22:22
*** k_mouza has joined #openstack-ironic22:27
openstackgerritRichard G. Pioso proposed openstack/ironic master: Add GPU reporting to idrac-wsman inspect interface  https://review.opendev.org/75100622:34
rpiosoiurygregory: ^^^22:34
iurygregoryI've put my +2 since I don't want to wait almost 3hrs for results hehe22:35
TheJuliaheh22:51
TheJuliayeah... I'm going to have to tag it later tonight22:51
TheJuliawhich I'm fine with22:51
TheJulia+2+A approved22:52
TheJuliaI'll check back on it in a couple hours22:52
*** tosky has quit IRC22:54
rpiosoiurygregory, TheJulia: Thank you so much!22:56
rpiosoiurygregory, TheJulia: I successfully ran the unit tests and built the release notes locally before pushing the new patch set.22:59
*** tkajinam has joined #openstack-ironic23:00
*** k_mouza has quit IRC23:00
TheJuliaawesome, thanks23:03
*** k_mouza has joined #openstack-ironic23:37
rpiosoTheJulia: :)23:43
*** k_mouza has quit IRC23:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!