Monday, 2021-12-06

opendevreviewSteve Baker proposed openstack/ironic master: Capture [pxe]loader_file_paths for distros  https://review.opendev.org/c/openstack/ironic/+/81539204:17
arne_wiebalckGood morning, Ironic!07:34
dtantsurmorning ironic07:38
dtantsurI'd really appreciate reviews on https://review.opendev.org/c/openstack/ironic/+/820017/ and https://review.opendev.org/c/openstack/ironic/+/819620 please. Ideally to fit into the upcoming release.07:47
opendevreviewArne Wiebalck proposed openstack/ironic-python-agent master: Burn-in: Add SMART self test to disk burn-in  https://review.opendev.org/c/openstack/ironic-python-agent/+/81950507:55
opendevreviewDmitry Tantsur proposed openstack/ironic master: Trivial: log current state when continuing cleaning  https://review.opendev.org/c/openstack/ironic/+/82047608:10
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Do not use RPC continue_node_deploy/clean in agent_base  https://review.opendev.org/c/openstack/ironic/+/75635408:11
* MahnoorAsghar Hello everyone! I am Mahnoor (she/her), an Outreachy intern, for the project Auto-Documenting REST API classes for Ironic. Excited to start working on the project!08:11
opendevreviewArne Wiebalck proposed openstack/ironic-python-agent master: Burn-in: Add SMART self test to disk burn-in  https://review.opendev.org/c/openstack/ironic-python-agent/+/81950508:12
arne_wiebalckstendulker: thanks for the review and ^^ :-)08:12
dtantsurHi MahnoorAsghar! You may want to sync with your mentors per email, none of them are currently online.08:12
* MahnoorAsghar Yes, I have synced with them :)08:13
stendulkerarne_wiebalck: :)08:14
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047908:34
opendevreviewVerification of a change to openstack/ironic master failed: Install isolinux on devstack  https://review.opendev.org/c/openstack/ironic/+/82039308:36
rpittaugood morning ironic! o/08:50
opendevreviewVerification of a change to openstack/ironic master failed: Install isolinux on devstack  https://review.opendev.org/c/openstack/ironic/+/82039308:53
rpittaudtantsur, TheJulia, I added the change for the meeting time here https://review.opendev.org/c/opendev/irc-meetings/+/82047708:56
dtantsurthx!08:57
dtantsurI suspect iury will need to ack that08:57
rpittauprobably, I added him to the review08:58
hgyHello, Ironic!08:59
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047909:06
dtantsurrpittau: as part of your stream-9 work, we should probably add a deprecation note that stream-8 will be best-effort supported starting with next cycle. WDYT?09:17
rpittaudtantsur: yes, sounds good09:18
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047909:26
rpittaudtantsur: I was starting to look at the bugfix releases, not sure we can justify one for inspector09:38
opendevreviewRiccardo Pittau proposed openstack/ironic-inspector master: [trivial] Fix Xena release notes versions  https://review.opendev.org/c/openstack/ironic-inspector/+/82048309:42
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047909:48
jandershey arne_wiebalck dtantsur rpittau hgy and Ironic o/09:55
rpittauhey janders :)09:55
arne_wiebalckhey hgy janders rpittau dtantsur o/09:57
rpittauhey arne_wiebalck :)10:00
dtantsuro/10:01
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Do not use RPC continue_node_deploy/clean in agent_base  https://review.opendev.org/c/openstack/ironic/+/75635410:02
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Prepare for bugfix release  https://review.opendev.org/c/openstack/bifrost/+/82048710:04
opendevreviewRiccardo Pittau proposed openstack/ironic-python-agent master: [trivial] Fix Xena release notes versions  https://review.opendev.org/c/openstack/ironic-python-agent/+/82048810:09
opendevreviewRiccardo Pittau proposed openstack/ironic-python-agent master: Prepare for bugfix release  https://review.opendev.org/c/openstack/ironic-python-agent/+/82048910:18
opendevreviewMerged openstack/bifrost stable/xena: Fix RedHat required defaults  https://review.opendev.org/c/openstack/bifrost/+/82034010:19
opendevreviewRiccardo Pittau proposed openstack/bifrost master: [trivial] Fix Xena and Wallaby release notes versions  https://review.opendev.org/c/openstack/bifrost/+/82051010:22
dtantsurarne_wiebalck: if you have a second today, this is quite trivial: https://review.opendev.org/c/openstack/bifrost/+/82017310:22
rpittauother 2 lightning-fast reviews for bifrost https://review.opendev.org/c/openstack/bifrost/+/787509 and https://review.opendev.org/c/openstack/bifrost/+/82036510:25
* rpittau needs coffee before reviewing ironic release notes10:26
arne_wiebalckrpittau: the 1st one is ready for review? (has a -1, and W-1)10:27
rpittauwhoops wrong one, I meant https://review.opendev.org/c/openstack/bifrost/+/81905710:31
* rpittau confirms he needs coffee10:31
opendevreviewAija Jauntēva proposed x/sushy-oem-idrac master: Migrate constants to enums  https://review.opendev.org/c/x/sushy-oem-idrac/+/81702810:35
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Use the combined Ironic service instead of API+conductor  https://review.opendev.org/c/openstack/bifrost/+/82039010:41
arne_wiebalckrpittau: heh10:44
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Prepare for bugfix release  https://review.opendev.org/c/openstack/bifrost/+/82048711:04
opendevreviewMerged openstack/bifrost master: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82017311:15
opendevreviewMerged openstack/bifrost master: Update tarballs base url  https://review.opendev.org/c/openstack/bifrost/+/82036511:20
opendevreviewDmitry Tantsur proposed openstack/bifrost stable/xena: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82049711:20
opendevreviewDmitry Tantsur proposed openstack/bifrost stable/xena: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82049711:20
dtantsurdamn, same branch :)11:20
rpittau:)11:21
opendevreviewDmitry Tantsur proposed openstack/bifrost stable/wallaby: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82049811:21
opendevreviewMerged openstack/bifrost master: Remove unused integration job  https://review.opendev.org/c/openstack/bifrost/+/81905711:37
opendevreviewDmitry Tantsur proposed openstack/ironic master: Avoid RPC notify_conductor_resume_{deploy,clean} in agent_base  https://review.opendev.org/c/openstack/ironic/+/75635411:42
opendevreviewRiccardo Pittau proposed openstack/ironic master: Prepare for bugfix release  https://review.opendev.org/c/openstack/ironic/+/82051511:46
opendevreviewDmitry Tantsur proposed openstack/ironic master: DNM remove force_remote_call for continue_node_*  https://review.opendev.org/c/openstack/ironic/+/82051611:51
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047911:52
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql  https://review.opendev.org/c/openstack/bifrost/+/82047911:53
jamesdentonGood morning, all. I recently upgraded from W->X, and am starting to see these errors with my existing images: "Agent heartbeat received for node f7945b37-...-1277d0a5777f without an agent token". I understand that tokens are now required, but even after trying one of the ramdisk images at https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/ still can't get it to work. Any ideas?12:20
opendevreviewMerged openstack/ironic master: Install isolinux on devstack  https://review.opendev.org/c/openstack/ironic/+/82039312:29
opendevreviewDmitry Tantsur proposed openstack/ironic master: All-in-one Ironic service with a local RPC bus  https://review.opendev.org/c/openstack/ironic/+/81962012:56
dtantsurjamesdenton: are you sure you're using Xena images?12:56
jamesdentoni'm pretty sure?  :D but i will double check here in a few12:57
jamesdentonipa-centos8-stable-xena.initramfs and the kernel should do the trick, right?12:59
dtantsuryeah12:59
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Add rpc_transport == none  https://review.opendev.org/c/openstack/ironic/+/82003613:03
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] use sqlite instead of mysql and rpc_transport none  https://review.opendev.org/c/openstack/bifrost/+/82047913:04
dtantsurjust to double check, rpittau, when is our meeting today?13:18
rpittaudtantsur: it should be at 16 UTC13:29
rpittauso one hour later 13:29
dtantsurthx13:38
TheJuliagood morning13:43
dtantsurmorning TheJulia 13:43
dtantsur(dunno about the good bit, I'm freezing)13:43
TheJuliaugh :(13:52
TheJuliaevil tilt in the plant's rotation13:52
opendevreviewDmitry Tantsur proposed openstack/ironic master: [WIP] Add rpc_transport == none  https://review.opendev.org/c/openstack/ironic/+/82003613:54
dtantsuron the bright side, https://review.opendev.org/c/openstack/ironic/+/819620 Works On Bifrost (tm) :)14:13
dtantsurwould really appreciate reviews (+ the parent patch)14:13
opendevreviewMerged openstack/bifrost stable/xena: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82049714:19
* TheJulia yawsn14:19
opendevreviewMerged openstack/bifrost stable/wallaby: Catch all exceptions from pbr when importing the root package  https://review.opendev.org/c/openstack/bifrost/+/82049814:39
opendevreviewMerged openstack/metalsmith stable/xena: Set binding:host_id on VIFs  https://review.opendev.org/c/openstack/metalsmith/+/81983114:50
jamesdentondtantsur no luck w/ that ramdisk, same issue. On the client side i am getting an error that corresponds to the server-side error - "Error sending heartbeat to http://10.20.0.11:6385: HeartbeatError: Error heartbeating to Agent API: Invalid Status Code: 400". Not sure which came first. Should there be any logs on the conductor side referencing a token request from IPA?14:59
dtantsurprobably rather in the API15:01
dtantsurcould it be that the old agent is still used anywhere?15:01
jamesdentonthat's where we see ERROR ironic.api.controllers.v1.ramdisk [req-03936361-d8dc-46be-87af-f4790078e69c - - - - -] Agent heartbeat received for node f7945b37-f43f-4b69-b987-1277d0a5777f without an agent token.15:02
dtantsuras there any logs before that in both the agent and API/conductor as to *why* the token wasn't generated or provided15:03
jamesdentonthe only reference to an agent token in the logs is when we see that message above.15:06
jamesdentonas for an old agent, this is the only node (with the new image and ramdisk) that's being deployed for testing.15:08
opendevreviewMerged openstack/metalsmith stable/wallaby: Set binding:host_id on VIFs  https://review.opendev.org/c/openstack/metalsmith/+/81983215:12
dtantsurjamesdenton: when the agent starts, it asks Ironic for the token (unless you use virtual media). There should be some traces of this conversation in the logs.15:14
jamesdentongood to know, thank you.15:22
TheJuliabraaaains15:27
opendevreviewDmitry Tantsur proposed openstack/ironic master: Add "none" RPC transport that disables the RPC bus  https://review.opendev.org/c/openstack/ironic/+/82003615:34
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Use "none" RPC by default, disable JSON RPC  https://review.opendev.org/c/openstack/bifrost/+/82047915:40
opendevreviewDmitry Tantsur proposed openstack/bifrost master: DNM expose_json_rpc  https://review.opendev.org/c/openstack/bifrost/+/82054315:41
* dtantsur is curious what the team will say when they see his list of proposed patches to review this week....15:44
* dtantsur drops a few bifrost patches for the sake of sanity15:46
TheJuliaheh15:46
* TheJulia adds https://review.opendev.org/c/openstack/ironic/+/820077 :)15:47
dtantsurcan I just add my stuff or should we discuss them on the meeting first15:48
TheJuliaI'd just add stuff15:50
TheJuliaThis is the end of the year after all15:51
dtantsuron the bright side, https://review.opendev.org/c/openstack/ironic/+/756354 is something we've wanted for years15:51
dtantsurremind me, who's chairing the meeting today?15:58
TheJulia#startmeeting ironic16:00
opendevmeetMeeting started Mon Dec  6 16:00:18 2021 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'ironic'16:00
TheJuliao/16:00
dtantsuro/16:00
TheJuliaHow does that work?16:00
TheJulialol16:00
dtantsur:)16:00
rlooo/16:00
ajyao/16:00
TheJuliaWelcome everyone to this week's Ironic meeting of Bare Metal irony!16:00
TheJuliaI'm TheJulia, and I'll be your Game Show host today!16:01
rpittauo/16:01
dtantsur\o/16:01
TheJuliaOur agenda this week is on the wiki, although I suspect it is out of date.... maybe?!?16:01
TheJulia#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting16:01
TheJuliaOnward~16:01
TheJuliaerr16:01
stendulkero/16:02
TheJulia#topic Announcements / Reminders16:02
dtantsurwell, the point about releasing *this week* is still important16:02
rpittauvery important16:02
TheJulia#info *we really* need to release this week.16:02
rpittauI started reviewing release notes, I wil ladd the patches to the prios16:02
TheJulia#info TC has semi-agreed that this release is a transition release from Python 3.6 for Centos8 to supporting Centos9 which would include Python 3.8 and 3.9 support.16:03
dtantsurI'd really like to get the single-process ironic there16:03
TheJuliaIn other words! we shouldn't expect 3.6 breakages to *really* appear at all this cycle.16:03
TheJuliaYour regular host, is on PTO this month, and I believe our last meeting of the year is next week.16:03
TheJuliaDoes anyone have anything else to add?16:04
TheJuliarpittau: much appreciated16:04
rpittauTheJulia: I have some patches to revert the py3.6 additional jobs https://review.opendev.org/q/topic:%22pure-yoga-tests%22+(status:open%20OR%20status:merged)16:04
TheJuliarpittau: tagged for the week's review?16:05
rpittauI will add them too, yeah16:05
TheJuliaOnward then!16:06
TheJulia#topic Review Action items from prior meeting16:06
TheJuliarpittau: well, I believe your action item was satisified, we are meeting an hour later today.16:07
rpittauit was indeed, thanks!16:07
TheJuliadtantsur: And I believe you also updated the whiteboard16:07
TheJuliaSo onward?16:07
dtantsurI did16:07
TheJuliaExcellent, then we can proceed!16:07
TheJulia#topic Review subteam status reports16:07
TheJulia#link https://etherpad.openstack.org/p/IronicWhiteBoard16:07
TheJuliaSo, our priorities for the cycle have not merged, but we have ?5? core reviewers who have voted on it, I think we can merge with lazy consensus at this point16:08
dtantsuryeah, and I've already populated the whiteboard based on it :)16:08
TheJuliaexcellent16:09
dtantsurbtw if anyone has any energy to figure out why DIB jobs are failing on bifrost, would be awesome16:09
dtantsurI'll get there.. one day. but maybe someone will get there earlier16:09
TheJuliai might be able to get there this week depending on interruptions16:10
dtantsurit was something about python vs python3 in DIB the last time I checked16:10
TheJuliaDoes anyone else want to review https://review.opendev.org/c/openstack/ironic-specs/+/815308/ before it is workflowed?16:11
TheJuliadtantsur: ugh16:11
TheJuliadtantsur: that is... not good16:11
rpittauI'll try to have a look as well during the week16:11
TheJuliaI think the whiteboard looks good, are we good to proceed onward?16:13
rpittaudtantsur: the biforst patch you mentioned for bifrost ARM CI is really for CentOS 9, do we want to split that ?16:13
dtantsurrpittau: no, sorry, I just derped because of lack of sleep and warmth :(16:14
dtantsuralthough, yeah, RHEL 9 support is a topic we need16:14
rpittauoh no worries!16:14
rpittaulet's add another topic, we're on that anyway16:14
dtantsurdone16:14
rpittauthanks!16:14
TheJuliaWe've already done some for that, but testing it internally will be much better overall16:15
TheJuliaOnward?16:15
dtantsur++16:15
rpittauyep16:16
TheJulia#topic Priorities for the coming week16:16
TheJulia#link https://review.opendev.org/q/hashtag:ironic-week-prio+status:open16:16
TheJuliaThere are a ton of items on here16:16
dtantsuryeah, I've recently added a few :)16:16
TheJuliaPerhaps we just all review after the meeting?16:16
rpittauheh my recent additions! :)16:16
dtantsurwell, rpittau's patches should be trivial to get merged, let's do it afterwards16:17
TheJulia++16:17
dtantsurmine are a bit more sophisticated, but I'm available for questions :)16:17
rpittauyes, very quick, just strings update for renos16:17
dtantsur(until I fall asleep for exhaustion)16:17
dtantsur"Use only Yoga tests" - should it wait for the template change first?16:18
* TheJulia adds https://review.opendev.org/c/openstack/ironic/+/820077 to the list16:18
TheJuliaThat is part of what ate my last week16:18
rpittauoh shoot I thought it was merged already16:18
rpittauI'll remove the tag16:19
arne_wiebalckthere are also some simple sushy patches from me on the list (well, code-wise simple) ... if anyone wants to take  a look16:19
TheJuliaarne_wiebalck: are they tagged with ironic-week-prio?16:19
arne_wiebalckyes16:19
TheJuliaack16:19
TheJuliaOkay, sounds good16:20
TheJuliaAnything else? Are we good to proceed?16:20
dtantsur++16:21
TheJuliadtantsur: I guess we don't need to discuss py36 anymore?16:22
dtantsurnope16:22
TheJuliaarne_wiebalck: anything to discuss for baremetal sig?16:22
TheJuliaFor this week, at least16:22
TheJulia#topic Baremetal SIG16:22
TheJuliadtantsur: aren't you doing a presentation ?tomorrow?16:22
* TheJulia need calendar invites or something16:22
arne_wiebalckNext week we have dtantsur on Metal3 (with demo!) :-D16:22
TheJuliaahh, next week!16:23
arne_wiebalckNext week I think16:23
TheJulia#info Next week dtantsur will present on Metal316:23
TheJuliaAnything else?16:23
dtantsuroh damn :D16:23
TheJuliadtantsur: heh16:23
arne_wiebalckheh16:23
arne_wiebalckI will send the mails and tweets in due time.16:23
arne_wiebalckThat's it from me for today.16:24
TheJuliaOkay, onward!16:24
TheJulia#topic RFE Review16:24
TheJuliaajya: I believe you have an RFE for us today?16:24
ajyano.. 16:24
ajyathat's from last time not removed16:24
dtantsurI think it also was last time16:24
TheJuliaokay then16:24
TheJulia#topic Open Discussion!16:24
TheJuliaAnything to discuss? Recipes to share? Worlds to conquer?16:25
TheJuliaNew books to read?16:25
dtantsurmmm, books16:25
TheJuliafwiw, Termination Shock is good, just... also hits close to home as it was clearly written during the beginning of the pandemic.16:25
dtantsurI haven't read a single one in a while :(16:25
dtantsurhave unfinished https://www.goodreads.com/book/show/45154547-a-desolation-called-peace on by phone16:26
rpittaujust started the 9th (and last) book in The Expanse saga, seems pretty good so far16:26
arne_wiebalckdtantsur: oh, just started the first one :)16:27
dtantsurnice :)16:27
TheJuliarpittau: umm... I thought there was a 10th book16:27
TheJuliaoh16:28
TheJulianow not shipping until 202216:28
TheJuliawell that is unfortunate16:28
rpittauTheJulia: seems like the 9th will be the last one, the 10th one is a collection16:28
TheJuliahmm16:28
TheJuliaInteresting16:28
TheJuliaI need to finish termination shock first before I get back on the expanse train16:28
rpittau*collection of short stories16:28
TheJuliaWell, is there anything else for us to discuss this week?16:29
dtantsurmidcycle?16:29
* dtantsur hides16:29
TheJuliaI suspect that may be best planned for sometime in January, but we should get our PTL engaged to help plan it16:30
TheJuliaMaybe we should start an ideas etherpad?16:30
opendevreviewMerged openstack/bifrost master: [trivial] Fix Xena and Wallaby release notes versions  https://review.opendev.org/c/openstack/bifrost/+/82051016:31
opendevreviewMerged openstack/bifrost master: Prepare for bugfix release  https://review.opendev.org/c/openstack/bifrost/+/82048716:31
dtantsurmaybe? I'm mostly planting the thought in the heads :)16:32
TheJuliaokay16:32
TheJulia#topic Who is going to run next week's meeting?16:32
dtantsurof course it won't be the same as a year ago *sobbing*16:32
TheJuliaAny volunteers?16:32
rpittauI can run the next one :)16:32
dtantsurwas about to say we can appoint rpittau :)16:32
rpittau:D16:32
TheJuliadtantsur: 2... years :(16:32
* TheJulia cries16:33
dtantsurdon't say that!16:33
* dtantsur hugs TheJulia16:33
TheJuliaOkay, rpittau has been appointed next week's victim for meeting running16:33
TheJuliaAnd with that, Thanks everyone!16:33
rpittauyay (?)16:33
TheJuliaNow, everyone! Go review!16:33
TheJuliachop chop, the code won't merge itself!16:33
TheJulia:)16:33
* TheJulia is in rare form today16:33
rloo(wonders if that is cuz TheJulia is running today's meeting...)16:34
* dtantsur curious about a rare form TheJulia16:34
TheJuliadtantsur: slightly more crazy than normal16:34
TheJuliaits all okay, I'm double caffinated today!16:34
dtantsur\o/16:34
TheJuliaThanks everyone!16:34
TheJuliaHave a wonderful week!16:34
TheJulia#endmeeting16:34
opendevmeetMeeting ended Mon Dec  6 16:34:53 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:34
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-12-06-16.00.html16:34
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-12-06-16.00.txt16:34
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2021/ironic.2021-12-06-16.00.log.html16:34
TheJuliaajya: w/r/t https://review.opendev.org/c/openstack/sushy/+/820076, I'm not disagreeing we can't consider fall back again, just there is a lot of risk and I'm not sure a ton of value to hide it from the library consumer16:39
TheJuliaor is that fall back forward16:39
rpittaudtantsur: the issue with DIB does not look like our fault, at least at a glance, I proposed a patch https://review.opendev.org/c/openstack/diskimage-builder/+/82056316:41
arne_wiebalckdtantsur: "a BMC implementation using headers.etags[0]" :-D16:41
arne_wiebalckdtantsur: not unlikely!16:41
dtantsurarne_wiebalck: I'd not be too surprised :)16:41
TheJuliaarne_wiebalck: I was just looking at that thinking the same thing16:41
* arne_wiebalck shudders16:41
dtantsurif somebody will review my combined ironic patches, https://review.opendev.org/c/openstack/bifrost/+/820479/ show-cases it (see the "ironic" service in the logs)16:44
ajyaTheJulia:  what is fallback in this context? The way I see it, there are 2 cases 1) legit AccessError 2) AccessError accessing SessionService before auth. The patch fixes 1), but it also breaks 2) Redfish Sessions for at least some BMCs.16:45
* dtantsur will go get some rest o/16:46
jamesdentondtantsur i think i'm about to feel real stupid. I had been changing the image properties but there were hardcoded deploy_kernel and deploy_ramdisk values on the baremetal nodes themselves. Curious - are the latter optional if they're set on the images?16:46
opendevreviewJulia Kreger proposed openstack/ironic master: Trivial: minor follow-up to redfish fix  https://review.opendev.org/c/openstack/ironic/+/82056416:46
dtantsurjamesdenton: I never used properties on images, tbh.. I'm surprised that it works.16:47
dtantsurI mean, it works for instance images, but for deploy images?16:47
jamesdentonprobably some misinforation/instructions on my part. Not sure it works, either16:47
TheJuliaarne_wiebalck: so should your etag fixes be backported?16:47
dtantsurdeploy images are used in situations where there are not instance images16:47
dtantsur(cleaning, inspection)16:47
jamesdentonahh right16:47
TheJuliaajya: So, in the identified case, the X-Auth-Token removal fix was not present which likely contributed to the failure, so in a sense it was case `. Case 2, I'm really not sure about and at that point it seems more like we've got bad input data. At least, I would think16:50
arne_wiebalckTheJulia: Maybe. I have already seen hardware which does not like etags to be passed back. So, I have an additional local patch to retry without when the etags have not changed.16:51
arne_wiebalckTheJulia: I am little worried in case we backport this, we will have others run into this16:52
TheJuliaajya: https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.14.1.pdf 15.3.4.1 seems vauge but also suggests finding details before authenticating...16:52
arne_wiebalckTheJulia: Not sure how to properly handle this.16:52
TheJuliahmm... very carefully then16:52
arne_wiebalckTheJulia: The logic I have downstream now is: try with etag, if that fails, check if the etag changed, if not retry without16:53
TheJuliaso lets see how it pans out, if we need to backport stuffs later I guess we can16:53
arne_wiebalckTheJulia: I think this is a server side bug, though: you cannot reject the etag you handed out.16:53
arne_wiebalckTheJulia: I guess I should get in touch with my vendor to see what they say.16:55
TheJulia++16:56
TheJuliaarne_wiebalck: feel free to loop me in if we need backports based upon that16:56
arne_wiebalckTheJulia: thanks!17:05
opendevreviewMerged openstack/ironic-specs master: Yoga Themes  https://review.opendev.org/c/openstack/ironic-specs/+/81530817:09
dking1So, I was looking at that issue I noticed with Inspector last week, and it certainly seems to me to be a memory leak. It seems to be related to the requests. We do pound the service rather hard, with 200 servers having all their data polled each minute, and it seems that the memory usage of Inspector is growing at a rather consistent 1M per five or six minutes. As the memory is expanded, if I dump the memory and thing pass it t17:10
dking1o `strings`, I mostly get "req-*" IDs, and a bunch of "fpfd", but that's probably not an actual text string.17:10
rpittaubye everyone! o/17:10
dking1rpittau: Have a good night!17:11
TheJuliadking1: hmmmm17:11
*** dking1 is now known as dking17:11
dkingI'm about to put in a story, but this is related to the issue I mentioned last week about Inpsector using up RAM.17:12
*** dking is now known as Guest779317:12
TheJuliayeah, sounds like it17:12
Guest7793It got to about 47G17:12
TheJuliathat at least gives us a solid place to start17:12
* TheJulia mv Guest7793 dking17:13
*** Guest7793 is now known as dking17:13
* TheJulia gets out superglue and glues dking's nick to the irc channel17:13
* TheJulia is totally loosing her mind17:13
dkingThanks. I can't seem to get this nick thing straight these days. lol17:13
TheJulialol17:13
dkingStory: https://storyboard.openstack.org/#!/story/200972717:17
dkingIs there many further debugging information that I should give? I didn't know whether it would be worth it for me to try to setup a docker container somewhere and try to create replication with fake or non-sensitive data in it.17:20
dkingMy guess is that somewhere there's going to be a variable that Python isn't cleaning up properly. 17:21
TheJuliadking: I can look in like 5 minutes17:23
*** sshnaidm is now known as sshnaidm|afk17:25
dkingThank you very much.17:25
TheJuliadking: any configuration details. How is it being launched, if you can suppply what the configuration is, that would also be helpful17:26
TheJuliasanitized of sensitive details of course17:26
TheJuliaJust so we have a reference when digging through the code17:27
dkingOkay. I added it in a comment: https://paste.opendev.org/show/811490/17:32
opendevreviewVerification of a change to openstack/sushy master failed: Use Etag in precondition header when setting boot options  https://review.opendev.org/c/openstack/sushy/+/81811417:34
opendevreviewMerged openstack/sushy master: Handle weak Etags  https://review.opendev.org/c/openstack/sushy/+/81811017:41
TheJuliabegin the merge party!17:44
arne_wiebalck\o/18:04
opendevreviewVerification of a change to openstack/ironic master failed: Use test_with_retry to get the tap device name  https://review.opendev.org/c/openstack/ironic/+/81870318:06
arne_wiebalckbye everyone, see you tomorrow o/18:13
* TheJulia gets out the suspicious corgi look18:13
TheJuliao/ arne_wiebalck 18:13
* TheJulia senses ci problem18:13
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: An idea for rbac positive/negative testing  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/81916519:07
opendevreviewMerged openstack/ironic master: Trivial: log current state when continuing cleaning  https://review.opendev.org/c/openstack/ironic/+/82047619:33
opendevreviewMerged openstack/ironic-inspector master: [trivial] Fix Xena release notes versions  https://review.opendev.org/c/openstack/ironic-inspector/+/82048319:34
opendevreviewMerged openstack/ironic master: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82007719:34
opendevreviewMerged openstack/ironic-python-agent master: [trivial] Fix Xena release notes versions  https://review.opendev.org/c/openstack/ironic-python-agent/+/82048819:34
opendevreviewMerged openstack/ironic master: Prepare for bugfix release  https://review.opendev.org/c/openstack/ironic/+/82051519:34
stevebaker[m]morning19:35
TheJuliaGood morning19:36
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: An idea for rbac positive/negative testing  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/81916519:45
opendevreviewJulia Kreger proposed openstack/ironic stable/xena: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82058819:54
opendevreviewVerification of a change to openstack/ironic master failed: Use test_with_retry to get the tap device name  https://review.opendev.org/c/openstack/ironic/+/81870320:00
opendevreviewJulia Kreger proposed openstack/ironic bugfix/18.1: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82058920:09
opendevreviewJulia Kreger proposed openstack/ironic bugfix/18.1: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82058921:01
opendevreviewJulia Kreger proposed openstack/ironic stable/xena: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82058821:06
opendevreviewJulia Kreger proposed openstack/ironic stable/wallaby: Remove redfish cache entry upon errors  https://review.opendev.org/c/openstack/ironic/+/82059721:48
TheJuliastevebaker[m]: if you have some time at some point, take a glance at the py3 failures on ^^^21:48
TheJuliaor, well, one failure which is making zero sense21:48
opendevreviewJulia Kreger proposed openstack/sushy stable/xena: Raise an AccessError with SessionService init  https://review.opendev.org/c/openstack/sushy/+/82059821:51
opendevreviewJulia Kreger proposed openstack/sushy stable/wallaby: Raise an AccessError with SessionService init  https://review.opendev.org/c/openstack/sushy/+/82059921:53
opendevreviewJulia Kreger proposed openstack/sushy stable/victoria: Raise an AccessError with SessionService init  https://review.opendev.org/c/openstack/sushy/+/82060021:56
opendevreviewJulia Kreger proposed openstack/sushy stable/ussuri: Raise an AccessError with SessionService init  https://review.opendev.org/c/openstack/sushy/+/82060221:57
opendevreviewJulia Kreger proposed openstack/sushy stable/train: Raise an AccessError with SessionService init  https://review.opendev.org/c/openstack/sushy/+/82060322:02
opendevreviewMerged openstack/ironic-tempest-plugin master: Move API base out of admin test location  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/81916422:58
opendevreviewSteve Baker proposed openstack/bifrost master: Don't create ironic_tftp_master_path directory, change default  https://review.opendev.org/c/openstack/bifrost/+/82031123:42
opendevreviewSteve Baker proposed openstack/bifrost master: WIP support grub network boot  https://review.opendev.org/c/openstack/bifrost/+/80722023:42
stevebaker[m]TheJulia: ok, I think this will fix the selinux denial https://review.opendev.org/c/openstack/bifrost/+/82031123:43
jandersgood morning Ironic o/23:50

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!