Tuesday, 2022-12-06

opendevreviewVanou Ishii proposed openstack/ironic master: Handle IPMI incompatibility in iRMC S6 2.x  https://review.opendev.org/c/openstack/ironic/+/86507500:59
arne_wiebalckGood morning, Ironic!07:26
kubajjGood morning Ironic o/08:35
rpittaugood morning ironic! 10:02
*** jpodivin_ is now known as jpodivin10:14
kubajjdtantsur: I am trying to understand what you said yesterday. Is there any point in checking what was stored in Swift if I mock the return value, or did you mean mocking something else?10:22
opendevreviewJakub Jelinek proposed openstack/ironic master: Enable alternative storage for inventory  https://review.opendev.org/c/openstack/ironic/+/86605610:35
iurygregorygood morning Ironic11:38
kubajjgood morning iurygregory 11:50
iurygregoryo/11:51
opendevreviewJakub Jelinek proposed openstack/ironic master: Enable alternative storage for inventory  https://review.opendev.org/c/openstack/ironic/+/86605613:04
dtantsurkubajj: you'll need to mock both storing and retrieving. You can check what was stored by checking the arguments the mock was called with (see assert_called_once_with and others)13:18
kubajjdtantsur: for retrieving, do you mean some function for retrieving? I am already testing the storing with the assert in the change above13:21
dtantsurlemme see the patch first13:21
dtantsurkubajj: left some reviews, but I think it's really good overall13:35
dtantsurno issues with the way tests are written13:35
kubajjdtantsur: thanks13:36
dtantsurkubajj: there is one aspect that we'll need to cover still. unlike data in the database, data in swift is not deleted on node deletion.13:36
dtantsur(it may be another patch if this feels to heavy already)13:36
kubajjdtantsur: could we do that in another one? I already started looking into the api after the current one. But I am happy to look into it. I have plenty of time now as I am working just on my project for next 2 weeks or so13:39
dtantsuryes, absolutely!13:51
dtantsur(and I'm very glad that we have you13:51
dtantsurI'll be here until ~ x-mas, should be able to review things and help13:51
kubajjdtantsur: thanks, I hope to implement as much as possible before then13:53
TheJuliagood morning14:17
*** jpodivin_ is now known as jpodivin14:17
dtantsurmorning TheJulia 14:22
kubajjmorning TheJulia 14:27
dtantsurajya: hi, could you find a minute today for https://review.opendev.org/c/openstack/sushy/+/866612 please?15:01
ajyadtantsur: ok, looking15:01
TheJuliadtantsur: w/r/t inspector db stuffs, will I be more likely to get the patches moved forward if I just squash them?15:09
dtantsurTheJulia: I'd prefer that, to be honest. Otherwise it feels like I need to review 2 things: one somewhat incorrect and one correct :)15:09
TheJuliawell, one correct for what was there and what is there with a working autocommit... the other... autocommit goes away15:12
TheJuliaOkay, I'll try to get that done in the next day or so15:12
TheJuliaI lost a note I made yesterday so need to find that for the patch I need to backport first15:12
opendevreviewJakub Jelinek proposed openstack/ironic master: Enable alternative storage for inventory  https://review.opendev.org/c/openstack/ironic/+/86605615:20
kubajjdtantsur: that should be your comments implemented ^15:22
TheJuliastevebaker[m]: you beat me to the backports15:22
dtantsurkubajj: thanks, will check asap15:35
rpittauTheJulia: are you already looking at the issues with python 3.11 ?15:53
opendevreviewRiccardo Pittau proposed openstack/ironic bugfix/21.0: Align iRMC driver with Ironic's default boot_mode  https://review.opendev.org/c/openstack/ironic/+/86665615:56
Kirill_I see some errors with freshly IDRAC 6.00.30.00 and delete_configuration step16:00
Kirill_Is it known behaviour or bug should be created?16:00
Kirill_2022-12-06 13:38:25.706 7 INFO ironic.conductor.cleaning [req-a099f2cf-888d-4750-9ee7-29ce1c181c0e - - - - -] Executing {'step': 'delete_configuration', 'interface': 'raid', 'abortable': False, 'priority': 0} on node 734732c4-d0a7-4a03-882b-d616:00
Kirill_82d5268f4716:00
Kirill_2022-12-06 13:38:31.778 7 INFO ironic.drivers.modules.drac.utils [req-a099f2cf-888d-4750-9ee7-29ce1c181c0e - - - - -] Completed: get real-time ready status with system 4c4c4544-004a-4610-8053-c7c04f584d33 manager 334d584f-c0c7-5380-4610-004a4c16:00
ajyaKirill_: RAID?16:00
Kirill_4c4544 for node 734732c4-d0a7-4a03-882b-d682d5268f4716:00
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils [req-a099f2cf-888d-4750-9ee7-29ce1c181c0e - - - - -] Node 734732c4-d0a7-4a03-882b-d682d5268f47 failed step {'step': 'delete_configuration', 'interface': 'raid', 'abortable': False, 'priori16:00
Kirill_ty': 0}: HTTP DELETE https://dt-sl-pcb107-rmi.be.gc.onl/redfish/v1/Systems/System.Embedded.1/Storage/RAID.SL.3-1/Volumes/Disk.Virtual.0:RAID.SL.3-1 returned code 501. Base.1.12.GeneralError: Unsupported parameter name @Redfish.OperationApplyTi16:00
Kirill_me. Extended information: [{'Message': 'Unsupported parameter name @Redfish.OperationApplyTime.', 'MessageArgs': ['@Redfish.OperationApplyTime'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.7.SYS029', 'RelatedProperties': ['#/@Redfish.16:00
Kirill_OperationApplyTime'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Remove the unsupported parameter and retry the operation.', 'Severity': 'Warning'}]: sushy.exceptions.ServerSideError: HTTP DELETE https://dt-sl-pcb107-rmi.be.gc.onl/redf16:00
Kirill_ish/v1/Systems/System.Embedded.1/Storage/RAID.SL.3-1/Volumes/Disk.Virtual.0:RAID.SL.3-1 returned code 501. Base.1.12.GeneralError: Unsupported parameter name @Redfish.OperationApplyTime. Extended information: [{'Message': 'Unsupported paramete16:00
Kirill_r name @Redfish.OperationApplyTime.', 'MessageArgs': ['@Redfish.OperationApplyTime'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.7.SYS029', 'RelatedProperties': ['#/@Redfish.OperationApplyTime'], 'RelatedProperties@odata.count': 1, 'R16:00
Kirill_esolution': 'Remove the unsupported parameter and retry the operation.', 'Severity': 'Warning'}]16:00
ajyaKirill_: yes, it's a known issue :( and there is a patch - https://review.opendev.org/c/openstack/sushy/+/864845 16:00
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils Traceback (most recent call last):16:00
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/conductor/cleaning.py", line 170, in do_next_clean_step16:00
ajyaCores, please review 16:00
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     result = interface.execute_clean_step(task, step)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/base.py", line 337, in execute_clean_step16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     return self._execute_step(task, step)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/base.py", line 299, in _execute_step16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     return getattr(self, step['step'])(task)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/drac/raid.py", line 1289, in delete_configuration16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     return super(DracRedfishRAID, self).delete_configuration(task)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/redfish/raid.py", line 863, in delete_configuration16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     task)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ironic/drivers/modules/redfish/raid.py", line 1177, in _submit_delete_configuration16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     response = volume.delete(apply_time=apply_time)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/sushy/resources/system/storage/volume.py", line 154, in delete16:01
JayFKirill_: please use a pastebin and do not spam chat16:01
JayFholy cow16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     r = self._delete(payload, apply_time, timeout)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/sushy/resources/system/storage/volume.py", line 140, in _delete16:01
JayFfungi: do you have access to mute Kirill_ real quick?16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     timeout=timeout)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/sushy/connector.py", line 412, in delete16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     **extra_session_req_kwargs)16:01
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/sushy/connector.py", line 148, in _op16:02
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     exceptions.raise_for_response(method, url, response)16:02
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/sushy/exceptions.py", line 170, in raise_for_response16:02
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils     raise ServerSideError(method, url, response)16:02
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils sushy.exceptions.ServerSideError: HTTP DELETE https://dt-sl-pcb107-rmi.be.gc.onl/redfish/v1/Systems/System.Embedded.1/Storage/RAID.SL.3-1/Volumes/Disk.Virtual.0:RAID.SL.3-1 returned code 516:02
Kirill_01. Base.1.12.GeneralError: Unsupported parameter name @Redfish.OperationApplyTime. Extended information: [{'Message': 'Unsupported parameter name @Redfish.OperationApplyTime.', 'MessageArgs': ['@Redfish.OperationApplyTime'], 'MessageArgs@odat16:02
Kirill_a.count': 1, 'MessageId': 'IDRAC.2.7.SYS029', 'RelatedProperties': ['#/@Redfish.OperationApplyTime'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Remove the unsupported parameter and retry the operation.', 'Severity': 'Warning'}]16:02
Kirill_2022-12-06 13:38:35.776 7 ERROR ironic.conductor.utils16:02
Kirill_2022-12-06 13:38:35.841 7 ERROR ironic.conductor.task_manager [req-a099f2cf-888d-4750-9ee7-29ce1c181c0e - - - - -] Node 734732c4-d0a7-4a03-882b-d682d5268f47 moved to provision state "clean failed" from state "cleaning"; target provision state is "manageable": sushy.exceptions.ServerSideError: HTTP DELETE https://dt-sl-pcb107-rmi.be.gc.onl/redfish/v1/Systems/System.Embedded.1/Storage/RAID.SL.3-1/Volumes/Disk.Virtual.0:RAID.SL.3-1 returned code 501. 16:02
Kirill_Base.1.12.GeneralError: Unsupported parameter name @Redfish.OperationApplyTime. Extended information: [{'Message': 'Unsupported parameter name @Redfish.OperationApplyTime.', 'MessageArgs': ['@Redfish.OperationApplyTime'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.7.SYS029', 'RelatedProperties': ['#/@Redfish.OperationApplyTime'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Remove the unsupported parameter and retry the operation.', 16:02
Kirill_'Severity': 'Warning'}]16:02
Kirill_got it,big thanks)16:02
Kirill_one more question - how can i find this patch? before i tried storyboard and in code. but didn't find any links to this patch16:03
ajyaKirill_: storyboard hasn't been created, probably should. Can always check open patches16:03
ajyaJayF, was there a plan to move back to launchpad or I'm missing something?16:04
Kirill_++, thanks16:04
JayFwe have plans to do it generally16:04
JayFno work has been done to get there16:04
ajyaand to repeat in case it got lost in the floods - can cores review the patch for this issue - https://review.opendev.org/c/openstack/sushy/+/864845 It's already in prio list. Thanks.16:05
opendevreviewMerged openstack/sushy master: Handle a different error code for missing TransferProtocolType  https://review.opendev.org/c/openstack/sushy/+/86661216:05
ajyaok, thanks, JayF 16:05
opendevreviewRiccardo Pittau proposed openstack/ironic bugfix/20.2: Align iRMC driver with Ironic's default boot_mode  https://review.opendev.org/c/openstack/ironic/+/86678016:08
TheJuliarpittau: I am not, I've got backports, issues, and end of year stuff to deal with for the foundation16:08
rpittauTheJulia: ok, I'll have a look then, I should have the cycles this week16:08
TheJuliathanks16:08
fungiJayF: sorry, am presenting on a call, but get up with me later about setting you up with chanop access so you can do it16:14
dtantsurwas about to say that the PTL should have these rights16:16
TheJuliachange the topic?16:17
TheJuliaYeah, the bot access has always been disjointed and weird. I think I last changed the topic via the bot back in... 2016/201716:18
opendevreviewRiccardo Pittau proposed openstack/sushy master: Handle proper code_status in unit test  https://review.opendev.org/c/openstack/sushy/+/86678216:23
opendevreviewRiccardo Pittau proposed openstack/sushy master: Handle proper code_status in unit test  https://review.opendev.org/c/openstack/sushy/+/86678216:24
JayFTheJulia: got a sec to sanity check a couple of shards things?516:25
JayFGET /v1/shards would return -> {'shards': [{'shard': 'shard1', 'count': 1}]}16:25
JayFgiven we aren't doing pagination for this, or get-one (e.g. /v1/shards/shard1 won't return anything), I don't think we need a links section16:25
JayFso I literally think the return is that simple16:25
TheJuliaI think it is literally that simple16:26
JayFLike, my controllers.v1.shard.Shards.get_all is {'shards': dbapi.get_shards()} 16:27
TheJuliasounds about right16:28
TheJuliaheh16:28
JayFsometimes when things look too simple it makes me think I screwd up16:28
opendevreviewJay Faulkner proposed openstack/ironic master: API support for CRUD node.shard  https://review.opendev.org/c/openstack/ironic/+/86623516:38
opendevreviewRiccardo Pittau proposed openstack/sushy master: Handle proper code_status in unit test  https://review.opendev.org/c/openstack/sushy/+/86678216:54
opendevreviewRiccardo Pittau proposed openstack/sushy master: Handle proper code_status in unit test  https://review.opendev.org/c/openstack/sushy/+/86678216:55
opendevreviewJay Faulkner proposed openstack/ironic master: API support for CRUD node.shard  https://review.opendev.org/c/openstack/ironic/+/86623516:57
JayFhttps://review.opendev.org/c/openstack/ironic/+/866235/5/ironic/tests/unit/api/controllers/v1/test_shard.py#64 16:58
JayFshould that be returning 404? 16:58
TheJuliaI *think* you need to use https://review.opendev.org/c/openstack/ironic/+/866235/5/ironic/api/controllers/v1/shard.py#67 with get_one instead for that to happen17:00
TheJuliasince get_all is a blanket (everything under this not otherwise routed)17:00
TheJuliabut I might bewrong17:00
kubajjI have a question about the ironic API. There is the controller for most of the stuff (NodeVIF, NodeHistory, etc). Is this the thing that handles the commands?17:00
JayFso basically, implement get_one and force it to 40417:00
JayFgot it17:01
TheJuliano17:01
TheJuliaI mean just only have a get_one method should just do it17:01
TheJuliasince get_one is a specific routing17:01
TheJuliaget_all is a everything17:01
JayFI thought get_all was a list? 17:01
TheJuliaso the call won't know where to go and 40417:01
JayFand get_one was for getting /v1/shard/lol17:01
JayFlooks like I need to go read some pecan docs, eh17:01
TheJuliayou could just also implement get_one to return 40417:02
TheJuliaso kubajj, the whole thing with JayF is why controller are used17:02
TheJuliayou route buckets to controllers in the routing17:02
TheJuliaand each controller is a glass with one or more methods17:02
TheJuliaand those methods are what does the thing17:02
TheJuliaJayF: pecan docs ++17:03
JayFthose docs indicate that get_all is for GET /resource/17:04
JayFget() is for GET /resource + GET /resource/specific_one17:04
JayFso I think I do just need to implement get_one and 404 it17:04
JayFhttps://pecan.readthedocs.io/en/latest/rest.html#url-mapping17:04
TheJuliasounds good17:18
rpittaugood night! o/17:25
JayFarne_wiebalck: please take a look at that shards spec; see if your concerns are addressed17:26
JayFarne_wiebalck: we're to the point where implementation is fully prototyped; almost done. We need to get the spec, and the supporting code, landed so the nova side can start17:27
opendevreviewMerged openstack/sushy master: Retry on iDRAC SYS518 errors for all requests  https://review.opendev.org/c/openstack/sushy/+/86491118:32
jrosserwhere do i start debugging `ironic.conductor.utils [-] Asynchronous exception for node 31311f0c-5428-4c77-97b1-b07d94824850: Node failed to start the first cleaning step: not all arguments converted during string formatting`18:52
TheJuliadeja vu18:57
TheJuliajrosser: manual cleaning?18:57
jrosserwell, this is when i first `openstack baremetal node provide <node>`18:58
jrosserand it then starts cleaning at that point18:58
TheJuliajrosser: any patches locally?18:59
jrossera little more log here https://paste.opendev.org/show/bm7bY1etR08VkHu86mOX/18:59
jrosserno, though i've had to build my own IPA image as this is aarch6419:00
jrosserimage/kernel/initramfs19:00
jrosserit's on yoga19:01
TheJuliaIPA is on yoga?19:02
jrosseri think so, this is how i made it https://paste.opendev.org/show/bf0XgtjoAzd4D5pTePXD/19:03
jrosseri had a terrible wild goose chase to build that, finding an OS i could boot on an accessible arm system that didnt result in failing to build IPA19:05
jrosserso thats actually rocky8 to build IPA19:05
TheJuliahmm19:08
TheJuliaso, okay19:08
TheJuliawhat version gets stored in driver_internal_info/agent_version ?19:08
jrosser 'agent_version': '8.5.2.dev1'19:10
TheJuliainteresting19:11
TheJuliajrosser: is there a mellenox card in that machine?19:13
jrosserthere are two, yes19:13
jrosserno three19:13
TheJuliaheh19:13
TheJuliaI wonder if there is an issue with the hardware manager...19:14
JayF/o\19:14
jrosserit's got an onboard cx4, and a PCIe and an OCP cx619:14
TheJuliahmm, looking closer, maybe not19:14
TheJuliathis is weird19:15
TheJuliaoh, apparently forum cfp is also open19:27
opendevreviewMerged openstack/ironic-inspector bugfix/10.9: Fix compatibility with oslo.db 12.1.0  https://review.opendev.org/c/openstack/ironic-inspector/+/85650619:38
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 Support  https://review.opendev.org/c/openstack/ironic-inspector/+/86073119:44
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 Support  https://review.opendev.org/c/openstack/ironic-inspector/+/86073119:45
jrosserTheJulia: i took a hunch and set debug=False in ironic.conf, restarted conductor and to my surprise, cleaning has now succeeded19:49
jrosserthat hunch based on nothing more than this is the last thing in the log before the async exception https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/agent_base.py#L84419:51
jrosserit's kind of late here now but tomorrow i can turn debug back on and see if it fails again19:52
TheJuliajrosser: oh, that is why it seems like deja vu for me19:53
TheJuliajrosser: thanks!19:53
opendevreviewJay Faulkner proposed openstack/ironic master: API support for CRUD node.shard  https://review.opendev.org/c/openstack/ironic/+/86623523:02
JayF^^ shards API implementation is complete enough for code review now; shards spec also needs review23:13
opendevreviewJay Faulkner proposed openstack/ironic master: API support for CRUD node.shard  https://review.opendev.org/c/openstack/ironic/+/86623523:32

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!