Thursday, 2021-09-23

*** pmannidi is now known as pmannidi|AFK02:20
*** pmannidi|AFK is now known as pmannidi05:15
iurygregorygood morning Ironic o/06:06
jandershey iurygregory o/06:07
iurygregoryhey janders o/06:07
arne_wiebalckGood morning janders iurygregory and Ironic!06:07
iurygregoryarne_wiebalck, o/06:08
jandershey arne_wiebalck o/06:16
opendevreviewAija Jauntēva proposed openstack/ironic stable/xena: Update iDRAC doc for idrac-redfish RAID  https://review.opendev.org/c/openstack/ironic/+/81048506:35
opendevreviewAija Jauntēva proposed openstack/ironic stable/wallaby: Update iDRAC doc for idrac-redfish RAID  https://review.opendev.org/c/openstack/ironic/+/81048606:35
opendevreviewAija Jauntēva proposed openstack/ironic stable/wallaby: Fix idrac-wsman having Completed with Errors jobs  https://review.opendev.org/c/openstack/ironic/+/81048706:36
*** rpittau|afk is now known as rpittau07:30
rpittaugood morning ironic! o/07:30
jandershey rpittau o/07:37
rpittauhey janders :)07:37
iurygregoryhey rpittau o/07:37
rpittauhey iurygregory :)07:38
arne_wiebalckGood morning rpittau o/07:47
rpittauhey arne_wiebalck :)07:49
dtantsurgood morning ironic07:49
iurygregorygood morning dtantsur 07:49
arne_wiebalckhey dtantsur o/07:50
rpittauhey dtantsur :)07:50
dtantsuriurygregory, TheJulia, who will write a Xena summary for ironicbaremetal.org?07:50
dtantsur(and we need to update the release version there)07:51
iurygregoryhummm good question, I don't have enough time today (but tomorrow for me should be fine)07:52
dtantsurit's not urgent-urgent, ideally to be done before the official OpenStack release07:53
iurygregoryoh ok :D07:53
dtantsur(maybe TheJulia will want it, it's easier for a native English speaker, after all)07:53
dtantsurThe version update is https://github.com/OpenStackweb/ironic-website/pull/42 thanks rpittau 07:56
rpittaunp :)07:56
jandershey dtantsur o/08:01
opendevreviewJacob Anders proposed openstack/ironic master: Make iDRAC management steps verify steps  https://review.opendev.org/c/openstack/ironic/+/80403208:03
rpittauif any core has a moment, we need approval on https://review.opendev.org/c/openstack/bifrost/+/810430 thanks!08:04
iurygregoryrpittau, done08:05
dtantsurand one more time done :)08:05
rpittauthanks! :D08:05
iurygregorydouble workflow LOL08:05
dtantsurI'll request a virtualbmc release, any objections?08:06
dtantsurI guess that's the last thing we haven't released?08:06
iurygregoryright08:06
rpittauI guess we'll do ironic-staging-drivers manually ?08:08
dtantsuroh. right. let me see if its CI still works even.08:08
iurygregory^ the best part08:08
dtantsuralways :)08:08
dtantsuris anyone going through the post-release checklist?08:10
iurygregoryI can check things08:12
iurygregorydo we still need to push an empty commit like we mention in our docs?08:14
iurygregory"create an empty commit with a Sem-Ver tag to bump the generated minor version. See example and pbr documentation for details."08:15
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Support debian 11 "bullseye"  https://review.opendev.org/c/openstack/bifrost/+/80632808:15
opendevreviewDerek Higgins proposed openstack/ironic-lib master: Allow qemu-img to write out zeros to disk  https://review.opendev.org/c/openstack/ironic-lib/+/80899308:16
opendevreviewMerged openstack/ironic-inspector master: Update master for stable/xena  https://review.opendev.org/c/openstack/ironic-inspector/+/81050708:16
opendevreviewMerged openstack/ironic-inspector stable/xena: Update .gitreview for stable/xena  https://review.opendev.org/c/openstack/ironic-inspector/+/81050508:17
opendevreviewMerged openstack/ironic-inspector stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena  https://review.opendev.org/c/openstack/ironic-inspector/+/81050608:17
opendevreviewMerged openstack/ironic-python-agent master: Update master for stable/xena  https://review.opendev.org/c/openstack/ironic-python-agent/+/81050308:17
opendevreviewMerged openstack/bifrost master: Add redfish vmedia jobs to gate  https://review.opendev.org/c/openstack/bifrost/+/81043009:07
dtantsurwow, the staging-drivers CI works! I'll proceed with a release09:33
opendevreviewMerged openstack/ironic-python-agent master: Add Python3 yoga unit tests  https://review.opendev.org/c/openstack/ironic-python-agent/+/81050409:35
opendevreviewMerged openstack/ironic-python-agent stable/xena: Update .gitreview for stable/xena  https://review.opendev.org/c/openstack/ironic-python-agent/+/81050109:35
opendevreviewMerged openstack/ironic-python-agent stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena  https://review.opendev.org/c/openstack/ironic-python-agent/+/81050209:35
dtantsuriurygregory: re Sem-Ver, I think the "Prepare master" commit does it09:40
dtantsuryep, see e.g. https://opendev.org/openstack/ironic-python-agent/commit/1665abca04ff55a4f91d3e1a6111499f4f3f233309:40
iurygregorydtantsur, oh :D ok (we need to update our docs :D )09:40
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Support debian 11 "bullseye"  https://review.opendev.org/c/openstack/bifrost/+/80632810:13
timeuHey everyone, after an inplace upgrade from OSP16.1 to OSP16.2, we get following error: 11:25
timeuDuring sync_power_state, could not get power state for node 5a770848-5ee9-4b96-8560-d0434088779a, attempt 1 of 3. Error: An exclusive lock is required, but the current context has a shared lock..: ironic.common.exception.ExclusiveLockRequired: An exclusive lock is required, but the current context has a shared lo11:25
timeuck11:25
timeuI already restarted the conductor service but it didn't help 11:26
iurygregoryOSP 16.2 is out? =O11:31
timeuyeah ;-) 11:32
opendevreviewVerification of a change to openstack/ironic master failed: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81032711:34
iurygregorytimeu, based on the ironic code this is a warning do you see the other attempts to sync the power state and they also fail?11:38
bkrahi! does anyone know how I can use ironic-conductor with both redfish and ipmi, i constantly get exceptions threwn like this for mutliple interfaces:  Failed to register hardware types. For hardware type 'redfish', no default value found for power interface.11:38
bkracant seem to find anything about it on the web, thanks11:39
bkrai did set the defaults in ironic.cfg though11:39
arne_wiebalckbkra: I think we have a config which handles both11:40
iurygregoryyou added the values after you got the error?11:40
janderssee you tomorrow Ironic o/11:40
timeuiurygregory: Well it tries to sync 3 times and then sets the node into maintenance. It works fine for another node. First I thought it can't reach the BMC or so but I tested it with ipmitool from the conductor container and it works11:40
iurygregoryif you updated the ironic.conf you need to re-start the services..11:40
dtantsurtimeu: it seems to be a bug in ironic, do you have a traceback?11:40
arne_wiebalckbkra: we set: enabled_hardware_types , enabled_management_interfaces , enabled_power_interfaces11:41
dtantsurbkra: you definitely can, but as arne_wiebalck said you need to set a couple more options11:41
bkrayeah, i restarted the kolla container ofcourse :) and tried all sorts of combinations11:41
arne_wiebalckbkra: I did as well initially :-D11:41
arne_wiebalckbkra: let me share my config ...11:41
dtantsurthe message is telling you that there is nothing compatible with redfish in your enabled_power_interfaces11:41
bkrathese are my enabled_ / default_ cfg lines :) https://pastebin.com/hjPXPtF911:42
dtantsurbkra: https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/tasks/hw_types.yml may give you some inspiration11:42
bkrathank you, will look into it11:42
arne_wiebalckbkra: this is how we added redfish: https://paste.opendev.org/show/809533/11:43
arne_wiebalckbkra: the console is our own11:43
dtantsurbkra: one more thing: I'd recommend against setting default_*_interface for driver-specific interfaces11:43
dtantsurdefault_power_interface=redfish will work poorly for ipmi nodes11:43
dtantsuron the other hand, once you have the necessary interfaces enabled, ironic can pick the right default11:43
timeudtantsur: Let me unset the maintenance and wait for the error message and I paste the DEBUG logs in a pastebin. The weird thing is that the node which was in use during the OSP 16.1-> OSP 16.2 update works fine now, however the other ones that were not in use are now failing11:43
bkraah right, i thought i had to set all defaults by hand11:44
bkrai'll try your input, thanks!11:44
dtantsurnope, only when ironic doesn't guess correctly11:44
dtantsurusually used with deploy interfaces11:44
timeuhttps://pastebin.com/v2ZBatmX: this is from the node update (unset of maintenance) to the first power sync error message (after 3 of them the node is set to maintenance again)11:45
timeuactually it already sets it right away to maintenance mode after the first failed power sync11:46
dtantsurno traceback, this is very unfortunate11:47
timeuyeah :-/ I don't understand this log line: "node state None does not match expected state 'None'. Updating DB state to 'None'". Maybe something wrong in the database ?11:47
dtantsurtimeu: I suspect it's a poorly worded message11:48
dtantsurtimeu: which driver do you use for this node?11:49
timeuipmi11:49
timeuthis is ironic on the undercloud btgw11:49
dtantsuryeah, so the None stuff is a red herring11:50
timeuthe node that works is identital to the other 2 nodes that fail (hardware, network, config, etc) and they did work before. I just had the overcloud deployed with 1 of the nodes and now wanted to dpeloy with 3 of them 11:50
dtantsur(I will fix it later on)11:50
timeuand the message: "An exclusive lock is required, but the current context has a shared lock." ? 11:51
dtantsuryep, this is an actual issue, and it's a bug. I'm just struggling to understand how it's triggered11:51
dtantsurwhat you see is pretty much an assertion, it should never be hit11:51
timeuhow are the locks managed ? are the database records ? 11:55
timeuI am wondering if something went wrong during the in place update (database migration) because the one node that doesn't show error was in use during the upgrade while the other two weren't 11:56
dtantsurtimeu: shared locks are not actually locks. I don't think you could trigger that, to be honest.11:57
opendevreviewDmitry Tantsur proposed openstack/ironic master: require_exclusive_lock: log traceback that lead to an error  https://review.opendev.org/c/openstack/ironic/+/81061511:59
dtantsurtimeu: do you think you can patch your conductor to include ^^ and retry?11:59
bkrais there a value that i can set that the configparser will interpret as None / not-set? Kayobe/Kolla requires me to set a value for the default_* cfg lines.12:00
timeudtantsur: sure12:00
bkracommenting all default_* works btw 12:01
dtantsurbkra: I don't think so, unfortunately. It's a bug in Kolla if it does require that.12:01
bkraalright, ill work something out 12:01
bkracheers12:01
dtantsurbkra: you can try empty strings, I don't remember if ironic will accept that though12:01
opendevreviewVerification of a change to openstack/ironic master failed: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81032712:04
timeudtantsur: FYI: I tested this with another node that was not deployed during the update and the same error happens again while another node that was not in use didn't show the error. Will apply your patch and post the traceback. 12:04
timeudtantsur: https://pastebin.com/T6bQQEKX12:16
* dtantsur blinks12:20
dtantsuroookay, this code was killed with fire in master, but is probably still in train12:20
dtantsurtimeu: okay, so it's probably a bug in the ussuri/train backports of this feature12:24
dtantsurwhich is probably newly added in 16.212:24
dtantsurfortunately, I have an easy workaround for you:12:24
opendevreviewVerification of a change to openstack/ironic master failed: Clean up caches periodically  https://review.opendev.org/c/openstack/ironic/+/81042812:24
dtantsur$ baremetal node set <node> --property vendor=ignoreme12:24
opendevreviewDmitry Tantsur proposed openstack/ironic stable/ussuri: [Stable only] remove require_exclusive_lock from detect_vendor  https://review.opendev.org/c/openstack/ironic/+/81065612:27
dtantsurtimeu: the real fix should be as simple as ^^^12:27
dtantsurTheJulia, hjensas ^^^12:27
dtantsurtimeu: you may need to file a bugzilla so that the OSP team can backport this fix12:29
dtantsuriurygregory: I've got staging-drivers stable/xena, now waiting for https://review.opendev.org/c/x/ironic-staging-drivers/+/81065712:36
iurygregorydtantsur, you have my +112:37
iurygregorythat's all I can do :D12:37
dtantsurlet me fix that12:37
dtantsuriurygregory: okay, I've added ironic-core to ironic-staging-drivers-core. I've also added Julia and you to ironic-staging-drivers-release. I hope pas-ha[m] and vdrok don't object.12:39
iurygregorydtantsur, ack12:40
* dtantsur has fixed bus factor12:40
* iurygregory just learned a new thing today - bus factor tks dtantsur 12:43
dtantsur:)12:44
opendevreviewVerification of a change to openstack/ironic master failed: Clean up caches periodically  https://review.opendev.org/c/openstack/ironic/+/81042812:44
timeudtantsur: Thanks the vendor workaround fixed the error and I will create a bugzilla bug12:49
opendevreviewDmitry Tantsur proposed openstack/ironic master: require_exclusive_lock: log traceback that lead to an error  https://review.opendev.org/c/openstack/ironic/+/81061512:49
dtantsurcool12:49
rpittaudtantsur: do we have the manual procedure for ironic-staging-drivers release documented somewhere?12:51
dtantsursigh, probably no. lemme do it.12:51
rpittausorry, didn't want to generate more work for that! :D12:52
timeucreated bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=200726812:55
dtantsurthx!12:58
dtantsuriurygregory, rpittau ^^^12:59
dtantsurooops, there is no bot here, right?12:59
dtantsurso the link is https://review.opendev.org/c/x/ironic-staging-drivers/+/81066212:59
rpittauI was lost for a moment12:59
dtantsurI'm so used to the bot that sometimes I do these ^^ automatically :D13:00
rpittauthanks for that dtantsur :)13:02
iurygregoryno wories :D13:03
iurygregoryworries*13:03
TheJuliarutro, busses, where?!?13:07
dtantsurchoo-chooo13:07
TheJuliaAlthough in the states, we prefer to use "win the lottory"13:07
dtantsurgood morning TheJulia 13:07
TheJulia"Oh, xyz won the lottory" "how much?" "Enough, they can actually retire"13:08
TheJulia"well, it was good working with them"13:08
* TheJulia tries to wake up13:10
TheJuliadtantsur: iurygregory I can write a summary. Likely later today or tomorrow, I'm in no rush. After chatting with sdanni yesterday I feel like writing another blog post on rbac13:12
iurygregoryack13:13
TheJuliaso, I started down the path of swapping over the default boot mode config13:14
TheJuliasine CI was so unhappy yesterday13:14
TheJulialong story short, OUCH13:14
dtantsur:)13:20
TheJulialots of tests have to be duplicated too13:21
TheJuliaat least it looks like in some cases we did actually appropriately cover the other test paths with uefi....13:36
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Enable authentication in sushy-tools  https://review.opendev.org/c/openstack/bifrost/+/81068613:57
dtantsurI keep seeing picture like this in our jobs https://zuul.opendev.org/t/openstack/build/92071bbcc45f435f8806436ae38e5bb6/log/controller/logs/ironic-bm-logs/node-3_console_log.txt14:03
dtantsurdoes anyone has a clue? is it a memory problem?14:03
dtantsurhuh, and PXE booting https://zuul.opendev.org/t/openstack/build/92071bbcc45f435f8806436ae38e5bb6/log/controller/logs/ironic-bm-logs/node-0_no_ansi_2021-09-23-13:09:26_log.txt14:04
iurygregorydtantsur, we did reduce VM_SPEC_RAM https://opendev.org/openstack/ironic/commit/34fd84560a3e3e43f978e2760ef448243553138b14:09
dtantsurhuh, it's a never-ending chase14:09
dtantsuralthough 2G should be more than enough for tinyIPA14:10
iurygregoryyeah14:10
TheJuliadtantsur: memory *or* logging is just not going to serial console :\14:20
TheJuliaso pxe booting issues I'd tie with neutron issues right now14:20
TheJuliain fact, I wonder if there has been any progress on the firewall code not working14:20
TheJuliaugh, current answe seems to pin OVS version14:21
TheJuliaugh14:22
dtantsurnon wow14:33
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Enable authentication in sushy-tools  https://review.opendev.org/c/openstack/bifrost/+/81068614:33
iurygregoryI remember someone was going to try to add some retry logic in neutron no?14:40
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Enable authentication in sushy-tools  https://review.opendev.org/c/openstack/bifrost/+/81068614:42
TheJuliaiurygregory: that was what was said originally, but a follow-up comment if memory serves noted that likely would be harder14:46
TheJuliaI think they are seeking to try and head off/prevent it from ever happening, but at some point race conditions are a thing and there has to be some handling should one thread get blocked unexpectedly for a little longer14:47
iurygregory=(14:47
TheJuliaNext, ask me how I know about shared memory cache on SMP systems and how that just doesn't exist as a workable concept on VMs14:48
dtantsuroh ffs, fixing one thing in sushy-tools breaks another....14:53
TheJuliaThats how I feel like with going from bios to uefi14:54
TheJuliaI literally ahve a few tests failing in ways I would have never expected14:55
dtantsur:(14:55
TheJuliawhich makes me worry there is a larger bug someplace that I need to find in all of it14:55
TheJuliawell14:55
TheJuliaif it is a minor behavior difference between two defaults, it is likely not a bug bug that needs to be backported14:55
TheJuliabut *still*14:55
TheJuliadown to 18 tests14:56
opendevreviewDmitry Tantsur proposed openstack/sushy-tools master: Fix passing configuration via WSGI  https://review.opendev.org/c/openstack/sushy-tools/+/81070314:59
dtantsura new attempt to fix everything ^^^14:59
iurygregoryops :D15:12
opendevreviewMerged openstack/ironic master: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81032715:19
dtantsurTheJulia: needs backporting ^^?15:58
TheJuliayes, like 7 minutes of ci runs is that... 16:01
sdanniTheJulia: hi! I wonder what the correct CLI for 'openstack baremetal node list' would be for a system scoped admin? I run "openstack --os-system-scope=all baremetal node list" but still see an empty list. 16:02
opendevreviewJulia Kreger proposed openstack/ironic stable/xena: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81073116:02
opendevreviewJulia Kreger proposed openstack/ironic stable/wallaby: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81073216:02
rpittaugood night! o/16:02
*** rpittau is now known as rpittau|afk16:02
sdanni"openstack --os-username=<username> --os-user-domain-name=<domain> --os-system-scope all token issue" gives me an error "Missing value auth-url required for auth plugin password", so I can't generate a token.16:02
mnasiadkaTheJulia: I'm back with a stupid question - it seems I need to set rootfs_uuid, but I'm using bifrost - so it seems it's rather root_uuid_or_disk_id in driver-internal-info - how do I set that using CLI?16:06
dtantsurdriver_internal_info is immutable, but there should be a way to set it differently16:07
dtantsurI think somebody has recently fixed it16:07
dtantsurdo you mean --instance-info image_rootfs_uuid=<uuid>?16:07
dtantsurmnasiadka: ^^16:07
dtantsur(apparently somebody was me. lol)16:07
mnasiadkait would be good to update the docs - those only mention rootfs_uuid image property16:08
dtantsurmnasiadka: in the bottom of https://docs.openstack.org/ironic/latest/user/deploy.html#image-information16:09
mnasiadkadtantsur: oh, but it's not mentioned on software RAID topic in the docs, anyway - thanks! ;)16:09
dtantsurI guess we should link/duplicate it in the software RAID docs16:10
dtantsurpatches welcome :)16:10
dtantsurhave a nice evening folks o/16:10
TheJuliasdanni: you'll need to provide an auth url to provide a token if you don't have the env var loaded16:13
sdanniTheJulia: thanks! it works after I export OS_AUTH_URL and OS_IDENTITY_API_VERSION16:16
TheJulia\o/16:17
opendevreviewVerification of a change to openstack/ironic master failed: Clean up caches periodically  https://review.opendev.org/c/openstack/ironic/+/81042817:53
stevebakergood morning20:32
TheJuliaGood morning20:50
opendevreviewMerged openstack/ironic stable/xena: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81073121:11
stevebakerTheJulia: So it looks like vexxhost nova doesn't support UEFI boot. I think that basically rules out upstream tripleo UEFI boot testing21:13
opendevreviewVerification of a change to openstack/ironic master failed: Clean up caches periodically  https://review.opendev.org/c/openstack/ironic/+/81042821:17
jandersgood morning Ironic o/22:02
opendevreviewMerged openstack/ironic stable/wallaby: Devstack: don't scan /opt, /etc looking for isolinux  https://review.opendev.org/c/openstack/ironic/+/81073222:26
TheJuliastevebaker: so not even explicitly on the flavors?22:37
TheJuliamnaser: ^^22:37
stevebakerTheJulia, mnaser: when booting with an image that has property hw_firmware_type=uefi, the eventual error logged was "UEFI is not supported, Code: 500"22:39
TheJuliapurely as VMs?22:40
* TheJulia needs migraine to go away22:40
stevebakeryeah nova vms22:42
TheJuliaahh22:44
TheJuliayeah22:44
TheJuliathat would do it22:44
TheJuliale sigh22:44
* TheJulia looks at the count of bios mode to uefi mode default change tests still failing and thinks.... maybe tomorrow22:45
stevebakerIt looks like nova compute nodes need to be set up with OVMF from EDK2, or something22:45
opendevreviewMerged openstack/python-ironicclient master: Add support for fields in drivers CLI  https://review.opendev.org/c/openstack/python-ironicclient/+/80597822:45
stevebakerI mean I wouldn't be doing anything with a migraine :)22:46
TheJuliaI spent a good chunk of time on the phone with a hardware vendor helping to troubleshoot weirdness this morning... and now at least I have new glasses22:49
stevebakerTheJulia: if you have the prescription then go have some fun on zennioptical.com. its so cheap it doesn't even matter if they don't work out23:13
TheJuliastevebaker: nice!23:14
stevebakerI'm iterating my way towards the perfect TV/doomscrolling bifocals23:18
TheJulia\o/23:51

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!