Wednesday, 2024-01-17

opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: RedfishFirmwareInterface - Unit Tests & More logs  https://review.opendev.org/c/openstack/ironic/+/90337902:11
opendevreviewlikui proposed openstack/ironic-lib master: Remove compatibility with Python < 3.8  https://review.opendev.org/c/openstack/ironic-lib/+/90581102:36
opendevreviewlikui proposed openstack/ironic-prometheus-exporter master: Update python classifier in setup.cfg  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/90581202:38
opendevreviewlikui proposed openstack/python-ironicclient master: Update python classifier in setup.cfg  https://review.opendev.org/c/openstack/python-ironicclient/+/90581302:40
opendevreviewlikui proposed openstack/ironic-ui master: Update python classifier in setup.cfg  https://review.opendev.org/c/openstack/ironic-ui/+/90581502:44
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/2023.2: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90572504:30
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/2023.1: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90572604:31
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/zed: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90583208:10
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/yoga: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90583308:11
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/xena: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90583408:11
opendevreviewTakashi Kajinami proposed openstack/ironic-python-agent-builder stable/wallaby: Drop TripleO job  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/90583508:13
rpittaugood morning ironic! o/09:07
masgharMorning!09:11
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Switch to snponly and stop copying the unused ipxe.pxe  https://review.opendev.org/c/openstack/bifrost/+/90584410:00
dtantsurTheJulia: we somehow missed bifrost ^^10:00
iurygregorygood morning Ironic11:01
opendevreviewMerged openstack/ironic-lib master: Raise on non-existing file in qemu_img_info  https://review.opendev.org/c/openstack/ironic-lib/+/90561211:11
*** ravlew is now known as Guest1442311:33
*** ravlew2 is now known as ravlew11:37
opendevreviewMerged openstack/ironic master: Revert "Revert "RBAC: Fix allocation check"" to use Unauthorized  https://review.opendev.org/c/openstack/ironic/+/90578413:14
Sandzwerg[m]Morning Ironic o/13:57
Sandzwerg[m]A Question to virtual media together with glance & ISO: Do I need to configure anything else apart from setting deploy_iso=<glance image id> in the driver-info of the node? Because I now tried it with a newer dell node instead of the old cisco and it still seems to skip the virtual-media part. But I have the feeling I'm missing something or do something wrong. Our ironic is still on xena in case that should matter14:00
TheJuliaGood morning14:14
* TheJulia starts sipping coffee14:14
TheJuliaSandzwerg[m]: so deploy_iso is much more "this is a pre-existing iso image" to use, where as we prefer to start with a kernel/ramdisk file and master an ISO on the fly as we include configuration in the ISO. If I'm recalling correctly, deploy_iso is much more a hold over from earlier virtual media drivers14:17
Sandzwerg[m]Hope you're healthy again14:17
TheJuliaI mean, it should work, but it might not be the preferred path/behavior14:17
TheJuliawork up this morning feeling human! So, so far so good, albeit a little cold for me in the desert this morning14:18
Sandzwerg[m]hmmmkay. Maybe I need to look into the build-iso on the fly then. I just build our IPA as ISO instead of bothering with installing the requirements to build ISOs on the fly. But so far I have the impression that it does not try to boot from the ISO (or any other kind of virtual media) instead it just does the default boot order which does not include virtual media14:20
Sandzwerg[m]Need to check in the bios if I can add virtual media to the boot order 🤔14:20
Sandzwerg[m]And feeling human sounds like a good start14:21
TheJuliaSandzwerg[m]: Are you in bios mode or uefi mode? I guess the thing is we explicitly ask for the CD device to be booted from14:23
TheJuliaor DVD device14:23
Sandzwerg[m]UEFI14:23
TheJuliaI guess it would help to figure out where things are *actually* falling sideways at14:23
TheJuliain your logs, you should see us assering a BootSourceOverrideTarget14:24
TheJuliaasserting14:24
Sandzwerg[m]yeah but that's still a mistery to me. Should something be in the conductor log?14:24
Sandzwerg[m]Allright let's dig goes into the  log mines https://www.youtube.com/watch?v=ytWz0qVvBZ014:24
TheJuliaSandzwerg[m]: yeah, that will be in the conductor log14:26
TheJulialook for errors from "sushy"14:26
TheJulia+2+A to the video14:26
TheJuliaMy wife has played that video far too many times in the home office14:28
Sandzwerg[m]There is nothing like "to manz times" when it comes to that video 👼14:28
Sandzwerg[m]OK the fact that did not set the node to virtual-media when I was 100% sure I did is a good explanation why it didn't work so far...14:31
Sandzwerg[m]hmm ok that is Interesting. I set it to virtual media, boot, and it's back at ipxe...14:32
Sandzwerg[m]I assume that should not happen?14:32
TheJuliano, that should not really happen14:33
TheJuliaon the node boot_interface field?14:33
Sandzwerg[m]yeah14:33
TheJuliaAFAIK there is no reason why that would have just "changed"14:34
TheJuliaat least, in ironic itself14:34
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent stable/2023.2: Add missing headers to the inspection callback  https://review.opendev.org/c/openstack/ironic-python-agent/+/90573914:34
opendevreviewDmitry Tantsur proposed openstack/sushy stable/2023.2: Handle a different related properties for missing TransferProtocolType  https://review.opendev.org/c/openstack/sushy/+/90574014:35
Sandzwerg[m]yeah seems very strange. BTW regarding boot mode: the node itself is set to UEFI but we haven't set the boot mode in ironic 14:35
Sandzwerg[m]I need to look into this tomorrow, need to run some errands and probably have no more time to today. Thanks for the help so far \o/14:35
TheJuliaSandzwerg[m]: as long as ironic's understanding matches, it should be fine then14:37
TheJuliait will try to assert whatever it thinks the state/mode is as well14:37
TheJuliaso at the end of the day it just noops14:37
TheJuliaor at least *should* noop14:37
Sandzwerg[m]For us we just want to boot via uefi instead of bios but don't need to set any specific uefi stuff. Just write the whole disk image to disk and we're fine 14:57
Sandzwerg[m]And I think setting uefi as boot mode would go in that direction but I would need to look it up14:58
TheJuliao/ can I get another review for https://review.opendev.org/c/openstack/ironic-inspector/+/905353 please?15:00
opendevreviewHervé Beraud proposed openstack/ironic master: bump eventlet to latest version that support python 3.12  https://review.opendev.org/c/openstack/ironic/+/90588715:00
opendevreviewHervé Beraud proposed openstack/ironic-inspector master: bump eventlet to latest version that support python 3.12  https://review.opendev.org/c/openstack/ironic-inspector/+/90588815:01
opendevreviewHervé Beraud proposed openstack/ironic-python-agent master: bump eventlet to latest version that support python 3.12  https://review.opendev.org/c/openstack/ironic-python-agent/+/90588915:02
iurygregoryTheJulia, looking now15:03
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.2: Revert "Revert "RBAC: Fix allocation check"" to use Unauthorized  https://review.opendev.org/c/openstack/ironic/+/90590115:04
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.1: Revert "Revert "RBAC: Fix allocation check"" to use Unauthorized  https://review.opendev.org/c/openstack/ironic/+/90590215:05
opendevreviewHervé Beraud proposed openstack/networking-generic-switch master: bump eventlet to latest version that support python 3.12  https://review.opendev.org/c/openstack/networking-generic-switch/+/90592115:10
dtantsurmy first reaction to https://review.opendev.org/q/topic:%22bump-eventlet%22 is to -2 all of them.. is there any reason why my first reaction should not become my final reaction?16:07
JayFWhy would that be your reaction at all?16:07
JayFI would -2 a reaction of that kind :D 16:07
dtantsurJayF: that's not the appropriate way to bump version16:07
dtantsurdo you suggest we change how requirements are managed?16:07
JayFare you in #openstack-oslo, I have been having this conversation in there with someone, too16:08
JayFwe have it bumped in u-c already16:08
dtantsursure, these two are not related16:08
JayFthere is no other way to force a lower bound16:08
dtantsurfortunately, you don't need to force it?16:08
JayFexcept in each project individually aiui16:08
JayFYou don't need to; I agree with the idea of proposing it everywhere due to the massive improvements/cleanups in eventlet16:08
JayF-1 with cause in a project would be a sensible thing. Rejecting out of hand is not sensible because of the significant increase in quality of the underlying library. We're saying "we don't want to support OpenStack on the unsupported, no CI tested versions of eventlet" which is a really good thing to say16:09
dtantsurTHIS is the argument I would buy16:10
JayFThis is 100% how I took it based on context that has been in the mailing list around eventlet16:10
dtantsuryeah, but the operators may not have this context16:12
dtantsurand eventlet updates are not to be taken lightly16:12
JayFYeah, it was suggested in -oslo an email should go out about the updates16:13
JayFI was like "sure whatver, if you think so"16:13
JayFthen you were like "WTF" in here literally 30 seconds later16:13
JayFthe continuity between the two conversations was uncanny :)16:13
TheJuliaheh16:13
dtantsurI'd rather have a release note, to be honest16:13
dtantsurmaybe we can write one ourselves16:14
TheJuliaI'm surprised to hear you say that 16:14
JayFI am going to pair with adamcarthur5 for a bit this morning, after that if someone else hasn't, I'll email the list about the update. 16:14
TheJuliasince you've pushed back on other dependency updates being inremented being explicitly noted in renos.16:14
JayFI am torn on the idea of a release note; but I don't want hberard getting beat up for what's ultimately a good thing with bad PR :D 16:14
dtantsurTheJulia: I have, but this for me is an exceptional case16:15
TheJuliafair16:15
dtantsurIt's less of "We now require this", it's more of "Eventlet has been broken, and now we require the only fixed version"16:15
dtantsurMy reasoning is usually "they'll see the update in the requirements". Here I want to convey the scope of the problem.16:15
JayFthis will also say loudly to downstreams16:16
dtantsurKind of a "fixes:" release note for Ironic changes16:16
JayFif you package openstack with old-eventlet16:16
JayFand you break things16:16
* dtantsur hopes he makes sense16:16
JayFhere, have all the pieces16:16
dtantsurexactly16:17
dtantsuranyway, thanks for limiting my first reaction to the first reaction JayF :)16:18
Sandzwerg[m]<Sandzwerg[m]> "For us we just want to boot..." <- So setting virtual media also sets boot mode to uefi, then on deployment (or shortly before but that would be surprising, still needs more testing) the ironic service user sets the node to maintenance, sends a patch call und unsets maintenance 16:31
opendevreviewVerification of a change to openstack/ironic-inspector master failed: Fix RBAC access for service/admin user access  https://review.opendev.org/c/openstack/ironic-inspector/+/90535316:37
opendevreviewMerged openstack/networking-generic-switch master: bump eventlet to latest version that support python 3.12  https://review.opendev.org/c/openstack/networking-generic-switch/+/90592116:41
opendevreviewAdam McArthur proposed openstack/ironic-lib master: [codespell] - Adding CI for codespell tox target  https://review.opendev.org/c/openstack/ironic-lib/+/90562416:44
opendevreviewAdam McArthur proposed openstack/ironic-lib master: [codespell] - Adding CI for codespell tox target  https://review.opendev.org/c/openstack/ironic-lib/+/90562417:02
rpittaugood night! o/17:04
opendevreviewAdam McArthur proposed openstack/ironic-python-agent master: [Codespell] Adding tox target to CI (Non-Voting)  https://review.opendev.org/c/openstack/ironic-python-agent/+/90598017:17
JayFAh, we need to make sure eventlet .34.3 is in all supported openstack platforms before we can require it. That may not be likely :/ 18:14
TheJuliaUnlikely is more so the thing.18:40
TheJuliaSandzwerg[m]: nothing we would be doing sets a node to maintenance to patch the node, and unsets it, that really doesn't explain it. What is calling Ironic in that context?18:41
JayFYeah, chatter in -tc makes this sound like it's more likely these changes won't land, but I get the impression either nobody read my ML post or does not want to engage in a longer form so the conversation is bifurcated instead of concentrated :( 18:42
TheJuliaThe nature is going to be generally bifrucated because human can only hold so much context18:43
JayFyeah, but that's exactly why I was hoping we'd venue-swap to something friendlier to async, since the actual-changes were pushed by someone who I think (hope?) is asleep18:45
TheJuliahope++18:47
TheJulia(but I think, yes, they should be afaik)18:47
opendevreviewJulia Kreger proposed openstack/networking-generic-switch master: Revert "bump eventlet to latest version that support python 3.12"  https://review.opendev.org/c/openstack/networking-generic-switch/+/90590418:50
TheJuliaI think there is also hope in terms of "we hope to move things on faster", and often not everyone is ready/able/capable to ride along at the same speed.18:50
TheJuliaI'm all for pushing it as a req if "we as a project *know* people should just not be running the version of eventlet because it is inherently broken"18:51
JayF*I* feel that way.18:51
JayFIt's not clear to me that is the project consensus.18:51
JayFAnd a good point was made in -tc that we are past C-2, it's maybe a bit late to make a change of this size.18:52
JayFand that argument hit me right in the ops18:52
TheJuliaoften humans will error on not changing until they are explicitly forced by some driving thing. Often it might be context18:53
TheJuliayour raising very good points though, fwiw :)18:53
TheJulia"right in the ops" is a new one18:53
TheJuliarutro https://498c49910e2e429f6f6d-5ec4103c89b5d2feb0a0ca5aa5752932.ssl.cf1.rackcdn.com/905353/1/gate/openstack-tox-docs/320c0d4/tox/docs/2-commands%5B0%5D.log18:58
JayFhahahahahahahahthud19:00
JayFCan you put that into an eventlet issue?19:00
TheJuliasure, likely in a few minutes19:01
TheJuliaI'm not sure I'm signed into gh on this laptop right now19:01
TheJulianope, doesn't look like it, checking other browser19:01
TheJuliaI don't have my password store handy eitehr at the moment (sitting in an urgent care lobby)19:02
TheJuliayeah, nope, it will have to wait until get back to the house19:02
JayFack; we can race, I'm about to pair again with adamcarthur5 19:03
JayFlook like IP_VERSION=4 doesn't block ipv6 use anymore19:03
JayFso I gotta find out what odes :D 19:03
TheJulianothing really does, I think devstack does 4+6 by default now19:04
TheJuliaalthough did it break in an unexpected way?19:05
TheJuliaalso can't repro issue locally, apparently the new build wants to build yappi which won;t build due to a syntax issue according to gcc19:05
JayFHe has a machine with no v6 access19:05
JayFand iP_VERSION=4 is set, it's still doing v6 stuff19:05
TheJuliathat is bizzare19:06
clarkbdevstack networking is independent of your v6 access. I don't find that bizarre19:08
clarkbyou can have a fully working v6 stack between the hypervisor and the VMs that neutron connects19:08
clarkband you can extend that to multinode testing via overlay networking. This is all important because the clouds we test on don't necessarily support v6 but we still need to test it19:09
clarkbbasically whether or not you have a publicly routable v6 address is completely irrelevant to devstack19:09
JayFit's disabled by their VPS at a kernel level :/ 19:09
JayFWe figured it out, we're going to fix the doc and it'll be obvious19:10
TheJuliaoh, becuase parameters still end up in the v6 style?19:12
JayFNo; bitrotted doc19:12
TheJuliaahh19:12
JayFIP_ADDRESS vs IP_VERSION19:12
JayFinconsistently in our doc19:12
JayFIP_VERSION is correct19:12
clarkbdisabling v6 at the kernel level is an interesting decision. Almost makes me wonder if the hosting provider doesn't provide adequate v6 isolation and they're faking it this way19:13
clarkbjust hope no one figures out how to override the setting and start acting as a router :)19:14
JayFI do not know. Not my choice of provider and I'm trying to go with the flow of adamcarthur5's setup as best as possible :)19:14
JayFhe has all sorta crazy devpoddy magic setup19:14
JayFoh, I think he could turn it on, but again I don't want to mess with his setup19:15
clarkbyou need v6 to test a large numebr of things19:16
clarkbmostly neutron related19:16
opendevreviewAdam McArthur proposed openstack/ironic master: Corrected IP_ADDRESS to IP_VERSION  https://review.opendev.org/c/openstack/ironic/+/90599619:16
clarkbagain not publicly routable v6 just v6 on your sytstem19:16
* TheJulia glares at the 67kB/sec download19:17
JayFYep, understood. For now just literally trying to get him to "hello world, this is an openstack API" so he gets to experience more of our world than setting up tox :D 19:18
TheJuliaerr, ugh, my inability to repro is python 3.12 *and* the lack of packaging for other 3.11 assets compat assets and another 3.12 incompatibility19:30
opendevreviewMerged openstack/networking-generic-switch master: Revert "bump eventlet to latest version that support python 3.12"  https://review.opendev.org/c/openstack/networking-generic-switch/+/90590419:55
opendevreviewMerged openstack/ironic master: Corrected IP_ADDRESS to IP_VERSION  https://review.opendev.org/c/openstack/ironic/+/90599620:06
opendevreviewJay Faulkner proposed openstack/ironic master: Fix bug in devstack-guide: readd server create  https://review.opendev.org/c/openstack/ironic/+/90600120:06
TheJuliaugh, we have doc issues in ironic21:41
TheJuliaand also 10 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.21:41
TheJuliais getting generated through autodoc's run21:41
JayFI can point brain in this direction after walking the dog; I suspect we're going to have to somehow monkey_patch() earlier *or* avoid monkey_patch in the sphinx/docs case, somehow(?)21:42
TheJulia... dunno21:43
TheJuliaIt fails in inspectors docs, ironic is configured the same, but we have some different errors I'm trying to chase down now21:43
TheJuliaautodoc: failed to import module 'nics' from module 'ironic.drivers.modules.inspector'; the following exception was raised:21:43
TheJuliaNo module named 'ironic.drivers.modules.inspector.nics'21:43
JayFTheJulia: so despite that being logged at error, it's nonfatal: https://github.com/eventlet/eventlet/blob/master/eventlet/patcher.py#L41221:45
JayFand I can confirm it was made a logged error because essentially greening locks after the fact never worked21:45
TheJuliayeah, ironic's gate runs without -W21:45
TheJuliainspector's does21:45
JayFthis was the root cause for ironic-unit-test-crap (we monkey_patched late + eventlet couldn't green the locks)21:45
TheJuliajust getting to other errors locall with ironic when I toggle the flag21:45
JayFmakes sense, I just was making sure that eventlet error wasn't fatal, logging and going on is an OK way to handle that21:46
* JayF brb need to walk the dog before too late21:46
TheJulia that sounds ominous21:46
TheJuliasomehow, autodoc is hellbent on assuming a ironic/drivers/modules/inspector/nics.py exists21:47
TheJuliarm -r -f .tox21:48
TheJuliaokay, my errors on ironic were just cached data21:55
TheJuliafrom other/older runs of autodoc21:55
TheJuliaI still get the lock error as autodoc comes along and opens the file21:56
TheJuliaso I think it is more werkzeug22:06
opendevreviewMerged openstack/ironic stable/2023.2: Revert "Revert "RBAC: Fix allocation check"" to use Unauthorized  https://review.opendev.org/c/openstack/ironic/+/90590122:06
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: Fix monkey_patch in autodoc execution  https://review.opendev.org/c/openstack/ironic-inspector/+/90601122:20
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: Fix monkey_patch in autodoc execution  https://review.opendev.org/c/openstack/ironic-inspector/+/90601122:20
TheJuliathat feels slightly dirty, but it works22:20
iurygregorytry except pass :D 22:35
iurygregoryonly in inspector the problem?22:35
TheJuliafor us afaik22:36
iurygregoryack22:36
JayFTheJulia: is this right re: https://github.com/eventlet/eventlet/issues/896 > There are cases, demonstrated by CI for https://review.opendev.org/c/openstack/ironic-inspector/+/906011/, where that isn't possible. In this specific case, Sphinx autodocs -- I'm not sure for scenarios like this how to sanely monkey_patch beforehand. 22:38
JayFI would strongly prefer us find a better solution than 906011 though. I might spend a little time on this.22:38
TheJuliayeah, it doesn't feel "right", but a very similar thing was done with sphinx in general 22:39
TheJulialocally, fwiw, I monkey patched sphinx, but the problem is it is a runner22:39
TheJuliaat least, I think it is a separate runner process22:40
JayFYeah, I think this is actually a reasonable case to ask eventlet to handle upstream or at least give an escape hatch for22:40
JayFI was going to take the approach of seeing if there's some sort of special variable set when doc's generation is running and if that variable is set not patching at all22:41
TheJuliayeah, already went looking through autodoc but it doesn't give us a firm handle of any sort22:41
JayFYou sure there's not even something in like actual environment? At a minimum, perhaps it passes through the external environment and we could set our own canary22:42
TheJuliahttps://github.com/openstack/ironic/blob/master/doc/source/conf.py#L19-L24 <-- the prior art which I tried appling but not sure it was actually patched22:42
JayFEither way I'll poke at it. I've done more with Sphinx than I care to admit22:42
TheJuliawell, I can dupe it in CI and I can dupe it locally on my desktop22:42
JayFYeah I'm pretty sure I won't have any trouble duplicating it. I'm still wrapping up from walking the dog but I should be able to go fully heads down on this in like 1522:43
TheJuliak22:44
JayFit happily duplicates lmao23:34
JayFand I can get around autodocs, then it just angers other stuff23:34
JayF(like oslo config generation) https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_mock_imports23:34
JayFheh, and I just realized: importing *anything* to determine if we're doing docs generation before monkey patching eventlet will break the non-monkey_patched case23:37
JayFheh TheJulia I have a solution IDK if you're going to like it, but I think it's better than 90601123:40
TheJuliapropose it!23:40
opendevreviewJay Faulkner proposed openstack/ironic-inspector master: Monkey patch eventlet at base module level  https://review.opendev.org/c/openstack/ironic-inspector/+/90601223:41
JayFI suspect this would be the Itamar-approved fix to this bug too, I sorta asked myself what he would do with zero openstack context, it was this, and it fixed it.23:42
TheJuliaThat should fix it23:43
JayFit did locally.23:43
TheJulia... I thought I looked for that actually and I thought it was there23:51
TheJuliaI guess i mixed that up23:51
JayFit is in Ironic23:52
JayFwhich is the other clue that pointed me there23:52
JayFsometimes fresh eyes :D 23:52
TheJulia++23:52
JayFthat's why there are N of us :D 23:52
JayFdid you see my hilarious docs patch?23:53
JayFhttps://review.opendev.org/c/openstack/ironic/+/90600123:53
JayFWhat a great refactor of that doc, made it 100x more readable at the expense of removing /the key piece of information from the doc/23:53
JayFd'oh23:53
TheJuliaheh23:53
TheJuliac'est la vie23:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!