Friday, 2024-11-01

opendevreviewTakashi Kajinami proposed openstack/ironic-lib stable/2024.2: Replace outdated python job template  https://review.opendev.org/c/openstack/ironic-lib/+/93389800:15
zigoHi there! This isn't the first time for me: test_format_inspector.TestFormatInspectors.test_iso_udf times-out and then my Ironic package is declared unbuildable.08:38
zigoDoes one know where to control the timeout value for this test? I've looked for it, but didn't find where.08:38
masgharzigo: no idea11:07
masgharhttps://www.irccloud.com/pastebin/IHCN1bu1/11:11
masgharOh whoa how did my message turn into a pastebin xD11:12
masgharFWIW the tests/scenario directory does exist in https://github.com/openstack/ironic-tempest-plugin11:19
iurygregorythis is weird11:23
iurygregoryto me it seems tempest didn't find our plugin, but it was installed from what I saw in the logs11:23
iurygregorymaybe something changed in tempest :thinking:11:24
masgharYeah it doesnt seem like a flake either, somehow - I think if we retest it will happen again11:24
masgharPerhaps, maybe its using a different branch?11:25
iurygregoryit failed on both jobs same way...11:25
masgharYes11:25
iurygregorylet me check something here11:25
masgharWell we record quite a lot of logs, don't we 😅11:39
masgharMaybe it is a flake? https://zuul.opendev.org/t/openstack/builds?job_name=ironic-lib-bios-ipmi-src&project=openstack%2Fironic-lib&skip=011:43
iurygregoryI have the feeling is a flake11:50
iurygregorybut trying to check if tempest changed something11:50
iurygregoryohhh we are using ubuntu-jammy, I remember there was a problem after we dropped py3811:53
iurygregorymaybe is related11:53
iurygregorywe need to switch the master to the new ubuntu I think11:53
iurygregorygmann, do you think this is the case? we saw some strange issues with tempest on master...11:54
masgharI see! Perhaps it could be related12:13
iurygregoryone way we can probably check is doing a recheck in stable job that also failed12:26
masgharLets recheck and se12:26
masgharsee*12:26
iurygregorybut the failure was today...12:26
iurygregorylet me see the thread in the ML12:27
iurygregoryoh ok, the thread was [qa][all] stable/2023.1 gate is blocked (hold the recheck)12:27
iurygregoryso maybe is another issue we have12:28
masgharYeah12:28
masgharI'm trying to find the where ironic-base is: https://github.dev/openstack/ironic-lib/blob/467e7ecc18f65985cbd833687b85b717ad363183/zuul.d/ironic-lib-jobs.yaml#L3-L412:29
iurygregoryin openstack/ironic =)12:29
masgharoh I see!12:29
TheJuliaGood morning13:16
JayFzigo: it's a test that uses a lot of io and has failed similarly in our CI. I suggest patching it out. Just flaky13:18
zigoJayF: I had the same issue with other projects...13:19
TheJuliaSorry for not being around yesterday... I woke up, remembered to check if I had jury duty...13:21
TheJuliaand you can imagine me running out the door heading towards the courthouse13:22
zigoJayF: Is there anyone working on https://bugs.debian.org/1084566 ?13:24
iurygregorysame failure D: 13:45
JayFZigo: There's no effort being made right now on python 3.13 compatibility. I suspect if you file an issue with us, someone will eventually get around to it, or you're welcome to submit a patch.14:12
JayFBut that being said, I would not assume that everything is going to behave hunky dory on python 3.13 unless you do full integration testing, just because I'm sure they're going to be more eventlet shenanigans14:12
masghariurygregory: controller/logs/devstacklog.txt does say 'Successfully installed ironic-tempest-plugin-2.11.1.dev8'14:17
opendevreviewcid proposed openstack/ironic master: Save ``configdrive`` in an auxiliary table  https://review.opendev.org/c/openstack/ironic/+/93362214:39
masgharI am comparing a successful run (https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_903/933029/2/check/ironic-lib-uefi-ipmi-src/903d952/job-output.txt) with the recent failed run14:43
masghariurygregory: If you put them side by side, and search for 'TASK [run-tempest : Run Tempest]',  you will see that the successful run was using setuptools (which is deprecated)14:44
masgharThe new run has lines like 'controller | all recreate', 'controller | all installdeps:' instead14:46
opendevreviewThomas Goirand proposed openstack/ironic master: Remove the use of crypt.crypt, removed in Python 3.13  https://review.opendev.org/c/openstack/ironic/+/93391714:54
zigoIs this done right? ^14:55
zigoJayF: Yeah, I'm worried with the Eventlet in 3.13. I need this fixed by February (date of Trixie freeze...)...14:58
opendevreviewcid proposed openstack/ironic master: Save ``configdrive`` in an auxiliary table  https://review.opendev.org/c/openstack/ironic/+/93362215:13
opendevreviewMerged openstack/sushy-tools master: Remove Python 3.8 support  https://review.opendev.org/c/openstack/sushy-tools/+/93325915:18
TheJuliaJayF: replied to some of your comments on https://review.opendev.org/c/openstack/ironic-specs/+/933612 to try and bring a bit more clarity. Adam from metal3 or even cardoe might have additional thoughts to help us clarify overall vision.15:22
cardoebah yes I need to do that thank you for reminding me. I'm over here chasing yaks and shaving squirrels.15:23
TheJulia... Why would you *shave* squirrels?!15:28
TheJuliaclearly you need a fleet of cats.15:28
opendevreviewThomas Goirand proposed openstack/ironic master: Remove the use of crypt.crypt, removed in Python 3.13  https://review.opendev.org/c/openstack/ironic/+/93391715:30
TheJuliaOr Corgis.15:30
TheJuliaGranted, the Corgi's will jsut keep the Squirrels away, or cuddle them. It remains to be seen exactly what one would do15:30
opendevreviewVerification of a change to openstack/ironic-python-agent master failed: Correct invalid docstrings; s/Found/Error/  https://review.opendev.org/c/openstack/ironic-python-agent/+/91159815:38
cardoeI knew I was doing something wrong. 15:38
cardoeI also need to reply to you on the ML.15:38
cardoeSomething is up with tempest and IPA15:39
TheJuliaSo, any further though on graphical consoles? The more I think about it, the more I want to just replace console == serial console15:49
TheJuliacardoe: got a job link?15:49
cardoehttps://zuul.opendev.org/t/openstack/build/f0e3e3dc4819477bbb5e40b141aa595f is one15:50
cardoehttps://zuul.opendev.org/t/openstack/build/32fdc8fa2ac24b64b92babae105a2594 is another15:50
cardoewhen you say "console == serial console". We're saying that the word "console" will mean serial console always?15:50
cardoeor just removing the ambiguous use of "console" and always prefixing it with "serial" or "graphical"?15:51
TheJuliatoday it does15:51
TheJuliaAnd shellinabox is dead... sooooo15:51
TheJuliamaybe best to just replace the idea/meaning to be more current with graphical15:51
TheJuliawe're also dealing with a barely used 10 year old concept15:51
TheJuliahttps://f3d5be211d88562b5165-96f941636ebce386b2873cc66e81b34b.ssl.cf5.rackcdn.com/927544/5/check/ironic-tempest-ipa-partition-uefi-pxe-grub2/f0e3e3d/controller/logs/ironic-bm-logs/node-0_console_2024-10-30-19%3A22%3A11_log.txt <-- trying to uefi boot cirros is known bad idea15:55
TheJuliawe need to fix that15:55
TheJuliathat second one is *weird*16:00
TheJuliaThe second one just appears to fall over from tempest? Did we change a job name?!16:06
TheJuliaWell, a test name16:06
TheJuliait never actually tries to deploy a node16:06
TheJuliacardoe: I guess instead of do more interface work, just move past "serial"16:13
TheJuliaHas anyone heard if HPE Gen11 gear, specifically with AMD processors, takes a long time to reboot?16:17
TheJuliaNevermind... https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00128973en_us16:18
JayFTheJulia: lol17:15
JayFTheJulia: I personally saw one take ~30 minutes17:15
JayFthere's a reason we want to enable OOB steps for OEM-exposed things in redfish, we're literally paying ^^ that amount of time each cleaning/deployment to do ESKM stuff17:15
TheJuliaeek, yeah17:18
JayFrpittau: are you working on a priorities spec for this ptg?17:20
JayFrpittau: I was aobut to do prep for a quick video on it, and realized that prep is like, 50% of the work to make that doc17:20
JayFI might draft something and put it up17:21
opendevreviewJulia Kreger proposed openstack/ironic master: Implement basic interfaces for GraphicalConsole Interface  https://review.opendev.org/c/openstack/ironic/+/54735617:22
TheJuliaso ^ makes me feel like a whole interface to keep serial is just... overkill17:22
TheJuliawhen maybe serial should be stamped out17:22
TheJuliaby the robo-feature-eliminator-500017:22
JayFI am +1 to retiring serial, but less +1 to reusing the same interface names for the new stuff17:22
JayFI think we deprecated shellinabox when I was PTL17:23
TheJuliaI'm sort of entirely against putting in a new interface17:23
JayFor at least decided to, sometimes I missed the follow thru one some things (/me looks at that python-ironicclient patch for shards)17:23
TheJuliawhich is a huge shift for me17:23
JayFWhat value, other than less code churn, does reusing the same interface have? 17:23
TheJuliasimilar meaning, we can reuse the same exact webapi interface17:24
TheJuliawhich we would realistically *need* to do to keep it simple as it sort of does exactly what is needed to day anyhow17:24
TheJuliajust the driver under-the-hood is a different flow17:24
JayFooooh, so like, we avoid client work, api ref updates, etc17:24
TheJuliabbiingo17:24
JayFand shellinabox was already "we give you a link to click" style, yeah?17:24
TheJulianew field, we begin cutting a harder path for ourselves17:24
TheJuliasort of like "we can take the path already paved and do some side work" or "we must bore through this mountain to walk forward"17:25
JayFhonestly I just got whooshed for a second that interfaces are API-facing17:25
cardoeI'm with Julia on depreciate and reuse.17:25
TheJuliashellinabox was sort of that path as well17:25
TheJuliaI don't remember the exact internals17:25
cardoeIf I understand the old interface correctly. Which I think was just "here's a link to click"17:25
TheJuliabut I know hpe had a patch for ages which returned a vnc console url for nova to proxy through to17:25
cardoeand it took you to a shellinabox web page thing.17:25
JayFyeah, which is what nova does, and what we'll probably do w/a vnc url I assume17:26
TheJuliayeah17:26
JayFyep, I'm completely sold now already17:26
cardoeYeaj17:26
TheJuliacool cool, we should like solidfy this consensus with concrete17:26
JayFit seems more like the wrong decision was delineating it as **serial console** originally17:26
TheJuliayeah17:26
TheJuliaWell, I think the argument was "people want to use both"17:26
TheJuliaand the more cases I've gotten pulled into with serial console use tanking host performance17:27
JayFI've never met a person who had that as a requirement tb h17:27
TheJuliathe more I'm just "no, don't do that"17:27
JayF++17:27
JayFit's old technology grafted on, like the CSM modules in UEFI firmwares17:27
TheJuliayeah17:27
TheJuliaand even with UEFI, the host is graphical out of the box anyway17:27
JayFI still see this as like, kinda separate? We can retire shellinabox right now if we want, and just leave no implementations, and fill this one in as we go17:27
TheJuliaits not like it is a serial port on a Vax17:27
JayFyep 17:27
TheJuliayup, exactly17:28
TheJuliaand just build a new redfish interface with the backend17:28
JayFonly real update that'd need to happen is probably in docs17:28
TheJuliaa whole/complete/secure backend17:28
JayFjust s/serial// in places17:28
TheJuliaI'm not even sure that will entirely be needed because we just famed it all as "console" by default17:28
TheJuliabut yeah, we'll need to check17:28
TheJuliaI'll go start a spec revision shortly17:28
JayFsgtm17:29
JayFalso I responded on container spec17:29
JayFtl;dr: "we can just trust podman for that part" sounds a lot like "we can just trust qemu-img for that part"17:29
JayFwhich means I'd think it's slightly wise to ensure they /actually are expecting use with untrusted images/17:29
TheJuliayes, but what is k8s doing at that point since contents are inherently untrustedf17:30
JayFand maybe even document that decision so if later they try to takesies backsies we got evidence LOL17:30
JayFI don't understand what you mean with your question?17:30
JayFif there was a bug, where I could somehow craft an OCI image that when executed by podman would ... say put the contents of /etc/passwd into that container instead17:30
JayFwould they treat it as security or just tell us to stop using untrusted images17:31
TheJuliaso I guess the issue in my mind is we use a tool like podman to do the underlying retrieval of the container out of the box and then the cracking open of it. It is not like we download it and then extract it17:33
TheJuliaso if there *is* a problem with the contents of the container, everyone else using the same tooling is impacted17:33
JayFthat was all true for our qemu-img bugs17:33
TheJuliano, we knew there issues and had code to guard against some of them17:34
TheJuliabut not all of them and we didn't knew the base issue was that bad17:34
JayFYes; but everyone using that same tooling was impacted17:34
TheJuliaotherwise, were creating our own retrieval and extraction code in python17:34
TheJuliawhich means, we;re sort of building a mountain of work17:34
TheJuliaIndeed, that sometimes happens, unfortunately17:34
JayFI'm not saying "don't trust podman", I'm saying it'd be extremely nice to point to a doc that indicates they have a security policy that matches our expectation17:34
TheJuliaAHH17:35
TheJuliaokay17:35
JayFbecause I no longer trust obvious assumptions about software we use17:35
TheJuliaThat, I think makes a ton of sense17:35
TheJuliaReasonable17:35
JayFwhere if a qemu-img style bug came out, instead of us writing python to detect bad containers, the oci/podman/docker ecosystem cares enough to fix it at that level17:35
shermanmon the topic of deprecating serial console and replacing with graphical: is this assuming that all supported hardware would expose a graphical kvm interface of some kind?17:35
JayFshermanm: the only interface currently implemented for serial is using an EOL, insecure project. The only proposed *new* interface is redfish, which AIUI is specified to be VNC17:36
TheJuliashermanm: the assumption is some can, not everything *can*17:36
JayFI don't think we'd -1 another serial implementation in this interface, but I also doubt it'll make sense for server hardware17:36
TheJuliawell, kvmip which is just vnc with some byte structures flipped17:36
TheJuliaThat is my feeling as well17:36
TheJuliaw/r/t if someone had a redfish raw serial interface17:37
shermanmI ask because we have a hard requirement to have some kind of user-facing console access for a pretty wide range of hardware variants, some of which don't support redfish at all, some of which still only have terrible java kvms, etc.17:37
shermanmdoesn't need to be a serial console, but a fair chunk of our hardware won't necessarily play nice with graphical consoles directly17:37
JayFThat's a borderline-impossible requirement to do with the resources I assume you have17:37
JayFat least completely out of band17:38
JayFHell last time I tried to use one of those old java-based consoles on a drac, I think I had to break out a VM with IE6 inside17:38
JayFor something similarly ridiculous17:38
TheJuliashermanm: the secret is, they don't with serial *either* :)17:38
shermanmwell, right now we're using the serial console ipmitool sol just fine, but it sounds like that's depending on an EoL project in ironic?17:38
JayFshermanm: if you're using the shellinabox implementation, and exposing it to customers; tldr AHHHHHHHHHH17:38
TheJuliashermanm: so, there are two ways under the hood in ironic today17:39
TheJuliashellinthebox and an awful ipmitool sol proxy driver17:39
JayFhttps://github.com/shellinabox/shellinabox (last commit: 6 years ago)17:39
shermanmit's not great, but it's core functionality for us, and we've got a not completely untrusted set of users17:39
TheJuliaerr, shellinabox17:39
TheJuliathe ipmitool socat console just gives you a port you telnet to...17:39
TheJulianote I said telnet, it is *awful*17:39
JayFSometimes I'm glad I haven't read /all/ the code in Ironic. :) 17:40
shermanmit is pretty bad, but it's "something" when a user builds a kernel and breaks their boot17:40
JayFshermanm: At Rackspace, with OnMetal, we developed rescue mode for this case17:40
TheJuliayeah, something is better than nothing in many cases17:40
JayFshermanm: so that is an alternative in cases where console may not be possible, and works more universally17:40
TheJulia++, I'll note with serial port stuff today, for hardware in UEFI mode, nothing may get logged17:41
TheJuliaat least, logged to the serial port17:41
TheJuliasince, the default mode of user interaction outside of headless gear is the graphics card17:41
shermanmthis does make sense, I'll need to dig into what we have, and see if anything's truly relying on serial only, and couldn't be resolved with rescue mode. I freely admit that we've got a lot of edge cases17:44
TheJuliayeah, we're not saying we're going to smash all things serial console with a giant mallot17:45
TheJulia.... As much as I kind of want to be a loony toons character with a mallot at the moment. :)17:46
TheJuliaJust driving towards graphical supercedes the use model17:46
TheJuliaAnd that is a road17:46
TheJuliaWhich we need to still drive down17:46
TheJuliawe're in the "aruging over which route to select in google maps" phase17:46
shermanmyep! I just wanted to raise a hand sayin "8 year old weird hardware still kinda needs this crappy thing"17:51
gmanniurygregory: tempest jobs should be fixed now, please recheck17:52
JayFshermanm: well, the only plan is to kill off shellinabox afaict right now; we haven't deprecated the serial proxy piece, so I think you're in good shape. You shouldn't deploy shellinabox anyway :D 17:54
jrosserI also use the ipmi/sol + serial proxy today for arm hosts17:56
jrosserI expect that could be more modern with redfish/ssh + serial proxy in the future17:57
TheJuliajrosser: this is encouraged, but I'm not sure the upstream cores can just make it happen without some assitance17:58
shermanmJayF: thanks for that clarification, scrolling up some more I realize I didn't have the full context. We're using the ipmitool-socat interface at the moment, the the serial proxy, not shellinabox17:59
shermanmI'd been considering writing a different backend using something like goconserver instead of ipmitool17:59
TheJuliaso thinking about it18:06
TheJuliaif we *just* return a VNC URL today, that plugs into nova's vnc proxy kind of cleanly18:07
TheJuliawe *could* create a proxy of our own, but is it *really* needed except for standalone users?!18:07
TheJulia#discuss :)18:09
shermanmeven if the proxy was out-of-tree, otherwise not recommended, having an interface we could leverage to deploy / build our own would be useful18:10
TheJuliaI don't think that is anything we would block :)18:12
opendevreviewJulia Kreger proposed openstack/ironic-specs master: Move graphical vnc console interface to retired status  https://review.opendev.org/c/openstack/ironic-specs/+/93393718:29
TheJuliaokay, that ^^^ commit message is a bit verbose18:29
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: WIP: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389418:59
JayFshadower: almost all of ironic is pluggable, even if we don't technically advertise support for it (or promise any API compat)19:25
JayFer, shermanm ^^ sorry shadow19:25
JayFshermanm: e.g. https://opendev.org/openstack/ironic/src/branch/master/setup.cfg#L87 you can use the same HWM pattern from IPA to inject interfaces and hardware types (just remember you'll need a custom hw type if you want to declare compat with that interface)19:25
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389420:29
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389420:35
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389420:35
JayFanyone know WTF is up with python-ironicclient functional tests? They appear massively broken, and are nonvoting https://zuul.opendev.org/t/openstack/build/034561574cb244cf98e2cc0826187a9720:50
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389420:50
JayFwe should find a path to fix them OR remove them entirely imo20:50
JayFadamcarthur5: ^ cc20:51
TheJuliaoh my20:56
TheJuliaso, they use some testing framework from oslo which I bet got nuked20:56
TheJuliaat least, that is what comes to mind when thinking about them20:56
JayFyeah; I think adamcarthur5 is going to try and reproduce in his local devstack and see if there's a quick path to a fix20:57
TheJuliaack20:57
TheJuliaI wouldn't mind just nuking those functional tests from high orbit if it came down to it20:58
adamcarthur5You guys overhauled auth right? https://opendev.org/openstack/python-ironicclient/src/branch/master/doc/source/contributor/testing.rst#functional-testing 20:59
adamcarthur5That could be why?20:59
JayFoh, the scope changes! of course!20:59
JayFdoes that line up?!20:59
TheJuliawell, I think we made changes to fix it20:59
TheJuliabut if the underlying object is missing....20:59
JayFreleasenotes/notes/change-default-rbac-policy-f2f154043910f26a.yaml was created Date:   Mon Nov 27 13:46:34 2023 -080021:00
JayFtiming does /not/ line up21:01
JayFnot saying it means it's not a cause, but less likely21:01
adamcarthur5Yeah but if this test has been non-voting for a while, the problems are likely stacked21:01
JayF++ exactly, itamar hit that with snmp 21:01
JayFfix vpdu; ironic snmp driver broke in the interim :( 21:01
adamcarthur5Julia you mentioned nuking this, Jay, how desperate are you on actually getting to the root cause of this as part of this patch?21:02
JayFlol21:02
JayFgive it a swift kick in the rear on the way outta the repo :D 21:02
JayFjust make sure it's a separate commit from shards21:02
JayFand provide lots of context in the commit message21:02
adamcarthur5Okay. In that case, I am going to investigate for 10-15 mins to maximise context I can give21:03
JayF++ that sounds like a good idea, make sure it's not low hanging 21:03
JayFthanks!21:03
* TheJulia whispers "nuke it"21:03
* TheJulia thinks it is time to go to the post office21:04
adamcarthur5Yeah, it seems the current error is auth: AttributeError: 'BaremetalPortGroupTests' object has no attribute 'auth_ref'21:06
adamcarthur5But that seems fixable.. I will note it down.21:11
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Support /v1/shards  https://review.opendev.org/c/openstack/python-ironicclient/+/93389421:12
adamcarthur5https://review.opendev.org/c/openstack/python-ironicclient/+/908828... Huh, I was a reviewer on this change. It must have been from my codespell work21:16
adamcarthur5Tests last past here: Over a year ago https://review.opendev.org/c/openstack/python-ironicclient/+/891560. There were only doc changes + running of older commits (some from 2017) in CI21:23
adamcarthur5That might align with an auth change?21:26
opendevreviewAdam McArthur proposed openstack/python-ironicclient master: Remove Functional Tests  https://review.opendev.org/c/openstack/python-ironicclient/+/93396721:33
adamcarthur5Let me know if this is enough detail, although I do have one suspicion for a fix. Let me know 21:33
JayFlet me put it this way: they either gotta be fixed or they gotta go21:43
JayFThe question is are they providing value21:43
JayFI don't know the answer to ^ that21:43
* adamcarthur5 my suspicion was wrong21:47
opendevreviewAdam McArthur proposed openstack/ironic master: api: Introduce new mechanism for API versioning  https://review.opendev.org/c/openstack/ironic/+/92891922:22
adamcarthur5The above was tested with https://gist.github.com/Sharpz7/97356eb57f77d3ee75892791c6cab15522:24
opendevreviewAdam McArthur proposed openstack/ironic master: api: Introduce new mechanism for API versioning  https://review.opendev.org/c/openstack/ironic/+/92891922:27
adamcarthur5Hey folks, I am trying to add some tests to ironic-tempest-plugin, I am assuming it needs a devstack to test locally. Does it need to be a fresh devstack? Or is the repo self-cleaning? Is it the normal pattern to test locally?22:50
opendevreviewAdam McArthur proposed openstack/ironic-tempest-plugin master: WIP: Microversion handling test  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93397022:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!