Wednesday, 2024-02-21

opendevreviewcid proposed openstack/ironic master: Remove unsupported rpc methods  https://review.opendev.org/c/openstack/ironic/+/90926302:09
opendevreviewMerged openstack/ironic master: Ensure all errors are passed during cleaning  https://review.opendev.org/c/openstack/ironic/+/90870307:04
rpittaugood morning ironic! o/08:35
rpittaufrickler: change looks good for jsonschema bump compatibility on our side08:40
fricklerrpittau: thanks, sadly not so good/no responses from other projects so far, so likely the bump will have to be deferred to the next cycle10:53
rpittauthat's perfect for us :)10:54
opendevreviewAlex Welsh proposed openstack/bifrost master: Improve downloaded deployment image support  https://review.opendev.org/c/openstack/bifrost/+/88488811:49
opendevreviewKamil Gustab proposed openstack/sushy master: Adds Ethernet Interfaces for manager.  https://review.opendev.org/c/openstack/sushy/+/90945112:26
dkingTheJulia: So, after I started looking into it, I realize that I was just being silly. There is already a `by-path` option for root device hints. So, the work is already done.13:46
dkingIt turns out that the problem is that it just isn't implemented yet in Metal3 Bare Metal Operator, as far as I can tell.13:47
dtantsurdking: I believe it is, just in a fancy way13:48
dtantsurdking: https://github.com/metal3-io/baremetal-operator/blob/main/pkg/provisioner/ironic/devicehints/devicehints.go#L19-L2413:49
SvenKieskehey there, I don't want to push anyone, but it would be nice to get reviews for this bugfix: https://review.opendev.org/c/openstack/networking-baremetal/+/903995 as there are users out there which are already impacted. thank you! :)13:57
dkingdtantsur: Thank you. I'm seeing that as I'm trying to sift through the code. Now I'm wondering why it didn't work for us.  I'm going to try to replicate, and then I'll need to check version number. For some reason, it was searching with a `name` hint type.14:00
TheJuliagood morning14:11
SvenKieskeo/ (already 15:00 here :D )14:11
dtantsurWhy do Europeans start their day in the afternoon? :D14:14
TheJuliaMagic?!14:22
TheJuliaExcellent Coffee?!14:23
TheJuliaThe best cafes?!?14:23
SvenKieskeI for one try at least starting before 11:00 (succeeding most of the time, due to meetings) :D14:36
TheJulia0600 most days14:49
dtantsurSince my wife is doing on-site job nowadays, 0700 is not uncommon for me any more15:03
JayFI work 0700 to 1600 local time most days for similar reason - - my wife is a teacher and I start work when she leaves and then mostly done when she gets home15:42
clarkbI'm a fan of having sunlight after work16:09
JayFYou know as well as I do that being off at 4pm isn't always enough to get sunlight where we live :D 16:13
JayFI guess to that effect, there are days in the depth of winter where "grey" is basically the color all day regardless of time of day lol16:13
clarkbya it doesn't always work out16:14
clarkbI would take year round DST for this reason though. My kids already walk to school in the dark for non zero days of the year16:14
JayFHow shortsighted! You aren't considering the needs of 19th century farmers, clarkb.16:16
JayF;)16:16
rpittaugood night! o/16:33
SvenKieskeI honestly don't understand ironics policies with regards to requirements bumps, but I guess we should discuss this in the review..17:01
TheJuliawe have to be careful on stable branches specifically17:08
dtantsurSvenKieske: not really Ironic policies, they're openstack-wide17:08
TheJulia++17:08
TheJuliaAnd even then, not always possible to ensure as not all packagers respect requirements.txt17:08
dtantsurif it's about https://review.opendev.org/c/openstack/networking-baremetal/+/903995/3/requirements.txt, you need to expect the current operators to use ANYTHING above 5.29.0. While I'd not recommend anyone to go with an ancient version, they might.17:08
dtantsurI do agree that oslo libraries are kinda special since most consumers will upgrade them with the services. But they're not current exempt from the policies.17:09
TheJulia*and* the only place that can *really* ever be changed is on the master branch17:09
TheJuliawe definitely cannot backport major changes like that17:10
dtantsurSvenKieske: in situations like this, we tend to ask patch authors if the patches can be rewritten in a way that is compatible with older versions while also conditionally using newer ones17:10
* TheJulia drifts back to the island of writing documentation17:10
dtantsure.g. if you want to use CONF.oslo_messaging_rabbit.rabbit_quorum_queue which, I presume, is not available earlier, you can use getattr(..., None)17:10
TheJuliaIt is a dessert island where creme brulee is in the fridge awaiting dinner17:10
dtantsurmmmmmmmm17:11
TheJuliadtantsur: ++++ on the last two messages17:11
dtantsurSvenKieske: another important angle to look at this problem. Some operators mix and match versions. I.e. they may be using a new Ironic on an ancient OpenStack.17:12
SvenKieskeokay, so this means, that ironic/networking-baremetal does not support quorum-queues before caracal, could that then at least be documented somewhere?17:12
TheJuliaor an ironic which requires archeology anytime they ask for insight17:13
* TheJulia wonders if a good new career could be digging up dinosaur bones17:13
dtantsurSvenKieske: which exactly part requires the version bump? We may figure out a conditional way to support them.17:13
dtantsurif it's the configuration option, just use getattr17:14
SvenKieskeyeah we need someway to check if quorum queues are used, if we use olso native stuff for that this requires a bump. I think we could use some hacks to avoid the version bump17:14
dtantsurand create a release note saying something along the lines of "If you use oslo.messaging before X.Y, you're in trouble"17:15
SvenKieskestill, this technically means networking-baremetal does not, in fact support quorum queues, because it can't be tested with it, if the minimum requirements do not require the needed oslo.messaging version17:15
SvenKieske*besides the current master branch, once this is merged17:16
TheJuliaHmm, I guess the model actually differs than what we expect most operators to actually do17:16
dtantsurThe minimum requirement changes nothing if it's already satisfied. I don't quite get your argument.17:17
TheJuliasingle rabbit which comes back alive, but that doesn't work for those things which store long running state in the queue17:17
dtantsurfor instance, https://github.com/openstack/requirements/blob/stable/zed/upper-constraints.txt#L159 means that we *recommend* Zed users to have 14.0.3.17:17
* TheJulia had to go look up the history and origin behind quorum quues17:17
dtantsurSo, while, we cannot *require* that on stable/zed, chances are 90% that the operators have it. And our CI definitely does.17:18
SvenKieskewell, I was asked to bump the minimum required version, because of course it needs to be guaranteed that the oslo.messaging mechanism of looking up if quorum queues are used works.17:18
SvenKieskeso the only way networking-baremetal can guarantee that quorum queues work in a given deployment is by having a minmium version of oslo.messaging which supports quorum queues, no?17:19
dtantsurNo. Or at least I cannot make this conclusion from your patch.17:19
dtantsurI mean, for operators - yes. If they want to use this feature, they'll need the version. But *we* don't require it.17:20
SvenKieskeso, for what are minimum requirements than useful, please explain it to me :)17:20
dtantsurSvenKieske: minimum requirement is: this project cannot work without that version of the library.17:20
SvenKieske"But we don't require it" <- I don't understand this. If this patch is merged as is, we do in fact require it?17:20
dtantsurYes, but that's the reason we won't merge it on stable branches.17:21
SvenKieskeyeah and if I want to support feature foo in library bar which is available in version zed, I need to require version zed, no?17:21
dtantsurPossibly. Sometimes we do conditional dependencies. In any case, supporting a new feature generally qualifies as a feature itself and is thus not backportable.17:21
dtantsurRaising the version on the master branch is not a huge deal.17:22
TheJuliaI suspect a "fix" to handle the case of the usage is reasonable, given it is the default as of rabbitmq 3.8.0 released back in october 201917:22
dtantsurThere is a fine border between "I want to support a new feature" and "your project misbehaves in the presence of the new feature".17:22
dtantsurI was reading your patch as fixing something like the latter. If it's more the former, let's merge it (and NOT backport).17:23
SvenKieskethat was not my point. my point was to ask to document - on older branches - that networking-baremetal does not support quorum queues, because that's what users actually deploy and want to use and run into bugs because nobody bumps the requirements of networking-baremetal.17:23
TheJuliathey renamed what was previously used as "classic" queues17:23
TheJuliaso the opportunity for confusion there is *huge*17:23
dtantsurSvenKieske: then why don't document that on literally every openstack project that did not bump oslo.messaging?17:23
dtantsurnone of them will have quorum queues if oslo.message is old. but that belongs in the oslo.messages docs, no?17:24
SvenKieskedtantsur: it would be good to do! I'm only aware of networking-baremetal until now :)17:24
dtantsurEverything uses or can use oslo.messaging17:24
dtantsurNeutron has old oslo.messaging requirement https://opendev.org/openstack/neutron/src/branch/master/requirements.txt#L3417:25
SvenKieskewell all I'm trying to say is it would be nice if requirements would be bumped more often so users can actually use 2 year old "new" features. I get why requirements don't get bumped and I appreciate being able to also use older lib versions, but I don't think the current balance is a good one to be honest :)17:26
dtantsurSvenKieske: I think you're mixing two things. Let's bump the requirement on master - well, you more or less convinced me. But there seems to be an actual bug fix in networking-baremetal (respecting the newer option), and I'd rather not tie it with the requirements bump.17:26
SvenKieskedtantsur: okay, fine with me. I only did bundle it because another reviewer requested it in there and I'm only a drive-by contributor. So I really rely on the expertise of all the people who know more than me :)17:28
dtantsurSvenKieske: sure, and we're very grateful for your contributions! Just trying to balance a lot of needs.17:28
SvenKieskeI really want to backport the actual fix, because people are running into this bug even on the zed release.17:28
dtantsurI'd say: go with getattr() and no bump, backport that, then follow up with a version bump on master only.17:29
dtantsurI'm saying that fully understanding that it's slightly more work for you.. Let us know if you want us to take over the 2nd part.17:29
SvenKieskeso, should I rewrite the detection logic of quorum queues to not use a newer oslo messaging version? I would need to think about how to do that in a not very ugly way :D17:29
SvenKieskemhm, do you have maybe an example for this getattr() stuff somewhere? I'm fine with adapting but I don't know how to do it without an example I guess17:30
SvenKieskeI guess I can figure it out myself also, but a link would be handy :)17:30
dtantsurSvenKieske: by detection, you mean the configuration option?17:31
SvenKieskedtantsur: yes; I guess something like this? https://opendev.org/openstack/oslo.config/src/commit/305ecd817836a90a8f5e495cbf6a770dcea1f7e8/tests/test_cfg.py#L215917:32
dtantsurSvenKieske: yes. A complete example https://review.opendev.org/c/openstack/networking-baremetal/+/903995/3/networking_baremetal/agent/ironic_neutron_agent.py17:33
dtantsuractually, a small correction, hold on17:33
dtantsuryep, now look17:34
SvenKieskethanks for your time already, much appreciated.17:34
SvenKieskewill adjust the patch accordingly17:34
dtantsurthank you!17:34
SvenKieskemhm17:38
SvenKieskewhat about the test though? https://review.opendev.org/c/openstack/networking-baremetal/+/903995/3/networking_baremetal/tests/unit/ironic_agent/test_ironic_agent.py17:38
SvenKieskecan I settattr that (is that even a thing) and older oslo releases just print a warning?17:39
dtantsurSvenKieske: what's the default value of this option?17:40
opendevreviewSven Kieske proposed openstack/networking-baremetal master: don't force amqp_auto_delete for quorum queues  https://review.opendev.org/c/openstack/networking-baremetal/+/90399517:40
SvenKieskedtantsur: good question! it is actually "False" according to oslo.messaging docs, so no need to set it!17:43
dtantsurproblem solved :)17:44
opendevreviewSven Kieske proposed openstack/networking-baremetal master: don't force amqp_auto_delete for quorum queues  https://review.opendev.org/c/openstack/networking-baremetal/+/90399517:47
SvenKieskethanks again all, and sorry for being stubborn :D17:48
dtantsurall good, I'm glad we all got better understanding of what is going on17:49
JayFDoes anyone have knowledge about ironic-standalone-redfish tests?17:49
dtantsurnow, nearly 7pm on our side of the Atlantic, Feierabend time17:49
JayFI've found something whihc appears so broken I am stunned, which implies to me I'm missing something entirely17:49
JayFTheJulia: ^ I know you've been looking at some of this17:49
SvenKieskethinking about it, I guess I can propose a second patch bumping the requirements and getting rid of the getattr stuff then in master at least, would that be something nice to have?17:49
dtantsurSvenKieske: it would. Just explain what is going on in the commit message.17:50
TheJuliaJayF: whats up, please bear with me I'm deep in writing how to adopt an ironic deployment documentation, so responses will be delayed17:50
SvenKieskedtantsur: sure, probably will take until tomorrow, as you rightly said, it's getting late here :)17:50
dtantsurJayF: feel free to pick my brain tomorrow your morning17:50
JayFTheJulia: 2024-02-21 03:05:38.832137 | controller | +++ /opt/stack/ironic/devstack/lib/ironic:enroll_nodes:2619 :   openstack --os-cloud devstack-system-admin baremetal node create --chassis e77c9ef2-ed44-4c22-a6fa-48059bf561cb --driver redfish --name node-3 --resource-class baremetal --property cpu_arch=x86_64 --property capabilities=boot_mode:uefi --driver-info redfish_address=http://10.208.225.208:9132 --driver-info 18:01
JayFredfish_username=admin --driver-info redfish_password=password --driver-info redfish_system_id=/redfish/v1/Systems/node-0 --driver-info redfish_system_id=/redfish/v1/Systems/node-1 --driver-info redfish_system_id=/redfish/v1/Systems/node-2 --driver-info redfish_system_id=/redfish/v1/Systems/node-3 -f value -c uuid18:01
JayFhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e3d/909263/3/check/ironic-standalone-redfish/e3d3d8a/job-output.txt18:01
JayFwe are setting redfish_sysem_id 3 separate times (?)18:01
JayFmy hypothesis is that we're relying on the behavior to be the last value wins18:08
JayFwhich means this works 18:08
JayFhypothesis confirmed in logs18:10
TheJuliaeww, that is likely a bug in the creation logic where the value is just getting appended18:10
JayFwe've been relying on this for 8 years.18:10
TheJulia\o/18:10
JayFwe should bug this, right?18:11
JayFbecause even if it works, it's terrible?18:11
TheJuliathat is awful18:13
TheJuliaAnd yeah, we should fix the logic in the plugin18:13
JayFhttps://bugs.launchpad.net/ironic/+bug/205459718:26
JayFcid said he'll pick this up soon18:26
cidJayF: informative session :D18:31
cidI'm looking forward to enjoying combing through logs like you18:31
JayFyou only think you are LOL18:31
JayFreading copious amounts of debug logging is a big part of development work, especially systems development, but I'm not sure I've ever looked forward to it LOL18:32
JayFI like INFO level logs when everything is going fine, just the services telling you about all the wonderful things they are doing LOL18:32
clarkbeventually you get really good at using grep18:33
clarkbits probably the one tool I don't have to look at the manpage periodically to remember flags for because I use it too often18:34
JayFI've started to get a HUGE affinity for `rg`18:34
JayFmore useful in dev scenarios18:34
clarkbisn't ripgrep largely just faster? Looks like it also respects .gitignore18:36
cidthis reassignment is an error LOL https://bugs.launchpad.net/ironic/+bug/162842218:36
cidMy bad, I was just asking what happens to hanging issues whose whose fix have been provide and even release.18:36
cidJayF: I did not work on that, it was adamcarthur who did18:37
JayFwhoooop18:39
JayFlol18:39
JayFfixed :D 18:39
JayFclarkb: it's faster, respects .gitignore and ignores some other common stuff18:39
JayFclarkb: but the "it's faster" is almost an understatement18:39
JayFit's absurdly fast18:39
clarkbJayF: I guess i never really notice that grep is slow and I'm often grepping in many gigabytes of data18:40
JayF`git grep` vs `rg` in an openstack repo has a noticible speedup18:41
* JayF does a test to see if it's just placebo18:41
clarkbkinda like how people pick text editors because they are faster. Never really noticed that vim was slow and need to neovim18:41
JayFless than half the time18:41
clarkbya but is that 200ms vs 100ms?18:42
clarkbbecause that doesn't matter :P18:42
JayFpretty much yeah :D 18:42
JayFI bet gentoo repo is where I felt the difference when I switched18:42
JayFbecause that repo will make any computer weep18:42
JayFyeah, that was .08 -> .42 seconds (rg vs grep)18:44
JayFmaybe doesn't matter bit it does feel better18:44
TheJuliadoes anyone have a running ironic deployment handy?19:00
JayFWhat do you need?19:00
* JayF can see how his devstack is doin'19:00
JayFactually probably not good since I had bad ram in that system19:00
TheJuliadoh19:00
TheJuliadoes the following work: openstack baremetal node list -c uuid -f value --owner None19:01
JayFcid: oh, I forgot to mention in our office hours: we have VMs coming for you. Two. One you can use to devstack and one you can use to do tests/etc without beating up on your local machine19:02
JayFTheJulia: wow that is not what Iexpected19:03
TheJuliai suspect that does not work19:04
JayFhttps://usercontent.irccloud-cdn.com/file/vdug9OxP/image.png19:04
TheJuliaUUID, gah19:04
JayFthat returns a null set19:04
TheJuliaand I take it you have no owner set?19:04
TheJuliawell, your devstack-admin at the moment19:04
TheJuliaeh, should still match only what it can see19:05
JayFthat all have an owner set(!)19:05
JayFI wasn't expecting that19:05
TheJuliamuahahahahahahaha19:06
JayFlet me unset on a couple ,this box is throwaway19:06
TheJuliaack, thanks19:07
TheJulialooks like it does a literal match, so we're expecting string input int he code19:07
JayFempty set19:07
TheJuliaquestion sort of falls to sqlalchemy at the end of the day19:07
TheJuliaugh, kind of what I suspected19:07
JayFthis is the root cause19:07
JayFof why I added ?sharded= ?unsharded=19:07
JayFbecause hooking up shard=None to filtering was a giant can of worms19:07
TheJuliayeah19:08
TheJuliaout of curisoity, if you ask for json output and ask for all nodes by UUID,Owner, what do you get?19:08
JayFBWAHAHAHA19:09
TheJuliawith field UUID,Owner19:09
JayFTheJulia: how about I give you a working command19:09
JayFhttps://usercontent.irccloud-cdn.com/file/6GfDqTgl/image.png19:09
TheJuliathat would be epic, but I think I'm going to need to filter it in jq19:09
TheJuliaoh, awesome!19:09
JayFs/None/\'\'//19:09
TheJuliaperfect, thanks!19:09
JayFyou'll pay me back helping figure out why cid's job failed ironic-standalone-redfish if we aren't saved by the recheck ;) 19:09
TheJuliasure19:10
JayFI think then I'll be down to only owing you a few dozen favors :P19:10
JayFactually, wait, lemme set an owner on one of these19:10
JayFand ensure it's filtered19:10
JayFbefore I update the ledger :P 19:10
opendevreviewAdam McArthur proposed openstack/ironic-python-agent master: WIP Adding support for viewing individual cpu process info  https://review.opendev.org/c/openstack/ironic-python-agent/+/90934619:10
JayFTheJulia: uh19:11
JayFTheJulia: that just doesn't filter at all, it seems :( 19:11
TheJuliaoh, le sigh19:11
JayFhttps://usercontent.irccloud-cdn.com/file/yjx7O6kl/image.png19:11
JayFTheJulia: time to implement ?unowned=19:12
JayFalthough I suspect if you're trying to solve a thing, that won't save you today19:12
TheJuliai need it for the docs, if you can just dump the two columns via json, I'll do it in jq19:12
TheJuliayeah, it won't19:12
TheJuliaopenstack baremetal node list -c UUID,Owner -f json19:13
JayFhttps://usercontent.irccloud-cdn.com/file/7NDGAAyt/image.png19:13
TheJuliamuchas gracias!19:13
JayFseriously though19:13
JayFwe need unowned queries to work19:13
JayFand this is a straightforward API improvement19:14
TheJuliaoh, how annoying, it uses the translated column name19:14
JayFit should take both19:14
JayF*should*19:14
JayFnot that it does19:14
JayFbut that it should19:14
JayFfrom a value judgement not specification perspective I mean :D 19:14
cidJayF: I will really love that. in the wait mode now19:15
cidrecheck is failing 19:16
TheJulia"cat blah.json |jq .[].UUID -r" == win19:19
JayF that doesn't filter for the owner you want?19:21
JayFor you just need owner:uuid mappings?19:21
TheJulianah, not yet19:21
TheJuliatrying to get it to filter now19:21
TheJuliahaving to look at jq docs for that19:21
TheJuliacat blah.json |jq -r '.[] | select(.Owner == null) | .UUID'19:28
opendevreviewAdam McArthur proposed openstack/ironic-python-agent master: WIP Adding support for viewing individual cpu process info  https://review.opendev.org/c/openstack/ironic-python-agent/+/90934620:03
JayFurgh https://review.opendev.org/c/openstack/ironic/+/909263 failed the same job in the sameish way20:29
TheJuliaansible driver eh21:14
TheJuliathat is weird21:24
JayFOoh, zuul ansible bump was recentish iirc?22:04
clarkbsaturday ish we dropped ansible 6 and added ansible 8. Everything has been asnsible 8 by default for some months now and I sent notice of this change to the service-announce list last week22:09
JayFAck so unlikely cause there i suspect22:15
opendevreviewSteve Baker proposed openstack/sushy-tools master: Add virtual-media-boot to openstack driver  https://review.opendev.org/c/openstack/sushy-tools/+/90676822:18

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!