| clarkb | TheJulia: the regex thing is just a warning | 00:13 |
|---|---|---|
| clarkb | it shouldn't prevent any jobs from running | 00:13 |
| TheJulia | Interesting, since I expected another job to be running and it was not. Could have been unrelated | 00:14 |
| * TheJulia shrugs | 00:14 | |
| hjensas | TheJulia: I think might be failing because OVN DHCPv6 does not reply to RELEASE. It goes release -> release in loop and then fails on OVN. With dnsmasq there is a release success reply and then TFTP transfer starts. | 01:10 |
| TheJulia | Joy :( | 01:21 |
| TheJulia | Well, at least it is progress…. | 01:22 |
| hjensas | yes, will try to find the code that does it in edk2 tomorrow so we have something to point ovn folks at. | 01:26 |
| ashinclouds[m] | I think I figured out the grub thing. Sort of, kind of | 01:27 |
| TheJulia | Err, yeah, sort of kind of… likely a reason for job failures with different host_ip defaults on job runs | 01:27 |
| TheJulia | Creating by cloud variablity | 01:27 |
| opendevreview | Verification of a change to openstack/ironic master failed: [CI] Unblock CI by fixing job regex and non-voting snmp https://review.opendev.org/c/openstack/ironic/+/893992 | 03:12 |
| opendevreview | Steve Baker proposed openstack/ironic master: [CI] Unblock CI by fixing job regex and non-voting snmp https://review.opendev.org/c/openstack/ironic/+/893992 | 03:58 |
| stevebaker[m] | I updated ^^ but I really shouldn't self-approve | 03:59 |
| TheJulia | +2’ed | 04:01 |
| dtantsur | good morning Ironic | 06:07 |
| janders | welcome back dtantsur o/ | 07:07 |
| adam-metal3 | Hello Ironic! FYI related to the database lock issue, Sunnat from our end started to run a huge number of tests on PR that builds on dtantsur's debug PR https://github.com/metal3-io/metal3-dev-env/pull/1290 and managed to reproduce the issue again, this is also documented under the "Flake issue" https://github.com/metal3-io/cluster-api-provider-metal3/issues/1082 | 07:25 |
| dtantsur | Thanks adam-metal3! | 07:25 |
| * dtantsur is just back from PTO | 07:26 | |
| adam-metal3 | And I have an question also, can you suggest me a version or just commit hash of Ironic that is recommended to use to avoid the lock issue? | 07:27 |
| opendevreview | Merged openstack/ironic master: [CI] Unblock CI by fixing job regex and non-voting snmp https://review.opendev.org/c/openstack/ironic/+/893992 | 07:28 |
| dtantsur | adam-metal3, it all went sideways when we removed autocommit from the database.. short of reverting that (which TheJulia won't recommend), there was no rock solid version. | 07:32 |
| dtantsur | the best I have for you is to use mariadb :( | 07:32 |
| * dtantsur has some hopes for https://github.com/metal3-io/ironic-image/pull/430#issuecomment-1709607070 | 07:34 | |
| adam-metal3 | dtantsur, thanks for the info I will then communicate this internally | 07:37 |
| opendevreview | Rafal Lewandowski proposed openstack/bifrost stable/xena: Fix for lack of log rotation in Bifrost https://review.opendev.org/c/openstack/bifrost/+/893946 | 07:43 |
| opendevreview | Rafal Lewandowski proposed openstack/bifrost stable/yoga: Fix for lack of log rotation in Bifrost https://review.opendev.org/c/openstack/bifrost/+/893947 | 07:45 |
| opendevreview | Rafal Lewandowski proposed openstack/bifrost stable/wallaby: Fix for lack of log rotation in Bifrost https://review.opendev.org/c/openstack/bifrost/+/893948 | 07:45 |
| opendevreview | Rafal Lewandowski proposed openstack/bifrost stable/2023.1: Fix for lack of log rotation in Bifrost https://review.opendev.org/c/openstack/bifrost/+/893949 | 07:46 |
| opendevreview | Rafal Lewandowski proposed openstack/bifrost stable/zed: Fix for lack of log rotation in Bifrost https://review.opendev.org/c/openstack/bifrost/+/893950 | 07:46 |
| dtantsur | TheJulia: the next wild guess: the sqlite problem happens during checkpointing https://sqlite.org/wal.html | 07:58 |
| dtantsur | I mean, https://sqlite.org/wal.html#checkpointing | 07:58 |
| opendevreview | likui proposed openstack/ironic master: CI: Remove ubuntu focal job https://review.opendev.org/c/openstack/ironic/+/894014 | 08:01 |
| opendevreview | likui proposed openstack/ironic-inspector master: CI: Remove ubuntu focal job https://review.opendev.org/c/openstack/ironic-inspector/+/894015 | 08:07 |
| opendevreview | likui proposed openstack/ironic-prometheus-exporter master: CI: Remove ubuntu focal job https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/894016 | 08:10 |
| dtantsur | TheJulia, I cannot believe I'm saying this, but I guess our only option is to temporary return autocommit for sqlite and deprecate its support.. meaning, phasing it out in Metal3 too (cc adam-metal3) | 08:40 |
| dtantsur | I've looked everywhere, and I don't see anything wrong with anything we're doing in the database | 08:40 |
| opendevreview | Takashi Kajinami proposed openstack/ironic-ui master: Fix release note build https://review.opendev.org/c/openstack/ironic-ui/+/894164 | 10:22 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Add inspection hooks https://review.opendev.org/c/openstack/ironic/+/893533 | 10:41 |
| ravlew | Good morning ironic | 12:10 |
| ravlew | I'm getting this error on Zuul jobs, "Unable to freeze job graph: The nodeset "fedora-latest" was not found" | 12:11 |
| ravlew | I'm guessing something has gone awry in CI | 12:11 |
| dtantsur | ravlew, yeah, we need to drop the Fedora CI jobs. I've started it, but then left for PTO. | 12:14 |
| dtantsur | https://review.opendev.org/q/Ie4e67491a62a4c6cb4ca9d6e700cfc226a121235 | 12:14 |
| dtantsur | yoga? | 12:15 |
| ravlew | dtantsur yes, I've only encountered it on yoga | 12:15 |
| dtantsur | I've rechecked the fix, hopefully it'll pass | 12:16 |
| ravlew | thanks dtantsur :) | 12:16 |
| TheJulia | dtantsur: let me caffeinate and then let’s talk db | 13:12 |
| dtantsur | don't forget whiskey in your coffee then :D (also good morning) | 13:13 |
| TheJulia | no whiskey, I have things I need to get done today since I fly out this time tomorrow morning | 13:17 |
| TheJulia | https://meet.google.com/yop-quoe-uyf <-- for discussion of sqlite | 13:28 |
| * TheJulia caffinates | 13:30 | |
| TheJulia | dtantsur: ^ | 13:30 |
| TheJulia | And anyone else interested in sqlite | 13:30 |
| TheJulia | adam-metal3: might be of interest to you as well | 13:30 |
| adam-metal3 | TheJulia, thanks! | 13:30 |
| * TheJulia wonders if whiskey is being poured into coffee mugs around the world | 13:32 | |
| TheJulia | dtantsur: maybe just peel back the bifrost jobs on yoga at this point? | 13:33 |
| TheJulia | Maybe it is too early to talk | 13:37 |
| * TheJulia sips coffee in an empty room | 13:37 | |
| hjensas | TheJulia: fyi, before I get rest - https://bugs.launchpad.net/ironic/+bug/2034684 to track the OVN IPv6 issue. I opened a Bz and talked to slaweq and lucas about it as well. | 13:53 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: DNM test new SQLA https://review.opendev.org/c/openstack/ironic/+/894198 | 14:12 |
| opendevreview | Julia Kreger proposed openstack/ironic master: [metal3|db] Provide a path to use autocommit, for now https://review.opendev.org/c/openstack/ironic/+/894200 | 14:41 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix two places that can cause issues under SQLite https://review.opendev.org/c/openstack/ironic/+/894204 | 15:04 |
| dtantsur | TheJulia ^^^ | 15:04 |
| TheJulia | quickish review: https://review.opendev.org/c/openstack/ironic/+/893112 | 15:43 |
| TheJulia | unrelated to DBs, but MTU's, or or habit of not appropriately pinning | 15:43 |
| TheJulia | additionally related to networking: https://review.opendev.org/c/openstack/ironic/+/893286 | 15:45 |
| opendevreview | Julia Kreger proposed openstack/bifrost stable/yoga: Remove Fedora from the CI https://review.opendev.org/c/openstack/bifrost/+/893754 | 16:30 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/2023.1: Set role default metalsmith_provisioning_timeout https://review.opendev.org/c/openstack/metalsmith/+/893955 | 16:48 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/zed: Set role default metalsmith_provisioning_timeout https://review.opendev.org/c/openstack/metalsmith/+/893956 | 16:48 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/yoga: Set role default metalsmith_provisioning_timeout https://review.opendev.org/c/openstack/metalsmith/+/893957 | 16:49 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/xena: Set role default metalsmith_provisioning_timeout https://review.opendev.org/c/openstack/metalsmith/+/893958 | 16:49 |
| opendevreview | Julia Kreger proposed openstack/metalsmith stable/wallaby: Set role default metalsmith_provisioning_timeout https://review.opendev.org/c/openstack/metalsmith/+/893959 | 16:49 |
| dtantsur | TheJulia, I've put the 2nd +2 on https://review.opendev.org/c/openstack/ironic/+/893112 but not approved since you need to rebase it | 16:52 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Fix our internal MTU settings https://review.opendev.org/c/openstack/ironic/+/893112 | 16:53 |
| TheJulia | Thanks, the parent is still doing it's thing | 16:54 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: CI: Fix PXE Ananconda cleanup test https://review.opendev.org/c/openstack/ironic/+/894212 | 17:20 |
| TheJulia | ^^^ silly fix with running unit tests without having dnsmasq installed locally :) | 17:23 |
| opendevreview | Merged openstack/ironic master: PXE: Remove DHCP option 210 from being set https://review.opendev.org/c/openstack/ironic/+/893286 | 18:33 |
| JayF | TheJulia: Do we have a decent doc for parent nodes I can put in the cycle highlights? | 19:29 |
| JayF | TheJulia: I'm not sure if I'm unable to find it or if all we have is the API doc | 19:29 |
| TheJulia | /me blinks | 19:30 |
| TheJulia | I thought I wrote stuff | 19:30 |
| JayF | there's a nonzero chance I can't find it | 19:37 |
| JayF | a very large chance | 19:37 |
| TheJulia | heh | 19:40 |
| TheJulia | so, helps if I *actually* have the right branch checked out | 19:42 |
| TheJulia | https://docs.openstack.org/ironic/latest/admin/cleaning.html#parent-nodes | 19:43 |
| TheJulia | and https://docs.openstack.org/ironic/latest/admin/cleaning.html#child-node-clean-step-execution | 19:43 |
| mmalchuk | good evening folks | 19:44 |
| mmalchuk | JayF didn't see your answer about https://review.opendev.org/q/hashtag:ironic-week-prio+status:open+topic:extend-dib-checksum | 19:45 |
| mmalchuk | is any chance to get them merged? | 19:45 |
| opendevreview | Julia Kreger proposed openstack/ironic master: [DOC] Add a little more clarity regarding parent/child nodes https://review.opendev.org/c/openstack/ironic/+/894222 | 19:52 |
| * TheJulia goes and starts some laundry | 19:56 | |
| JayF | mmalchuk: going to be honest, I was on the fence about backporting that extend-dib-checksum change. I didn't block it but I don't intend on +2'ing it either. | 20:00 |
| mmalchuk | why? we merged dependend backports | 20:08 |
| mmalchuk | really curious | 20:09 |
| TheJulia | they can be construed as features | 20:09 |
| mmalchuk | we merged dependend changes - they also features | 20:09 |
| mmalchuk | double standards? | 20:10 |
| JayF | DIB_CHECKSUM has been supported in diskimage-builder for a long time | 20:10 |
| samcat116 | Is there anyway for ironic to use separate subnets for BMCs (for redfish virtual media) vs what the IPA uses to reach back out to ironic to download the user image? trying to use network_data here to set this up on bonds, so no dhcp here. The node running ironic has interfaces on both of these networks, but it seems like ironic hardcodes IPs of the main network its on | 20:10 |
| JayF | so it can be seen (as I do) as a bug that we never supported it | 20:10 |
| JayF | but AIUI the new hashing algos are *only* in master DIB, yeah? | 20:10 |
| JayF | so we'd be backporting a feature not present in those older versions of DIB | 20:10 |
| mmalchuk | DIB not related, these changes doesn't depend on DIB | 20:10 |
| JayF | It's related in the sense that this is a tool that depends on dib, uses a lot of DIB functionality and configuration | 20:11 |
| TheJulia | its environment variable based and independent, really | 20:11 |
| JayF | so it's reasonable for a user to assume support for common, DIB-namespaced things like DIB_CHECKSUM work roughly the same across them | 20:11 |
| mmalchuk | no... the save variable... the same behaviour... but not dependend | 20:11 |
| JayF | so in xena-release of diskimage-builder, DIB_CHECKSUM supported e.g. sha256? | 20:12 |
| clarkb | yes, it always did sha256 and md5 | 20:12 |
| clarkb | it was a binary flag | 20:12 |
| JayF | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892969/1/dib/ironic-ramdisk-base/cleanup.d/99-ramdisk-create#91 so the only real "new" thing this introduces then | 20:12 |
| mmalchuk | anywey https://review.opendev.org/q/I6128fbad5d9db2ea6469f196f3ee13b6fab76837 - is a feature which was merged | 20:12 |
| JayF | is the ability to set it to md5 or sha256 | 20:12 |
| JayF | I misunderstood | 20:12 |
| clarkb | also note that there isn't a xena dib | 20:13 |
| JayF | I know, your comment saved me from having to do release date math to go back to that era of dib :) | 20:13 |
| JayF | mmalchuk: with the new info, I will +2A the change | 20:13 |
| * mmalchuk confused | 20:14 | |
| mmalchuk | anyway thanks | 20:14 |
| clarkb | instead dib tries really hard to be backward compatible with the expectation that dib today will operate much the same way as dib several years ago with new support for new distros and continued support for old distro releases | 20:15 |
| clarkb | there have been a small number of notable exceptions to that rule (dropping python2 was one) but in general this has held | 20:15 |
| JayF | that makes sense | 20:16 |
| JayF | part of why this is so confusing to me is how IPA-b is structured | 20:17 |
| JayF | Bluntly, as both an operator and a developer, I never understood why it exists as anything other than "more dib elements" | 20:17 |
| JayF | and when I used it downstream, I always just used the elements but called diskimage-builder directly | 20:17 |
| TheJulia | Separated so we can patch stuff based upon version where we were otherwise blocked with dub | 20:17 |
| TheJulia | Dib | 20:17 |
| JayF | ooooh so it's more about having a version that's in the cycle with ironic | 20:18 |
| JayF | that can be mostly relied on to build a ramdisk for that cycle | 20:18 |
| TheJulia | Bingo | 20:18 |
| TheJulia | Yup | 20:18 |
| mmalchuk | lol | 20:18 |
| JayF | mmalchuk: fwiw, never take me not voting +[anything] on your change as a negative vote; in this case it was more an indication that I was unsure, didn't fully understand, and bluntly I've been pretty busy so it's easy to kick those down the priority list :) | 20:19 |
| TheJulia | Also, we are all pulled in many many many directions | 20:19 |
| JayF | (and you handled it right by being persistent) | 20:19 |
| mmalchuk | JayF you always can ask question | 20:20 |
| JayF | what TheJulia said is relevant here | 20:20 |
| JayF | the null vote is indicative that I kept bumping it because I was pulled in other directions :P | 20:21 |
| mmalchuk | JayF I'm always here, use bouncer, so don't hesitate to contact me | 20:23 |
| JayF | You have done everything right; it's solely a question of hours in the day and a todo list that gets longer rather than shorter each of those days :D | 20:23 |
| mmalchuk | yep | 20:24 |
| JayF | which means, rightly or wrongly, sometimes when I'm reviewing and I hit a question I move to the next one and "will come back to it eventually" | 20:24 |
| JayF | but I have a saying about technology and the word "eventually" | 20:24 |
| JayF | there is no eventually. it never comes. eventually is never. | 20:24 |
| JayF | eventually is that ticket at the bottom of your backlog from 28 months ago that nobody is ever going to solve :) | 20:24 |
| mmalchuk | never say never again) | 20:24 |
| TheJulia | and we are never our best own interrupt drivers | 20:25 |
| JayF | I am using the "squirrel" interrupt queuing system | 20:25 |
| JayF | which is not the best way to organize work | 20:25 |
| JayF | (that means whatever got my attention last probably gets to keep it) | 20:25 |
| TheJulia | SQUIRREL! | 20:25 |
| JayF | where?!?! | 20:25 |
| mmalchuk | I'm didn't use any system) it all in my mind) | 20:26 |
| JayF | TheJulia: https://www.youtube.com/watch?v=y8Kyi0WNg40 | 20:26 |
| TheJulia | Lol | 20:26 |
| JayF | mmalchuk: my mind has been swapping to disk since about three months into 2020 ;) | 20:26 |
| TheJulia | JayF: have you tried ballooning ? | 20:27 |
| mmalchuk | covid di it> | 20:27 |
| mmalchuk | covid di it? | 20:27 |
| TheJulia | Hot air balloons are fun! | 20:27 |
| JayF | I mean, the real answer is just that I've always had a bad working memory and getting older doesn't fix that | 20:27 |
| JayF | but the endless cycle of shenanigans just makes me feel like I get older faster :) | 20:27 |
| mmalchuk | buy new memory))) | 20:28 |
| JayF | TheJulia: My wife is deathy afraid of heights. I don't like them much either ;) | 20:28 |
| JayF | TheJulia: We should go on a ship again though. We both like the water. | 20:28 |
| TheJulia | Neither do I, but I did it! | 20:28 |
| TheJulia | My last on a boat involved a submarine | 20:29 |
| TheJulia | And now that sub is no more :( | 20:29 |
| JayF | :( :( | 20:30 |
| TheJulia | At least the banyon tree survived | 20:30 |
| JayF | I think the last time I was on a boat with Vanessa was a few years back, they were doing free boat tours of the Port of Tacoma as some kinda anniversary celebration. Basically one of those "sightseeing cruises" except instead of seeing normal stuff, it's like "that's a giant awesome crane" and "that's another giant awesome crane with a giant awesome ship" :D | 20:31 |
| TheJulia | “And there is an LR11000! It can lift 98 tons!” | 20:33 |
| clarkb | I'll be on a boat wednesday | 20:36 |
| JayF | nice | 20:42 |
| samcat116 | for anyone following along from yesterday, firmware updates fixed the redfish-virtual-media issue on Lenovo XCC and it seems to be working fine now. Only gotcha is that if try and mount a url that the bmc can't access (like if ironic is misconfigured to use a different interface/network), then the BMC gives a very misleading 500 error | 20:53 |
| TheJulia | At least it raises an error :) | 20:54 |
| samcat116 | Should do the published DIB images support the network-data configuration or do I need to build my own to include simple-init as described here: https://docs.openstack.org/ironic/latest/admin/dhcp-less.html | 21:04 |
| TheJulia | ... I don't know | 21:08 |
| TheJulia | I think you need to build your own in that case | 21:08 |
| JayF | iurygregory: I hate to put it so directly, but https://review.opendev.org/c/openstack/ironic/+/885425 needs attention ASAP if we want it to make the release, and firmware interface is not super useful without an implementation. | 21:10 |
| JayF | iurygregory: I'm writing cycle highlights as if that's merged, please help keep me from lying :) | 21:11 |
| JayF | TheJulia: rpittau: iurygregory: dtantsur: stevebaker[m]: other Ironic contributors; please make time to review the cycle highlights: https://review.opendev.org/c/openstack/releases/+/894228 -- please review for correctness as well! You're going to know better than me if I over-summarized your feature :D | 21:16 |
| JayF | proxying for zigo from -neutron: networking-baremetal appears to be broken against new neutron-lib -> AttributeError: module 'neutron_lib.db.model_base' has no attribute 'HasProjectPrimaryKeyIndex' | 21:19 |
| JayF | zigo: can you file a bug with the full details and we can look? likely have a CI problem too if we didn't see this in CI | 21:19 |
| zigo | Yeah, that's during test discovery in my bookworm backport build box ... | 21:20 |
| JayF | https://github.com/openstack/neutron-lib/commit/673e48a1890c721654ce0de9cd9e0897c791bd6a | 21:20 |
| JayF | zigo: you filing an LP bug on this for us? | 21:21 |
| zigo | Well, just do: | 21:21 |
| zigo | PYTHON=python3.11 stestr run --parallel --subunit networking_baremetal.tests.* | 21:21 |
| zigo | with the latest neutron-lib and you'll get that failure/// | 21:21 |
| zigo | Ok, filling it... | 21:21 |
| JayF | thanks, it's helpful to have it tracked and I don't wanna mispaste something outta IRC :) | 21:21 |
| zigo | JayF: https://bugs.launchpad.net/networking-baremetal/+bug/2034761 | 21:28 |
| JayF | zigo: interesting | 21:29 |
| JayF | > File "/usr/lib/python3/dist-packages/neutron/db/qos/models.py", line 135, in <module> | 21:29 |
| JayF | model_base.HasProjectPrimaryKeyIndex): | 21:29 |
| JayF | are you sure you're using a correct version of neutron? | 21:29 |
| JayF | it looks like we don't call that code directly, neutron does | 21:29 |
| zigo | It's probably building with the antelope version of neutron since I didn't package the b3. | 21:30 |
| zigo | Though it built fine in unstable/experimental with that version, so... | 21:30 |
| JayF | this is a neutron bug in master anyway, if that's in neutron-lib 2023.2 | 21:30 |
| JayF | https://github.com/openstack/neutron/blob/master/neutron/db/qos/models.py#L135 | 21:30 |
| zigo | My rough guess is therefore that the change is in neutron-lib, because that's the only difference with my build in unstable that worked... | 21:31 |
| JayF | yeah, that one commit in neutron-lib is on the face of it incompatible with the neutron-lib code | 21:32 |
| opendevreview | Merged openstack/ironic-python-agent-builder stable/2023.1: Extend the DIB_CHECKSUM variable usage https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892913 | 21:47 |
| JayF | btw; for anyone reading this scrollback not in -neutron: that line of code doesn't have Index (the one that was removed was PrimaryKeyIndex, the one on that line is PrimaryKey) | 21:47 |
| JayF | this seems like just an unfortunate reality of our requirements: the library freeze released version of networking-baremetal picks up the library-freeze-released version of neutron-lib and 2023.1 neutron and mashes them together and it goes boom | 21:48 |
| TheJulia | yup | 22:21 |
| TheJulia | happens every so often | 22:21 |
| TheJulia | I can likely look at it next week if hjensas doesn't beat me to it | 22:21 |
| JayF | I put an item on the PTG etherpad (not for PTG so much as... so I see it when I'm looking at next cycle planning) | 22:21 |
| JayF | neutron has some kind of job that run that found it so they could bump requirements on their libraries | 22:21 |
| JayF | we should try and emulate that | 22:21 |
| JayF | and try to get them to run it, too, on neutron-lib | 22:22 |
| TheJulia | anyhow, I'm going to do some more laundry and start packing a suitcase | 22:22 |
| JayF | it sounded like haleyb was maybe somewhat amenable to that | 22:22 |
| mmalchuk | TheJulia antelope merged, could you please +W next : https://review.opendev.org/q/hashtag:ironic-week-prio+status:open+topic:extend-dib-checksum | 22:39 |
| mmalchuk | thanks | 22:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!