| opendevreview | Merged openstack/ironic bugfix/20.2: Fix pxe image lookups https://review.opendev.org/c/openstack/ironic/+/858572 | 04:35 |
|---|---|---|
| -opendevstatus- NOTICE: review.opendev.org (Gerrit) is currently down, we are working to restore service as soon as possible | 07:31 | |
| arne_wiebalck | Good morning, Ironic! | 08:04 |
| kubajj | Good morning arne_wiebalck and Ironic! | 09:39 |
| kubajj | Is it just me, or is review.opendev.org unreachable? | 09:48 |
| ajya | kubajj: it's a known issue, see #opendev | 09:57 |
| kubajj | thanks ajya | 09:58 |
| iurygregory | morning Ironic | 11:31 |
| arne_wiebalck | hey kubajj ajya and iurygregory o/ | 12:40 |
| iurygregory | arne_wiebalck, o/ | 12:40 |
| ajya | hi arne_wiebalck | 12:57 |
| TheJulia | good morning | 13:08 |
| -opendevstatus- NOTICE: review.opendev.org (Gerrit) is back online | 14:26 | |
| *** knikolla[m] is now known as knikolla | 15:27 | |
| kubajj | TheJulia: Hi, does this error message tell me in what way they are out of sync or not? I have been staring at this for a while now. https://paste.opendev.org/show/bJwftrKi9NtmiI6cXJ0E/ | 16:35 |
| JayF | kubajj: I'm taking a look | 16:37 |
| kubajj | JayF: Thanks | 16:37 |
| JayF | *blink* | 16:37 |
| JayF | kubajj: I think there needs to be more/better output to troubleshoot that; or I'm completely barking up the wrong tree | 16:40 |
| kubajj | JayF: Here is the Zuul job: https://zuul.opendev.org/t/openstack/build/aa87c15e911244b4a492acf82f48e09a Here is the change: https://review.opendev.org/c/openstack/ironic/+/862569/ | 16:41 |
| JayF | kubajj: I'm lost, maybe TheJulia will have better insight. If she doesn't come around to help in a few; maybe we can sync up in a zoom or something? | 16:44 |
| JayF | kubajj: what TZ are you in? | 16:44 |
| TheJulia | I can look in like 15 minutes | 16:44 |
| TheJulia | trying to wrap current thought | 16:44 |
| kubajj | JayF: UTC | 16:45 |
| JayF | Alright, I have that on my world clock lol | 16:46 |
| JayF | the company I work for is UK-based | 16:46 |
| kubajj | Nice | 16:49 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep https://review.opendev.org/c/openstack/ironic-inspector/+/860731 | 16:58 |
| TheJulia | kubajj: I think we would actually need to look at some work in progres code | 16:59 |
| TheJulia | ohhh | 16:59 |
| TheJulia | already posted | 16:59 |
| * TheJulia looks | 16:59 | |
| JayF | TheJulia: johnthetubaguy: We should coordinate a time to sync up tomorrow about nova<>ironic driver stuff if you're both available | 17:03 |
| JayF | or sometime this week in general | 17:04 |
| TheJulia | JayF: ++ | 17:05 |
| TheJulia | kubajj: interesting | 17:05 |
| JayF | TheJulia: johnthetubaguy: Tomorrow is actually meeting-free for me. Anything in business hours PST is perfect for me. | 17:05 |
| johnthetubaguy | so daylight savings is a pain this week I suspect, its 5pm with me in the UK right now | 17:11 |
| johnthetubaguy | what time did you want to start, UTC wise? | 17:12 |
| JayF | I can go as-early-as 1400 UTC | 17:12 |
| JayF | 1500 UTC is like, dead on perfect time for me (8am) because it'll give my body time to metabolize the caffeine :D | 17:12 |
| JayF | TheJulia: what does your schedule look like? | 17:12 |
| johnthetubaguy | ah, well that sounds better | 17:13 |
| TheJulia | looking | 17:13 |
| johnthetubaguy | 3 or 4 UTC for an hour or two totally works for me | 17:13 |
| JayF | My usual working hours, with UTC as it is now, is like 1400-2400 | 17:13 |
| JayF | er. that can't be right | 17:13 |
| JayF | 1400-2300 (?) | 17:13 |
| TheJulia | I can do after 1600 UTC | 17:14 |
| JayF | johnthetubaguy: TheJulia: 1600-1700 UTC? | 17:14 |
| TheJulia | sure | 17:14 |
| JayF | I'll send out calendar invites with zoom links | 17:14 |
| johnthetubaguy | that works nicely for me too | 17:14 |
| TheJulia | ack | 17:14 |
| johnthetubaguy | cool, thank you! | 17:14 |
| JayF | TheJulia: your @gmail OK for the meeting invite? | 17:14 |
| TheJulia | JayF: sure | 17:15 |
| TheJulia | JayF: actually, jkreger@redhat please | 17:15 |
| JayF | dot com? | 17:15 |
| TheJulia | yes please | 17:15 |
| JayF | np | 17:16 |
| TheJulia | I've not had that other laptop open and I've got a few meetings that morning, so odds are it just won't get opened unti lthe afternoon | 17:16 |
| JayF | speaking of, I need to get on my MDM laptop and check email too | 17:19 |
| kubajj | TheJulia: any clue on what is wrong? | 17:24 |
| TheJulia | kubajj: yup, one minute | 17:26 |
| TheJulia | kubajj: like 2 minutes and I should have comments posted | 17:30 |
| kubajj | TheJulia: thanks | 17:31 |
| TheJulia | hmm | 17:32 |
| TheJulia | kubajj: so, I think it is all indexes and the first part of the reporting is a red herring | 17:48 |
| TheJulia | because it is comparing data structures | 17:48 |
| TheJulia | and the second entry doesn't match, not the first | 17:48 |
| kubajj | TheJulia: so you think the indexes are wrong? | 17:54 |
| kubajj | I am not sure I understood that | 17:54 |
| TheJulia | sorry, apparently I didn't click send in gerrit | 17:55 |
| TheJulia | refresh your change | 17:55 |
| JayF | If there are no objections; I'll be shortly pushing changes to all older ironic-prometheus-exporter branches to remove the integration jobs | 18:07 |
| JayF | they are failing for unexplainable reasons, and frankly nothing gets backported to them so I don't want to waste my time | 18:07 |
| opendevreview | Jay Faulkner proposed openstack/ironic-prometheus-exporter stable/train: CI: Various fixes https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/860184 | 18:12 |
| opendevreview | Jay Faulkner proposed openstack/ironic-prometheus-exporter stable/ussuri: CI: Various fixes https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/860183 | 18:14 |
| kubajj | TheJulia: I see. Thanks | 18:14 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep https://review.opendev.org/c/openstack/ironic-inspector/+/860731 | 18:19 |
| JayF | ironic-inspector bugfix/10.7 branch is in really, really bad shape in CI https://review.opendev.org/c/openstack/ironic-inspector/+/860171 | 18:23 |
| JayF | wondering how folks would feel about just retiring out that branch? If we support the last three, it goes away on it's own when we cut the first bugfix release of this cycle AIUI | 18:24 |
| JayF | RFR https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/860183 -- it's passing tests now that I pulled the integration tests from the branch | 18:25 |
| TheJulia | I'm all for just retiring the branch | 18:26 |
| JayF | for inspector, yeah? | 18:27 |
| JayF | Who is the downstream for that, consuming it? | 18:27 |
| JayF | RH has a product aligned with that branch, right? | 18:27 |
| JayF | So like, auditing the bugfix branches right now, and it looks like we made a bugfix release between xena and yoga which isn't listed as supported anymore? | 18:39 |
| TheJulia | I don't have a good way to easily check that | 18:44 |
| TheJulia | I would assume dtantsur, iurygregory, or rpittau should be able to answer that | 18:45 |
| JayF | thank you, I'm actually going to hit the list about some of this I think | 18:53 |
| JayF | because there is a bigger task to be done w/r/t bugfix branches | 18:53 |
| * JayF just put together https://etherpad.opendev.org/p/IronicBugfixBranchCleanup | 18:53 | |
| JayF | https://zuul.opendev.org/t/openstack/build/f4cbd438269e4837be9e6588739624d9/log/job-output.txt I wonder if this is reflecting a real break in wallaby-era bifrost | 18:56 |
| JayF | > libvirt.libvirtError: XML error: unexpected root element <domain> expecting <device> | 18:56 |
| JayF | this is another one I can just kill that job on, but the CI failure looks less random are more suspicious than the others... | 18:57 |
| TheJulia | that looks like something changed unexpectedly | 18:59 |
| JayF | Yes. That's why I called it out. Was hoping it was recognizable | 19:00 |
| JayF | certainly not one to just drop or paper over, even in ir-prom-exporter | 19:00 |
| TheJulia | https://zuul.opendev.org/t/openstack/build/f4cbd438269e4837be9e6588739624d9/log/job-output.txt#5484-5490 | 19:04 |
| TheJulia | the ansible community colleciton is no longer compatible with the libvirt on that test vm | 19:04 |
| JayF | So this is a job broken by the new zuul ansible? | 19:06 |
| TheJulia | quite possible | 19:06 |
| JayF | yeah, and we pushed a version sticky to bifrost for this, I think | 19:07 |
| JayF | but it's a ticking bomb | 19:07 |
| JayF | it's arguably setup wrong that bifrost runs in the same ansible context as zuul | 19:07 |
| TheJulia | or the package which is getting pulled in for the deploy has updated and broken | 19:07 |
| JayF | For *this particular instance* is it worth keeping as an example and fixing? Or just nuke the job? | 19:08 |
| TheJulia | at the same time, the opposite likely can be argued :( | 19:08 |
| JayF | I'm sure someone could make that argument; I wouldn't :) | 19:08 |
| TheJulia | I think during our next weekly meeting we should likely just have a specific discussion about ansible versions, since this is going to continue to happen | 19:09 |
| TheJulia | as zuul wants to be able to enter their move quickly stride | 19:09 |
| TheJulia | since ansible is doing the same | 19:09 |
| TheJulia | obviously a static pinned version is the way to remedy this | 19:10 |
| JayF | Bifrost is a weak point in my understanding; can I add an item to the agenda and you own discussion on this? | 19:10 |
| TheJulia | but that won't work for the zuul job itself and it needs to be a disjointed operation | 19:10 |
| TheJulia | sure | 19:10 |
| JayF | and/or find a victi^W kind volunteer to delegate it to | 19:10 |
| TheJulia | just put enough context of what the issue is (i.e. ansible collection no longer compatible with base os...) | 19:10 |
| JayF | of course | 19:11 |
| JayF | also I'll note, see the list about bugfix branch stuff, I just hit it | 19:11 |
| TheJulia | (... libvirt, in particular here, is a fraking nightmare since it is binding build as opposed to a full api | 19:11 |
| TheJulia | ) | 19:11 |
| JayF | dtantsur: iurygregory: stevebaker[m]: Please check list with [ironic][release] tag for post about bugfix branch lifecycle; I feel like you three are the major stakeholders there and I'd like your input | 19:11 |
| iurygregory | JayF, ack | 20:05 |
| opendevreview | Kirill proposed openstack/ironic master: add vnc console https://review.opendev.org/c/openstack/ironic/+/860689 | 21:05 |
| opendevreview | Kirill proposed openstack/ironic master: add vnc console https://review.opendev.org/c/openstack/ironic/+/860689 | 21:06 |
| opendevreview | Kirill proposed openstack/ironic master: add vnc console https://review.opendev.org/c/openstack/ironic/+/860689 | 21:08 |
| JayF | TheJulia: related to our earlier chat... | 21:10 |
| JayF | https://zuul.opendev.org/t/openstack/build/2cf36d058f124c73b74ffd77fcc35f40/log/job-output.txt | 21:10 |
| JayF | ironic-prom-exporter train seems to be broken at a very basic level | 21:10 |
| JayF | importlib-metadata being used is incompatable with setuptools being used | 21:10 |
| JayF | which implies to me there's something very basic broken in CI; and likely not in an ironic-specific way (maybe?) | 21:11 |
| JayF | AFAICT we do not pin either of those packages in requirements for that branch/project combo | 21:11 |
| JayF | and this is in /unit/ tests | 21:15 |
| TheJulia | ugh.... | 21:31 |
| JayF | I spent a nontrivial amount of time trying to root cause this, and I'm to the point of "if someone else doesn't find something obvious, hit the list" | 21:32 |
| TheJulia | :( | 21:36 |
| TheJulia | I think anything train is forever broken since the queue change + the jobs base is now jammy as anything image buildy is now toast due to jammy on train | 21:37 |
| JayF | TheJulia: well, I think the issue above is separate | 21:38 |
| JayF | TheJulia: the issue above is *a unit test job* not even buiilding images | 21:38 |
| TheJulia | well, requirements can't be fixed either | 21:38 |
| JayF | so we're looking at, e.g. having to EOL train rather than being able to unwedge it by disabling tempest tests | 21:39 |
| JayF | I proposed we retire train at one point, so I obviously don't mind that | 21:39 |
| TheJulia | That is my impression, yes | 21:39 |
| JayF | but I'm just pointing out that there's no middle ground | 21:39 |
| JayF | we invest the effort, seemingly infinite, to fix it | 21:39 |
| JayF | or we just punt it | 21:39 |
| TheJulia | agree 1000% | 21:39 |
| JayF | You wanna hit the list? I think you have a better understanding of the moving parts | 21:39 |
| JayF | and we can add it to the meeting agenda for monday | 21:39 |
| JayF | train will be dead by thanksgiving lol | 21:39 |
| TheJulia | stevebaker[m]: ^^^ how do you feel about this | 21:40 |
| TheJulia | if steve doesn't want to throw stuff at me... | 21:40 |
| JayF | If you need a bad cop, I can be the bad cop: I do not think I have the time or skills or motivation to fix train CI. | 21:41 |
| JayF | We cannot pretend train is in EM if it's effectively EOL due to dead CI. | 21:41 |
| TheJulia | nah, I don't think anyone needs a bad cop on this | 21:41 |
| TheJulia | just... reality | 21:41 |
| JayF | So in my opinion, the default action is "It's going to be EOL'd" until or unless someone volunteers the time to unbreak the CI | 21:41 |
| JayF | I already had the uniform pressed and everything! LOL | 21:41 |
| TheJulia | ... | 21:42 |
| TheJulia | Okay then! | 21:42 |
| JayF | Hey, yesterday was halloween | 21:42 |
| JayF | costume jokes are still timely, right? | 21:42 |
| TheJulia | but already pressed!? | 21:42 |
| TheJulia | I guess | 21:42 |
| TheJulia | Anyway, my brain is fried | 21:42 |
| JayF | lol I don't think they make them in my size anyway :P | 21:42 |
| TheJulia | and I need to go to the post office in a few | 21:42 |
| JayF | have a nice one, thanks for sharing CI misery with me all day :| | 21:42 |
| JayF | misery loves company. Almost as much as it would love CI fixes. lol | 21:43 |
| TheJulia | :( ^ 1 million | 21:43 |
| stevebaker[m] | JayF: yeah I think its time for train to end, which raises the question of ussuri and victoria | 21:43 |
| JayF | https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030717.html only took a month for everyone to come around, LOL | 21:44 |
| stevebaker[m] | JayF: indeed :) we've had a major release downstream since then, so wallaby is the new train for us | 21:46 |
| JayF | don't say "wallaby is the new train" to someone who's been on the CI train going in circles with train all day :P | 21:47 |
| JayF | wallaby is a better train, with working CI, and flowers | 21:47 |
| JayF | that's better LOL | 21:47 |
| JayF | I seriously wonder if we do this, retire T/U/V, if we should start actively monitoring CI on projects we care about. | 21:47 |
| stevebaker[m] | didn't I hear wallaby bifrost is broken? | 21:47 |
| JayF | wallaby ironic-prom-exporter was, I think | 21:48 |
| stevebaker[m] | ah | 21:48 |
| JayF | for i-p-e, I'm basically pulling tests to get it passing | 21:48 |
| JayF | because I see zero evidence of patches being backported | 21:48 |
| TheJulia | oh, for some reason I was thinking that was train | 21:48 |
| TheJulia | I think this demonstrates how fried my brain is right now | 21:48 |
| JayF | there are i-p-e problems up and down the list | 21:48 |
| JayF | and I've been complaining about them on different branches | 21:48 |
| JayF | my brain has also derailed lol | 21:48 |
| JayF | FYI: I pushed PRs today to openstack/releases to cut a bugfix version release of all ironic projects in X/Y that have had changes | 21:50 |
| TheJulia | I recommend finding one furchild, and capturing their warmth | 21:50 |
| JayF | we released W last week in anticipation of it being EM'd | 21:50 |
| JayF | TheJulia: eh, I'm in my last hour. I'll make it :) | 21:51 |
| ashinclouds[m] | I should have called it a day like an hour ago :/ | 21:52 |
| ashinclouds[m] | Anyway, to the post office! | 21:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!