| opendevreview | Verification of a change to openstack/ironic master failed: Add DB API for Firmware and Object https://review.opendev.org/c/openstack/ironic/+/883062 | 01:56 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 04:28 |
| opendevreview | Verification of a change to openstack/ironic master failed: Utilize the JSON-RPC port https://review.opendev.org/c/openstack/ironic/+/879215 | 04:28 |
| opendevreview | Verification of a change to openstack/bifrost master failed: Fix key-order[task] linter warnings https://review.opendev.org/c/openstack/bifrost/+/874854 | 05:53 |
| opendevreview | Merged openstack/bifrost stable/2023.1: chore: allow ironic-inspector to work with IPv6 disabled https://review.opendev.org/c/openstack/bifrost/+/877494 | 06:08 |
| rpittau | good morning ironic! Happy Friday! o/ | 06:55 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove python 3.6 mock hack https://review.opendev.org/c/openstack/ironic/+/887023 | 07:03 |
| dtantsur | JayF, rpittau, any plans to release python-ironicclient? | 08:40 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 08:42 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 08:43 |
| rpittau | dtantsur: I think the release was blocked by JayF because we were waiting for a patch to go in | 09:02 |
| rpittau | nvm it was merged yesterday https://review.opendev.org/c/openstack/releases/+/887497 | 09:04 |
| dtantsur | ah, hmm, apparently I cannot read | 09:14 |
| iurygregory | good morning Ironic | 11:43 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 12:07 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 13:17 |
| TheJulia | good morning | 13:28 |
| dtantsur | morning TheJulia | 13:28 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 13:41 |
| iurygregory | good morning TheJulia | 13:54 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Build tinyipa with tinycore 14.x https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/887754 | 14:17 |
| TheJulia | oh osc format, you make my head hurt | 14:30 |
| opendevreview | Merged openstack/ironic master: Remove python 3.6 mock hack https://review.opendev.org/c/openstack/ironic/+/887023 | 14:41 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 14:58 |
| iurygregory | I'm giving up in trying to understand what is happening in the standalone jobs .-. | 15:04 |
| iurygregory | next week I will be out, so not much time to spend on it .-. | 15:05 |
| TheJulia | revert jammy? | 15:05 |
| iurygregory | was thinking about this tbh | 15:05 |
| TheJulia | it had to have passed at some point... why does it continue to fail though is the question | 15:05 |
| TheJulia | ... i mean aside from DHCP hating us | 15:05 |
| iurygregory | yeah | 15:06 |
| iurygregory | didn | 15:06 |
| iurygregory | ops | 15:06 |
| iurygregory | didn't we had some similar issue in the past? | 15:06 |
| iurygregory | and we decided to pin to an old version or something? | 15:06 |
| opendevreview | Julia Kreger proposed openstack/python-ironicclient master: WIP: Parent_node support https://review.opendev.org/c/openstack/python-ironicclient/+/887535 | 15:06 |
| iurygregory | or I have a lot of things in my mind lol | 15:06 |
| TheJulia | well, we had issues with ipxe in general failing | 15:07 |
| TheJulia | but it looks like things are partially working there | 15:07 |
| TheJulia | we've had to pin ipxe | 15:07 |
| TheJulia | I guess we just need to reproduce, really | 15:07 |
| rpittau | we have another job not running on jammy already, but it was constantly failing -> ironic-tempest-wholedisk-bios-snmp-pxe | 15:10 |
| rpittau | the standalone job passed most of the times in the patch and I was not able to reproduce the issue locally | 15:10 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/869052 looking at it I see similar failures for the standalone .-. | 15:14 |
| iurygregory | the same response timeout etc | 15:15 |
| iurygregory | .-. | 15:15 |
| TheJulia | so... I guess my challenge at the moment is you guys sort of said opposite things, at least that is how my brain interpretted it | 15:15 |
| rpittau | yeah, but it passed most of the times | 15:15 |
| TheJulia | And we have a ton of variables at play, is there a variable we know we can unwind in this? | 15:16 |
| TheJulia | ... (realistically, aside from jammy would be nice, but also being realistic I know that is the huge variable at the moment unless we can reproduce it locally | 15:16 |
| iurygregory | and looking at the history seems like it much worse after we switched to jammy .-. | 15:17 |
| iurygregory | https://zuul.opendev.org/t/openstack/builds?job_name=ironic-standalone-redfish&project=openstack%2Fironic&branch=master&skip=0 | 15:17 |
| rpittau | again, I was not able to reproduce the problem when I tried locally on jammy, but if you think moving back to focal for standalone would improve the situation, I have nothing against it | 15:18 |
| iurygregory | on my mind switching to focal would be the last thing to try | 15:19 |
| rpittau | I doubt I will have the time to troubleshoot further next week anyway | 15:19 |
| TheJulia | iurygregory: why would it be the last thing to try? | 15:20 |
| TheJulia | if it does suddenly start behaving as expected, that is a solid data point | 15:20 |
| TheJulia | we can then print t-shirts, something along the lines of "jammy ipxe == pain" | 15:20 |
| iurygregory | perfect! | 15:21 |
| rpittau | I don't exclude jammy is the problem, considering that one other job fails 100% of the times | 15:21 |
| TheJulia | which job is that? | 15:21 |
| rpittau | ironic-tempest-wholedisk-bios-snmp-pxe | 15:21 |
| iurygregory | lol https://askubuntu.com/questions/1420601/dns-not-working-on-22-04-server | 15:21 |
| iurygregory | I just googled jammy ipxe and this was the first thing on google for me | 15:22 |
| TheJulia | https://zuul.opendev.org/t/openstack/builds?job_name=ironic-tempest-wholedisk-bios-snmp-pxe | 15:22 |
| rpittau | TheJulia: that job is pinned to focal now cause it was never passing in the jammy patch | 15:23 |
| rpittau | sorry, I need to leave now, if you decide to pin standalone to focal I will approve it | 15:23 |
| rpittau | or we can continue talking on Monday | 15:23 |
| rpittau | have a great weekend o/ | 15:23 |
| TheJulia | have a wonderful weekend rpittau | 15:24 |
| iurygregory | have a great weekend rpittau | 15:24 |
| TheJulia | lets pin the job to focal, and then start a single patch to work the effort to unpin the three jobs | 15:26 |
| TheJulia | and use that for troubleshooting/diagnosis | 15:26 |
| iurygregory | ok | 15:26 |
| iurygregory | want me to push the patch to pin? | 15:27 |
| TheJulia | if you wouldn't mind doing so, I'd appreciate it | 15:29 |
| iurygregory | finishing to write the commit message | 15:30 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Move standalone jobs to focal https://review.opendev.org/c/openstack/ironic/+/887971 | 15:31 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: [DEBUG] Moving back to Jammy https://review.opendev.org/c/openstack/ironic/+/887972 | 15:34 |
| iurygregory | and the debug patch =) | 15:34 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM Enable OVN https://review.opendev.org/c/openstack/ironic/+/885087 | 15:35 |
| TheJulia | so I didn't realize the job was still on focal, that might just sort of work now | 15:38 |
| * TheJulia crosses fingers and waits | 15:38 | |
| iurygregory | TheJulia, seems like your patch had a merge conflict https://review.opendev.org/c/openstack/ironic/+/879215 | 15:38 |
| TheJulia | yeah, hold steps | 15:39 |
| * TheJulia fixes | 15:39 | |
| dtantsur | mm, interesting https://review.opendev.org/c/openstack/governance/+/887966 | 15:41 |
| TheJulia | I'm very concerned by the fact we would need to opt-in | 15:42 |
| TheJulia | to keep a branch, that is | 15:42 |
| dtantsur | I guess that's similar to our past "if there is no vendor behind it, it's not going to last" discussion | 15:47 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Utilize the JSON-RPC port https://review.opendev.org/c/openstack/ironic/+/879215 | 15:48 |
| TheJulia | yeah | 15:48 |
| TheJulia | Challenge is there seems to have been a classical disassociation and automatic assumption | 15:48 |
| TheJulia | fixed the merge conflict on the release mapping changes that happened in other patches | 15:50 |
| TheJulia | also fixed the reno, since I missed the version update last time I had to rebase it | 15:50 |
| iurygregory | ack, I will check after lunch | 15:51 |
| TheJulia | iurygregory: it just occured to me what is happening in https://zuul.opendev.org/t/openstack/build/5a4438deadf94548b4df9d944b579493/log/controller/logs/ironic-bm-logs/node-1_console_log.txt | 15:51 |
| TheJulia | we're not actually changing over to the other loader | 15:52 |
| TheJulia | well, to the config pointer from ipxe | 15:52 |
| iurygregory | humm | 15:52 |
| TheJulia | That seems like a hard break in ipxe behavior | 15:52 |
| iurygregory | "perfect" | 15:52 |
| TheJulia | but then there are other questions, but it could be just different builds | 15:52 |
| iurygregory | I saw another different error, the one where we just timeout when trying to start and can't download NBP | 15:53 |
| * iurygregory brb lunch | 15:54 | |
| TheJulia | ... why wouldn't it | 16:00 |
| dtantsur | mmm, snponly.efi just crashes? lovely.. | 16:07 |
| TheJulia | erm... Jul 06 18:55:19.592727 np0034598195 neutron-server[60220]: DEBUG neutron_lib.api.validators [req-76cdd891-8a79-4869-bb7d-9c6ee1e987c6 req-eca8ce79-251a-4289-a997-9445f57a1479 service ironic] tag:!ipxe,67 is not in valid_values {{(pid=60220) validate_values /usr/local/lib/python3.10/dist-packages/neutron_lib/api/validators/__init__.py:165}} | 16:11 |
| TheJulia | I'm not sure it is crashing | 16:11 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 16:12 |
| dtantsur | TheJulia: ouch! are we on OVN now? | 16:12 |
| TheJulia | nope | 16:12 |
| TheJulia | https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/validators/__init__.py#L166C41-L166C41 | 16:13 |
| dtantsur | I wonder if valid_values got changed | 16:16 |
| dtantsur | anyway, see you next week o/ | 16:21 |
| TheJulia | I'm not sure it is filtering it out | 16:21 |
| TheJulia | so the port has both a v4 and a v6 address at the same time | 16:23 |
| TheJulia | ... I wonder if that confuses dnsmasq | 16:23 |
| TheJulia | since we end up trying to do both parameters | 16:23 |
| TheJulia | i think we need a tcpdump capture | 16:45 |
| TheJulia | or to precisely reproduce | 16:45 |
| TheJulia | or just intercept a ci node | 16:45 |
| iurygregory | hummm | 16:48 |
| iurygregory | we can probably compare how it was in focal (if we had v4 and v6) | 16:49 |
| TheJulia | ++ | 16:51 |
| TheJulia | I've asked for the next standalone job build failure to get held for us | 16:51 |
| TheJulia | it looks like it is using the v4 options, given it logs them separately | 16:53 |
| iurygregory | perfect the standalone failed in focal also | 16:57 |
| JayF | TheJulia: dtantsur: I am in full support of that TC res, and helped Kristi edit it some. If there is feedback from Ironic-in-general I'm happy to share it, but you can trust that I won't support something that'd restrict our ability to support releases (especially since we have our own (broken, lol) CI infra) | 16:57 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/31cdb311f65e48629cc3ead0ea00e649/log/controller/logs/ironic-bm-logs/node-1_console_log.txt | 16:58 |
| JayF | that being said I'm very close to turning into a pumpkin for my trip to london | 16:58 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 17:00 |
| TheJulia | JayF: I'm more worried that it eradicates the ability for those loosely engaging to share patches without becoming fully engaged in upstream development, which is a hard lift for some of the ops folks out there that are supporting older releases. in other words, we end up heading down a path of unwinding what they sought because extra assumptions got piled in past the original simple ask of ops in Sydney from the 2018 | 17:01 |
| TheJulia | change | 17:01 |
| JayF | TheJulia: I'll note that 1) those operators don't really exist in practice, or at least have not moved the needle in contribution and 2) that a supported branch requires someone committing to support it, not just someone willing to toss patches up occassionally | 17:02 |
| JayF | so I think we might just disagree here; I'd rather us take the bitter pill of saying "we don't have enough folks to support this" than misleading operators into thinking something is supported when, in reality, it isn't | 17:02 |
| TheJulia | I think some of the folks in the room were disagreeing to the first perception, to be entirely honest. Granted, most of the time it is the upstream developers trying to be nice people too, so it just makes it cloudy in the fog sort of way | 17:03 |
| TheJulia | and the latter is obviously an issue, even thought hat is not what the operators were explicitly asking for upfront | 17:03 |
| TheJulia | we, as a community, expanded the requirements | 17:03 |
| JayF | I think it'll be incredibly difficult to get contributors -- including me -- to agree to a system where we leave something open and hope that some kind people will toss patches into it | 17:04 |
| TheJulia | yeah | 17:05 |
| JayF | because then someone runs a half-broken Ironic $oldVersion and we get hit with the bad PR | 17:05 |
| TheJulia | we inherently want it working too, by default | 17:05 |
| JayF | nobody is going to draw a line when theregister.com publishes "Ironic breached at $megaCorp, 10k servers pwned" | 17:05 |
| JayF | from $megaCorp using an open branch that we said was in '$anyWord maintenance" | 17:05 |
| TheJulia | agree completely | 17:06 |
| JayF | that's mainly where the buck stops for me; watering down branch support requirements to get more contributors is a nice idea; but we can't ship low quality software without it damaging us | 17:06 |
| JayF | so instead, flip the default to retire-first and allow people to opt in to longer maintenance, and have that maintenance guaranteed by a person willing to say that they'll maintain the branch | 17:07 |
| TheJulia | I agree, the challenge though, is will we get fixes like dave hill's change ever if there is not a path | 17:07 |
| * TheJulia tries to convince dave hill to maybe change it a little more | 17:07 | |
| JayF | I don't think we're doing people operating train that much of a favor by enabling them to continue to run 3 year old software :/ | 17:08 |
| JayF | and I've worked places where the ... generosity of EM support was used as a method to keep delaying upgrade projects | 17:08 |
| JayF | that's ^ not really playing into my TC-resolution-appoving logic, but I do think it's the truth | 17:09 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 17:09 |
| TheJulia | that is really just going to create more work for developers who work for firms who support those versions, unfortunately. At the same time, it is best for upstream to not feel encumbered by it. | 17:14 |
| JayF | I'll note I worked places that would sometimes do that stable branch work on their own | 17:15 |
| JayF | because backporting the patches into our set of patches was easier than rebasing everything we had on top | 17:15 |
| JayF | I sure hope nobody has that many patches at this point, though | 17:15 |
| TheJulia | yeah, nobody wants to be that deep down a rabbit hole | 17:18 |
| * TheJulia tries to convince dave to add a unit test | 17:20 | |
| iurygregory | looking at a failed job and green one from standalone, they have same message in the neutron server "tag:!ipxe,67 is not in valid_values" | 17:30 |
| TheJulia | I'm thinking they might just be red herrings | 17:31 |
| TheJulia | because if they were disregarded we wouldn't get anywhere | 17:31 |
| iurygregory | yup, agree | 17:32 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM Enable OVN https://review.opendev.org/c/openstack/ironic/+/885087 | 17:35 |
| TheJulia | I'm seeing if we can recheck https://review.opendev.org/c/openstack/ironic/+/887271 and get the ironic-standalone node held if it fails | 17:37 |
| TheJulia | since obviously we failed in a different way it seems | 17:38 |
| iurygregory | ack | 17:39 |
| iurygregory | oh nvm seems like in focal is working, I was looking at the debug patch <facepalm> | 18:03 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/887971 | 18:03 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix db migration tests for sqlalchemy 2.0 https://review.opendev.org/c/openstack/ironic/+/887432 | 18:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/ironic/+/886020 | 18:16 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 18:31 |
| TheJulia | filing under: maybe it was eventlet all along: https://paste.opendev.org/show/bLg4eqoQnZBS1PphUu63/ | 18:32 |
| TheJulia | JayF: ^ | 18:32 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 18:33 |
| TheJulia | could I get another review on https://review.opendev.org/c/openstack/ironic-python-agent/+/882367 ? | 18:40 |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 18:52 |
| JayF | TheJulia: what does that mean, though | 18:57 |
| JayF | I'm terms of repair action | 18:57 |
| JayF | **in | 18:57 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887990 | 18:58 |
| TheJulia | dunno | 18:58 |
| TheJulia | walking to lunch for a little bit | 18:58 |
| JayF | I'm sus that excluding os is finally coming home to roost https://github.com/openstack/ironic/blob/master/ironic/cmd/__init__.py#L26 | 19:05 |
| JayF | I'll look at open bugs and such but won't really have time to go deep. | 19:06 |
| JayF | (today) | 19:06 |
| ashinclouds[m] | Eh, that shouldn’t cause unit test issues… at least I don’t think | 19:06 |
| * ashinclouds[m] awaits her salad | 19:06 | |
| opendevreview | David Hill proposed openstack/ironic stable/train: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887954 | 19:09 |
| opendevreview | David Hill proposed openstack/ironic master: Make sure there's no stale path_tmp https://review.opendev.org/c/openstack/ironic/+/887991 | 19:14 |
| iurygregory | TheJulia, when you have time https://review.opendev.org/c/openstack/ironic/+/887971 o/ | 19:33 |
| TheJulia | ack, still hoping to snag us a CI node to diagnose before that merges | 19:39 |
| TheJulia | :) | 19:39 |
| iurygregory | gotcha =) | 19:47 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887990 | 19:58 |
| JayF | https://github.com/eventlet/eventlet/issues/798 | 19:59 |
| JayF | interesting one from gibi | 19:59 |
| JayF | not sure it's ours but worth a glance | 19:59 |
| TheJulia | interesting | 20:00 |
| TheJulia | well | 20:00 |
| TheJulia | hmmm | 20:00 |
| JayF | heh it showing up in standalone job more often is interesting | 20:00 |
| JayF | because if nova is blcoking bad eventlets in requriements | 20:01 |
| JayF | would that save us from a bad eventlet in tempest but not in standalone/unit tests? | 20:01 |
| JayF | uh, yeaaahhh | 20:01 |
| JayF | that might check | 20:01 |
| JayF | science | 20:02 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Limit to eventlet 0.30.1 and newer https://review.opendev.org/c/openstack/ironic/+/887996 | 20:02 |
| JayF | we allow a lot older eventlet than nova | 20:03 |
| JayF | I'll poke at that from the airport tomorrow | 20:03 |
| JayF | fwiw, gibi's reported bug is .33.1 so these are opposed ideas (if that bug impacts us, we are already on newer eventlet) | 20:04 |
| * JayF is going to be mostly out of IRC for the rest of the day, but will be around some tomorrow killing time in the airport | 20:05 | |
| iurygregory | safe travels JayF =) | 20:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Cleanup if images.fetch fails https://review.opendev.org/c/openstack/ironic/+/887990 | 20:24 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Allow md5 to be disabled from the conductor https://review.opendev.org/c/openstack/ironic-python-agent/+/882367 | 20:40 |
| JayF | fyi standalone failed on 887996 and freeze shows eventlet==0.33.3 | 20:58 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: DNM: Eventlet science https://review.opendev.org/c/openstack/ironic/+/887996 | 21:05 |
| TheJulia | it feels super close with jammy and ovn | 21:06 |
| JayF | it'll be great when you get that working | 21:15 |
| JayF | I know you've been poking it for ages | 21:16 |
| TheJulia | it looks like either it doesn't see the packets, or I've got the bind wrong, or it is not setting up dhcp | 21:17 |
| TheJulia | I won' thave new clues until next week since I've messaged one of the ovn folks | 21:17 |
| * TheJulia lets CI spin and hopefully we'll have a node isolated soon to dig into | 21:19 | |
| iurygregory | bye everyone, see you on July 17 o/ | 21:56 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix the HTTP code of the BadRequest exception https://review.opendev.org/c/openstack/ironic/+/887271 | 23:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!