| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Fix batching error due to outdated etcd3gw https://review.opendev.org/c/openstack/networking-generic-switch/+/886404 | 00:09 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 00:38 |
| opendevreview | OpenStack Release Bot proposed openstack/python-ironic-inspector-client stable/2023.2: Update .gitreview for stable/2023.2 https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/895093 | 01:36 |
| opendevreview | OpenStack Release Bot proposed openstack/python-ironic-inspector-client stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2 https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/895094 | 01:36 |
| opendevreview | OpenStack Release Bot proposed openstack/python-ironic-inspector-client master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/895095 | 01:36 |
| *** osmanlicilegi is now known as Guest0 | 04:33 | |
| opendevreview | Merged openstack/ironic-ui master: Fix release note build https://review.opendev.org/c/openstack/ironic-ui/+/894164 | 09:24 |
| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Fake: support adding a random sleep and injecting failures https://review.opendev.org/c/openstack/networking-generic-switch/+/874793 | 10:42 |
| opendevreview | Merged openstack/ironic master: devstack - configurable ipv6 address mode https://review.opendev.org/c/openstack/ironic/+/893622 | 10:43 |
| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Honor ngs_save_configuration setting when using batch commands https://review.opendev.org/c/openstack/networking-generic-switch/+/886405 | 11:24 |
| iurygregory | good morning ironic | 11:44 |
| dtantsur | morning iurygregory | 11:44 |
| dtantsur | does anyone understand why the Nova's grenade plugin is trying to use the non-existing cirros 0.6.2 image? And why it does not break anyone else? | 11:44 |
| dtantsur | ref https://a86af8e24720e7d1aa6e-c7277a3e95ece459b8b2ecfff9ffad89.ssl.cf5.rackcdn.com/894015/1/check/ironic-inspector-grenade/1b5ab34/controller/logs/grenade.sh_log.txt | 11:44 |
| dtantsur | hmmm, it's coming from the old Ironic plugin. why doesn't it break Ironic then? | 11:45 |
| iurygregory | dtantsur, I think we have the info in some jobs | 11:45 |
| iurygregory | hardcoded | 11:45 |
| dtantsur | ehhm, old Ironic uses 0.6.1, old Ironic in the inspector gate uses 0.6.2. WUT | 11:46 |
| iurygregory | so we have 0.6.1 in our job config and devstack/tools/ironic/scripts/cirros-partition.sh | 11:49 |
| dtantsur | then how does inspector manage to use 0.6.2? Oo | 11:49 |
| * iurygregory checks inspector | 11:49 | |
| dtantsur | 0.6.2 is indeed the devstack's default | 11:50 |
| iurygregory | in case we don't set it will get the default of CIRROS_VERSION_DEVSTACK | 11:51 |
| iurygregory | so yeah it would make sense since is the default there | 11:51 |
| dtantsur | I'm starting to get an idea of it. The image is only created on the old devstack, and the version there is different (0.5.2) | 11:54 |
| iurygregory | this is only happening in inspector-grenade? | 11:55 |
| dtantsur | I suspect Ironic overrides the version | 11:56 |
| dtantsur | yep, we do | 11:57 |
| iurygregory | dtantsur, https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L931 | 11:58 |
| iurygregory | in ironic-grenade we set the version we want | 11:59 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-inspector master: Use one version of Cirros in the grenade job https://review.opendev.org/c/openstack/ironic-inspector/+/895164 | 11:59 |
| dtantsur | let's try ^^^ | 11:59 |
| iurygregory | ++ | 11:59 |
| iurygregory | yeah | 11:59 |
| iurygregory | quick question, how bad would be change the value of [conductor]power_state_change_timeout to 90 ? | 12:00 |
| dtantsur | not ideal. kind of a last resort thing if there is no hope to fix the hardware. | 12:01 |
| iurygregory | my clean_step to update the firmware said it failed with "Failed to set node power state to power on.", the update was executed successfully .-. | 12:02 |
| iurygregory | and I've found this on the logs https://paste.opendev.org/show/b7k95XWrDquLeUXgzpUO/ | 12:03 |
| iurygregory | maybe 60 is not enough... | 12:03 |
| dtantsur | that may just mean that your hardware is not available for a long time. | 12:04 |
| dtantsur | it's something to handle during the firmware upgrade, but I don't think raising the general timeout is a good idea. | 12:04 |
| iurygregory | ack | 12:04 |
| dtantsur | what is actually happening to the node in the meantime? | 12:05 |
| dtantsur | do power calls just error? return None? return a stale value? | 12:05 |
| iurygregory | let me try to find in the logs | 12:05 |
| iurygregory | the node was on PowerOn, I've sent the command to do the clean step to update the firmware, task is created firmware is updated, it triggers a reboot_to_finish_step, the node goes to PowerOff and then timeout because it failed to change power state to 'power on' by 'rebooting': Failed to set node power state to power on https://paste.opendev.org/show/bhGNkcKx8vqteeVq2awC/ | 12:34 |
| dtantsur | interesting! do you have any insight into what was going on with the hardware itself? | 12:56 |
| dtantsur | I suspect it was doing the actual upgrade.. | 12:56 |
| opendevreview | Dmitry Tantsur proposed openstack/sushy-tools master: Use WAL mode for SQLite cache https://review.opendev.org/c/openstack/sushy-tools/+/895168 | 13:09 |
| iurygregory | agree, I will try to watch the console and see what happens | 13:23 |
| TheJulia | iurygregory: similarly, on some other machines, I think I've seen them take 2-3 minutes for their bmcs to update. From that other big box vendor. | 13:37 |
| iurygregory | TheJulia, oh god to know | 13:38 |
| * TheJulia suspects froidian slip there | 13:38 | |
| iurygregory | I just noticed that for ilo in the management interface they have a wait parameter... <checking what it does> | 13:39 |
| TheJulia | we likely want the lock held while waiting, unfortuantely. | 13:39 |
| dtantsur | we can wait for power on longer in *this case*, but I don't think we should raise the default timeout | 13:43 |
| TheJulia | ++ | 13:55 |
| TheJulia | I'd be fine waiting something like 5 minutes, on some level we might also be masking a failed upgrade as well, or my favorite "the partial upgrade" that didn't completely work. | 13:56 |
| *** tosky_ is now known as tosky | 14:06 | |
| dtantsur | bloody inspector grenade.. can I just nuke it? | 14:07 |
| * TheJulia steps back | 14:07 | |
| TheJulia | who are you, and what did you do with our dear friend dtantsur ?! | 14:07 |
| TheJulia | ;) | 14:08 |
| TheJulia | I think you asking that way tells us all we need to know | 14:08 |
| dtantsur | :D | 14:08 |
| TheJulia | It doesn't make sense to keep... although have we merged an official "we're deprecating this stand along service" release note yet? | 14:08 |
| dtantsur | We have not. But giving its virtual zero amount of changes, a grenade job may be simply an overkill. | 14:10 |
| TheJulia | Then we should do both | 14:11 |
| TheJulia | Provide notice "this is merging into ironic, stay tuned!" | 14:11 |
| TheJulia | and kill the grenade job | 14:11 |
| iurygregory | ok the error (couldn't power on) happened a few seconds before the iLO GUI showed that the firmware update was finished and said iLO is being reset.. | 14:14 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-inspector master: Update the project status and move broken jobs to experimental https://review.opendev.org/c/openstack/ironic-inspector/+/895164 | 14:16 |
| dtantsur | TheJulia, like this ^^? | 14:16 |
| JayF | dtantsur: TheJulia: My only question: is it possible some of the half-support in Ironic could break upgrades | 14:40 |
| * JayF puts that in gerrit review | 14:41 | |
| dtantsur | JayF, not impossible (but we don't run the inspector grenade job on ironic, so it won't help much) | 14:41 |
| JayF | I'm more saying; are we sure we *didn't already break the upgrade* | 14:42 |
| JayF | if we're punting getting the job to work we need to feel super confident about that | 14:42 |
| JayF | You're talking to someone who literally just got burned *yesterday* for pushing a change without CI so I'm a little extra sensitive to this lol | 14:43 |
| TheJulia | I think the release note paints the picture appropriately | 14:44 |
| dtantsur | given how cryptic grenade is, it's not even trivial to figure out why it fails.. but the new module takes a different code path, so I'm also struggling to imagine how it will break grenade without breaking everything else | 14:44 |
| TheJulia | "not done, in progress, marking this in maintenance" | 14:44 |
| JayF | dtantsur: I know almost nothing about inspector, so some of those questions are just asking defensive questions | 14:45 |
| JayF | dtantsur: if the answer is "the only way that could break something is it they put ironic in the catalog as the inspector" or something similar, that's an A++ answer | 14:45 |
| TheJulia | Well, even putting ironic in as inspector in the catalog wouldn't really work since it is aiui, not a 1-1 move | 14:52 |
| TheJulia | They would need to intentionally drive off the happy path, as I see it | 14:53 |
| TheJulia | Maybe we need an explicit issue in ironic’s release notes to advise against doing so? | 14:54 |
| JayF | I'd +1 such a change | 14:55 |
| JayF | btw, please review prelude | 14:55 |
| JayF | it's ht'd ironic-week-prio | 14:55 |
| TheJulia | Wheeee IRC | 14:59 |
| TheJulia | I added some spacing between items on https://etherpad.opendev.org/p/ironic-ptg-october-2023 because it was getting hard to separate items | 15:09 |
| TheJulia | And another day, another item :) | 15:16 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 15:35 |
| TheJulia | okay, that fixed the doc build issue in that | 15:36 |
| JayF | TheJulia: hmm | 16:27 |
| JayF | TheJulia: so locally on https://review.opendev.org/c/openstack/ironic/+/895007 | 16:27 |
| JayF | TheJulia: it's building 2023.1 notes under 2023.2 notes | 16:27 |
| JayF | and that's what I expected based on TC discussions | 16:28 |
| JayF | so I'm a little weirded out to not see it working that way | 16:28 |
| JayF | (in the gate) | 16:28 |
| TheJulia | unless it was added as explicit logic to reno, it doesn't work that way by default | 16:29 |
| TheJulia | unreleased is always what we're about to branch off, and the content is only ever that branch until the prior stable branch | 16:29 |
| JayF | Hmm. I thought changes had been made to that degree | 16:30 |
| JayF | I'll have to go lookup the governance change b/c I don't see reno changes | 16:30 |
| TheJulia | maybe not released or maybe the version changed | 16:30 |
| TheJulia | and honestly, wrapping it all together is *awful* | 16:30 |
| TheJulia | it means duplication across pages | 16:30 |
| JayF | TheJulia: OK, so regardless of what we do for slurp, this isn't a slurp lol | 16:33 |
| TheJulia | bahahahaha | 16:33 |
| JayF | TheJulia: Y->A->C and this is B, and I was confused | 16:33 |
| * JayF has not been at his best the last two or three days | 16:33 | |
| TheJulia | well, time to re-write the prelude! | 16:33 |
| JayF | well and stop worrying about it being too verbose | 16:34 |
| TheJulia | By prelude, it will be a prelude! | 16:34 |
| JayF | since I can delete it all | 16:34 |
| TheJulia | so interestingly, there *seems* to be an issue with microversion issue in the sdk | 16:34 |
| TheJulia | at least, trying to write a test :\ | 16:35 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [releasenotes] Prelude for 2023.2/bobcat https://review.opendev.org/c/openstack/ironic/+/895007 | 16:44 |
| -opendevstatus- NOTICE: The lists.airshipit.org and lists.katacontainers.io sites will be offline briefly for migration to a new server | 16:47 | |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [releasenotes] Prelude for 2023.2/bobcat https://review.opendev.org/c/openstack/ironic/+/895007 | 16:54 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [releasenotes] Prelude for 2023.2/bobcat https://review.opendev.org/c/openstack/ironic/+/895007 | 16:54 |
| JayF | Those are updated, ready for re-review. Especially curious what folks think about the upgrade note I added about possibility to skip this release. I'm on the fence if we should keep it but I feel like it's good context to have in the release notes. | 16:56 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/zed failed: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892594 | 17:04 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/yoga failed: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892687 | 18:04 |
| TheJulia | JayF: able to reproduce shard being dropped :) | 18:42 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/xena failed: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892595 | 18:42 |
| TheJulia | allow_unknown_params *is* broken | 18:49 |
| JayF | for everything, or just baremetal? | 18:50 |
| TheJulia | not sure yet | 18:51 |
| TheJulia | oh, I sort of see what is going on | 18:51 |
| TheJulia | okay, so... so the doc string actually provides the clarity | 19:02 |
| TheJulia | "True to accept, but discard unknown query parameters" | 19:02 |
| TheJulia | it was never an override in other words | 19:02 |
| TheJulia | it says "False will result in a validation exception", but that is not true, it is *always* true. | 19:03 |
| JayF | aha so it's about discarding unknown values instead of raising on them | 19:04 |
| TheJulia | yup | 19:04 |
| JayF | so it was not possible to do what we wanted, that's nice to know | 19:04 |
| TheJulia | yeah, unfortunately | 19:04 |
| JayF | but does indicate we will need to take a workitem to ensure openstacksdk supports modern Ironic API abilities | 19:05 |
| TheJulia | indeed | 19:05 |
| TheJulia | granted, where this gets difficult, there is a non-zero percentage we just won't care about at all | 19:05 |
| TheJulia | at least, in terms of openstacksdk for other openstack service consumption | 19:05 |
| JayF | I don't know what you mean by that | 19:06 |
| JayF | we who? | 19:06 |
| TheJulia | roughly half our consumers don't use nova at all | 20:13 |
| JayF | I guess I'm missing how that ties back to "make sure Ironic's API is fully represented in openstacksdk" -- I thought openstacksdk had use cases outside of openstack<>openstack comms | 20:14 |
| TheJulia | some like ansible, but it is not automatically at the top of any mind | 20:17 |
| TheJulia | obviously there is room for improvement, and I think part of the challenge is the style | 20:18 |
| TheJulia | but it might not be that bad really once you wrap your head around it | 20:18 |
| JayF | that is mostly what I'm counting on | 20:18 |
| JayF | that it'll be "figure out how to add a query param and test it" then do that a dozen times | 20:19 |
| TheJulia | exactly | 20:19 |
| TheJulia | and I'm done with the first one basically | 20:19 |
| * TheJulia runs tox and waits | 20:19 | |
| TheJulia | so lets see, where did I put that tab with our webapi version history | 20:21 |
| * TheJulia whistles as adding sdk support for stuff | 20:53 | |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/wallaby failed: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892596 | 20:58 |
| TheJulia | I posted a series of patches to address sdk support of the node object | 22:17 |
| JayF | I will h-t them ironic-week-prio as well so we get ironic cores lookinga t them | 22:20 |
| JayF | ah, that may not be enabled in ACLs in openstacksdk; I can't do it | 22:20 |
| JayF | a core or you may be able to | 22:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!