| janders | TheJulia JayF iurygregory given the time difference, Friday off for most of RHAT and some downstream branching deadlines, I'd very much appreciate feedback on https://review.opendev.org/c/openstack/ironic/+/888746 if you have time. This is a fix needed for Cisco machines to be able to provision off vMedia (more details in reno and linked bug). | 00:06 |
|---|---|---|
| janders | Thanks in advance! | 00:06 |
| JayF | about to dinner but i opened it in a tab | 00:13 |
| janders | ty! | 00:14 |
| TheJulia | a grand total of one database is locked so far today | 00:17 |
| janders | thank you for the review TheJulia o/ | 00:23 |
| * TheJulia looks at the weather model for this weekend and reconsiders plans | 00:25 | |
| iurygregory | I'm finishing the review in a few min o/ | 00:26 |
| TheJulia | if https://review.opendev.org/c/openstack/ironic/+/891872 ends up fixing the database is locked issues for the most part, I'm going to be blown away | 00:27 |
| opendevreview | Merged openstack/bifrost stable/2023.1: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/890789 | 00:34 |
| JayF | I wish we had API response time metrics hooked up somehow in CI | 00:35 |
| JayF | it'd be interesting to see if there is real-world impact for the more usual use cases | 00:35 |
| iurygregory | 891872 looks interesting | 00:37 |
| JayF | janders: Has this been tested against other brands of hardware? | 00:40 |
| JayF | janders: I wonder if we should line it up behind an ironic change w/git-sushy installed to get 3rd party CI to run and pass | 00:40 |
| JayF | TheJulia: iurygregory ^ do you know the answer? | 00:41 |
| JayF | I guess HPE 3rd party CI looks broke anyway :( | 00:41 |
| JayF | not a lot we can do about it then | 00:41 |
| iurygregory | was about to say that we could wait for HPE but seems really broken | 00:42 |
| iurygregory | but this shouldn't break any deployments | 00:42 |
| JayF | I'll make a note to contact that team about their failing third party ci | 00:42 |
| iurygregory | considering the logic added was more to handle the Cisco HW with redfish =( | 00:42 |
| iurygregory | and we had them testing before proceeding with this approach upstream | 00:43 |
| JayF | my only real concern was wanting to know if it was tested against anything but the target hardware, I don't need to see proof upstream for that | 00:43 |
| JayF | but tbh, if the third party CI is neglected to the point it's not running; there's not a lot we can do and it's silly to leave someone broken | 00:43 |
| iurygregory | I know janders is testing with redfish mockup of different HW =) | 00:44 |
| JayF | hehe | 00:44 |
| iurygregory | but I don't remember the full details on it | 00:44 |
| iurygregory | it's like a mockup of the whole BMC of Dell Machine etc | 00:44 |
| JayF | nice | 00:45 |
| JayF | honestly, some of what you're seeing is being damaged by the ipmi era | 00:45 |
| iurygregory | he went out for some time to help his wife, he will be back in a few I think | 00:45 |
| iurygregory | yeah | 00:45 |
| JayF | I mean, it's landing already, there's no way upstream to ask him to do what I want + you have a customer broken in front of you | 00:45 |
| JayF | I'm glad we talked about it; if I had gotten to it before it landed; I probably would've put this on a comment and clogged it up | 00:46 |
| iurygregory | I just learned a new word in english O.o clogged | 00:47 |
| iurygregory | had to google for it | 00:47 |
| JayF | clog is a good word, it sounds like what it is lol | 00:47 |
| JayF | and glug is the sound it makes when you unclog a pipe :D | 00:47 |
| iurygregory | perfect! | 00:48 |
| JayF | (when the big air bubble comes out and the water gurgles; it's a glug) | 00:48 |
| janders | sorry been AFK back now and catching up on thread | 01:11 |
| janders | I should have an R640 lab to test this on, good point | 01:11 |
| JayF | it's already landing, if you wanna test it for peace of mind I obviously wouldn't object :) | 01:12 |
| janders | I am not concerned (due to what iurygregory said with mockups - we have an internal repo with BMC mirrors we vetted this against) but there's code inspection and then there's actual test | 01:12 |
| JayF | realistically, if it breaks someone's hardware, they should've had third party ci :/ | 01:12 |
| janders | yeah I like the idea of peace-of-mind test, will try to make a container with this fix together with in-flight secureboot fix and see if both vmedia and secure boot handling still works | 01:13 |
| janders | SecureBoot fix is this one for context: https://review.opendev.org/c/openstack/ironic/+/863999 | 01:13 |
| janders | TL;DR this needs a LOT of reboots to work | 01:13 |
| janders | but it does work and even both ways (no one asked for support for disabling secure boot but being able to do that is a life saver for testing) | 01:14 |
| JayF | fwiw; I think using wall-time for timeouts is discouraged | 01:14 |
| JayF | because if the clock changes it changes your timeout | 01:14 |
| JayF | don't take that as an actual review comment as I'm just skimming, but one of my "stop and read carefully" flags in reviewing code is the word "timeout" and time.time() in close proximity :D | 01:15 |
| janders | noted, thanks Jay | 01:15 |
| janders | this one will need more work (hopefully later today) so will try factor this feedback in | 01:15 |
| JayF | I put myself on as a reviewr, I'll try to look at it and offer concrete advice if there's something I don't like | 01:15 |
| janders | since Iury mentioned BMC mockups I figured I may share more detail | 01:16 |
| janders | DMTF have a tool that "mirrors" BMC, it is mostly used for demonstrating proposed Redfish enhancements | 01:16 |
| janders | but it can be used to just mirror the whole BMC so that we can have reference of all its responses without having a server in the lab | 01:17 |
| JayF | I imagine that's in the category of that publically navicable redfish api mockup | 01:17 |
| JayF | just more interactive? | 01:17 |
| janders | I collect those for different models and store them in an internal repo | 01:17 |
| janders | super-useful | 01:17 |
| janders | reason why it's internal is IP concerns | 01:17 |
| janders | (not sure if justified, just being careful) | 01:17 |
| JayF | I am always 1000x more careful with someone elses' information | 01:17 |
| janders | but I wonder if it would make sense to have some sort of an upstream initiative like this | 01:18 |
| janders | "want your BMC supported by Ironic? give us a mockup!" | 01:18 |
| janders | (and consent it to be on opendev) | 01:18 |
| janders | WDYT? | 01:18 |
| JayF | if you want that, you have to be able to feed it into sushy-tools, yeah? | 01:18 |
| janders | so far I was just using it as a standalone reference | 01:18 |
| JayF | sushy-tools -> emulate this specific BMC that is specified here | 01:18 |
| janders | coding something up, thinking "how would a BMC XXX react to this code change" | 01:19 |
| JayF | my brain goes CI first | 01:19 |
| janders | and then I look at relevant BMC responses and I know with a good probability whether I will break it or not | 01:19 |
| JayF | so I saw visions of ironic-redfish-brandname123-mockup jobs :D | 01:19 |
| janders | making those mockups run through sushy-tools and actually respond to queries would be a much bigger project | 01:19 |
| JayF | Well I only know about them in concept, so I have no idea what is possible or what is not | 01:20 |
| JayF | I'm just imagining cool possibilities :D | 01:20 |
| janders | yeah | 01:20 |
| janders | we kinda did things like this internally at small scale for some bugs | 01:20 |
| janders | ("break sushy-tools to match broken hardware") | 01:20 |
| janders | (and develop fix against it) | 01:21 |
| JayF | it's a clever approach | 01:21 |
| janders | worked a charm but if it could just "eat" a mockup then hell yeah even better | 01:21 |
| JayF | well, is the mockup a document | 01:21 |
| JayF | or like... a fixture | 01:21 |
| JayF | literally like "I can point at http://localhost:123/redfish and dude I got a Dell!" or just a spec | 01:21 |
| janders | it's a directory structure with json files mirroring BMC responses to every GET query you can send it | 01:21 |
| JayF | so could trivially be wired up to a read only redfish api | 01:22 |
| JayF | interesting | 01:22 |
| JayF | at a minimum, using that information to build knobs on sushy-tools to change behavior to create the matrix would be cool | 01:22 |
| JayF | I guess I'm assuming that the differences are logical and make sense, which may not be the case | 01:22 |
| janders | for the record we had a quick chat with Jay about the mockup things in PM (due to some code sharing) and it sounds like it may be a PTG topic which I am happy to pick up | 01:34 |
| janders | there may be some potential there to make things easier for inter-op | 01:34 |
| opendevreview | Merged openstack/ironic master: Retry connecting vmedia through a DVD device if available https://review.opendev.org/c/openstack/ironic/+/888746 | 02:10 |
| janders | \o/ | 02:11 |
| janders | baremetal testing to follow | 02:11 |
| opendevreview | Merged openstack/ironic master: Don't hold on to the API request dbapi https://review.opendev.org/c/openstack/ironic/+/891872 | 02:23 |
| *** dmellado8191813 is now known as dmellado819181 | 05:06 | |
| janders | JayF as per our earlier discussion with Iury I tested https://review.opendev.org/c/openstack/ironic/+/888746 on real hardware (Dell R640) - all good: https://paste.openstack.org/show/b23jRtwEjrZZtyw1xtPr/ | 06:15 |
| janders | (this was using metal3 hence the extra complexity but the code changes can be seen inside the ironic container) | 06:15 |
| janders | thanks for your help and for a great chat earlier! :) | 06:16 |
| rpittau | good morning ironic! o/ | 07:23 |
| mmalchuk | morning Ironic! o/ | 07:36 |
| mmalchuk | TheJulia please remove -W from https://review.opendev.org/c/openstack/bifrost/+/891134 | 09:36 |
| mmalchuk | it stuck | 09:36 |
| iurygregory | good morning Ironic | 11:49 |
| iurygregory | do we have a fix for inspector client with functional tests broken? | 12:28 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/python-ironic-inspector-client master: [WIP] Test Fix https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/891976 | 12:32 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/python-ironic-inspector-client master: [WIP] Test Fix https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/891976 | 12:34 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/python-ironic-inspector-client master: [WIP] Test Fix https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/891976 | 12:53 |
| iurygregory | ok, at least now I'm getting a real failure | 13:10 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/d3de4a8b86a04c6b9c1f685fd3d0c370/log/job-output.txt#1978 | 13:11 |
| TheJulia | rutro, that is weird | 13:15 |
| iurygregory | in patch 2 I got database locked issue lol | 13:16 |
| TheJulia | ... for inspector? | 13:16 |
| iurygregory | yeah inspector client https://zuul.opendev.org/t/openstack/build/6d0b54d54c9a48509a57dac6011675c6/log/job-output.txt#1835 | 13:16 |
| iurygregory | the diff from 2 to 3 is the hash commit from inspector | 13:17 |
| TheJulia | omg | 13:18 |
| iurygregory | I'm looking at inspector to see if we changed something that could cause the functional test to fail | 13:19 |
| TheJulia | at a glance, it looks like it expects a exception and got a 202... | 13:21 |
| TheJulia | ... a functional test really shouldn't expect a failure though | 13:21 |
| TheJulia | ... I'm not sure we've done much with inspector this cycle | 13:22 |
| iurygregory | yeah | 13:22 |
| iurygregory | but we haven't updated the hash in a while .-. | 13:22 |
| iurygregory | last time was wallaby =( | 13:22 |
| iurygregory | Last updated: October 27, 2020 (Wallaby cycle). | 13:23 |
| iurygregory | TheJulia, maybe https://github.com/openstack/ironic-inspector/blob/53778d62f16c446e2861877f20a91ac3ba6940da/ironic_inspector/main.py#L408 ? | 13:29 |
| iurygregory | maybe we need to update the functional test to reflect this | 13:30 |
| * iurygregory forgot to ask if TheJulia is working.. since today is recharge day.. and say good morning =) | 13:31 | |
| TheJulia | heh | 13:41 |
| TheJulia | not really working today, fwiw | 13:41 |
| TheJulia | I need to take the wife to work and we're going up the mountain today | 13:41 |
| TheJulia | at least for overnight | 13:41 |
| TheJulia | we're supposed to get a hurricane to hit our area | 13:42 |
| TheJulia | on Sunday/Monday | 13:42 |
| TheJulia | iurygregory: likely, it could just be the functional test on the client is horribly broken | 13:42 |
| iurygregory | oh wow, be safe! | 13:42 |
| iurygregory | yeah | 13:43 |
| TheJulia | tl;dr your on the right track | 13:43 |
| TheJulia | I think fixing the inspector functional test makes sense since 202 is, as we know, the right response | 13:43 |
| TheJulia | previously it might have raised an exception for some silly reason | 13:43 |
| TheJulia | wow, it is a category 3 right now | 13:45 |
| TheJulia | gusting to 150 | 13:45 |
| opendevreview | Merged openstack/bifrost stable/xena: Create the log file for the disk-image-create command https://review.opendev.org/c/openstack/bifrost/+/891134 | 13:48 |
| iurygregory | ack, I will try to update the test | 13:51 |
| iurygregory | oh wow 150 O.o | 13:51 |
| TheJulia | okay, i'll check the computer intermittently today | 13:56 |
| iurygregory | interesting we haven't landed this in stable/zed https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/856419 | 14:30 |
| JayF | TheJulia: I was about to give you the NC-resident-hurricane-prep spiel and deleted it | 14:31 |
| JayF | TheJulia: sometimes I forget how long we've known each other :D | 14:31 |
| JayF | hurricanes are scary in the whole but the most fun storm to just like, stand outside and feel yourself blow around | 14:32 |
| TheJulia | We will just get rain… downside is my home is in a flood plain | 14:40 |
| JayF | oh no | 14:41 |
| rpittau | gosh stay safe TheJulia | 14:42 |
| TheJulia | We have an elevated slab by 2 feet which helps a ton, but roads may not be quickly passable | 14:43 |
| TheJulia | And depending on how much rain falls on the mountains will also dictate how things progress | 14:46 |
| rpittau | btw I've updated the version of ironic on the website https://github.com/OpenStackweb/ironic-website/pull/59 | 14:49 |
| TheJulia | Cool | 14:53 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: [WIP] Add centos9 based job https://review.opendev.org/c/openstack/metalsmith/+/869374 | 15:03 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: [WIP] Add centos9 based job https://review.opendev.org/c/openstack/metalsmith/+/869374 | 15:03 |
| rpittau | so this ^ is working \o/ | 15:04 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: Add centos9 based job https://review.opendev.org/c/openstack/metalsmith/+/869374 | 15:04 |
| rpittau | ready for reviews | 15:04 |
| iurygregory | ack | 15:06 |
| rpittau | bye everyone, have a great weekend! o/ | 15:18 |
| opendevreview | Merged openstack/bifrost master: Revert "Pin systemd to 252-15 for CentOS Stream 9" https://review.opendev.org/c/openstack/bifrost/+/891014 | 17:38 |
| JayF | TheJulia: holy crap, epiphany: we don't need TFTP to run CI on OVN. | 19:56 |
| JayF | TheJulia: we can setup the test VMs to use ipxe roms, then serve up an ipxe config to an http-serving ironic endpoint | 19:57 |
| * JayF is pretty sure ipxe supports getting initial config over http | 19:57 | |
| JayF | if you have to tftp over a small config first that'll be very annoying | 19:57 |
| ashinclouds[m] | We already do that actually… | 19:57 |
| ashinclouds[m] | Just not for all jobs | 19:57 |
| JayF | I don't think it's a pattern we should use for all jobs (we need to cover tftp setup), but it could allow us to use OVN jobs for many things | 19:58 |
| JayF | IDK if this is net new information; I was just thinking about it and thought I'd share | 19:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!