| arne_wiebalck | Good morning, Ironic? | 07:46 |
|---|---|---|
| iurygregory | good morning arne_wiebalck and Ironic o/ | 07:55 |
| arne_wiebalck | good morning iurygregory o/ | 07:57 |
| * arne_wiebalck was cut off yesterday :) | 07:58 | |
| iurygregory | why? | 07:59 |
| arne_wiebalck | irccloud issues | 07:59 |
| iurygregory | ouch | 08:00 |
| arne_wiebalck | and meetings :-D | 08:00 |
| iurygregory | I hope you don't have meetings today :D | 08:02 |
| iurygregory | because TGIF! | 08:02 |
| arne_wiebalck | heh, only one | 08:06 |
| arne_wiebalck | yesterday I tried redfish on a node for cleaning | 08:07 |
| arne_wiebalck | it failed, so I may need your help | 08:07 |
| arne_wiebalck | when setting the boot source Ironic got https://paste.opendev.org/show/810963/ | 08:09 |
| arne_wiebalck | I tried with curl, same issue, until I explicitly set the header with '--header 'If-None-Match: ""' | 08:10 |
| arne_wiebalck | known issue? | 08:10 |
| iurygregory | let me check | 08:23 |
| iurygregory | arne_wiebalck, I never saw this error before tbh what type of server ? maybe a FW issue? | 08:24 |
| arne_wiebalck | iurygregory: looks more like the server requires the req to be in a certain format | 08:28 |
| arne_wiebalck | iurygregory: so, the req Ironic sends is not compatible/accepted | 08:29 |
| iurygregory | arne_wiebalck, we saw some interesting things like this but not this one :D | 08:29 |
| arne_wiebalck | iurygregory: could totally be the implementation | 08:29 |
| iurygregory | like for supermicro | 08:29 |
| iurygregory | probably their implementation is wrong and we need to find a workaround.. | 08:30 |
| arne_wiebalck | we will need to see if it is wrong or just using sth Ironic is not expecting | 08:39 |
| arne_wiebalck | maybe dtantsur or mraineri ran into this and have suggestions? | 08:54 |
| arne_wiebalck | yeah, seems like Ironic is missing sth, see the chapter 6.5 on etags in the redfish spec: if the service provides an etag, an if-(none-)match needs to be part of the header | 08:59 |
| janders | good morning Ironic, Happy Friday o/ | 09:31 |
| iurygregory | arne_wiebalck, oh interesting, so maybe we need something in sushy/ironic to handle this =) | 09:37 |
| iurygregory | hey janders o/ | 09:37 |
| arne_wiebalck | hey janders o/ | 09:37 |
| arne_wiebalck | iurygregory: yeah, probably ... would be good to get a better understanding to define the next steps | 09:38 |
| iurygregory | arne_wiebalck, agree, I know some machines have etag but I never saw this happening... maybe we forgot to backport something? which version of ironic you are using? | 09:39 |
| arne_wiebalck | this is on victoria | 09:39 |
| dtantsur | Friday \o/ | 09:40 |
| iurygregory | dtantsur, \o/ | 09:40 |
| dtantsur | arne_wiebalck: yet another creative redfish implementation.. | 09:41 |
| arne_wiebalck | dtantsur: yeah, that is what I think | 09:41 |
| dtantsur | iurygregory: we have basically no etag support yet | 09:41 |
| dtantsur | arne_wiebalck: what's the hardware vendor/model? | 09:42 |
| arne_wiebalck | dtantsur: this is a Gigabyte server | 09:42 |
| iurygregory | yeah, but the thing is that some machines we use does have etag information and we never saw this.. :D | 09:42 |
| dtantsur | because nobody in their right mind would make etags mandatory | 09:42 |
| iurygregory | ++ | 09:42 |
| dtantsur | I guess we can start with a quick workaround, we already use etags in one place | 09:43 |
| iurygregory | we have checks for etag in vmedia | 09:44 |
| dtantsur | https://opendev.org/openstack/sushy/src/branch/master/sushy/resources/manager/virtual_media.py#L153-L155 | 09:44 |
| arne_wiebalck | wouldn't an empty if header be a work-around? | 09:44 |
| iurygregory | yeah | 09:44 |
| dtantsur | arne_wiebalck: I'd be concerned with using anything too creative | 09:44 |
| dtantsur | I think we can do it properly, see the link | 09:44 |
| arne_wiebalck | dtantsur: ++ | 09:44 |
| iurygregory | the yeah was about the link dtantsur pasted :D (I was about to paste the same hehe) | 09:44 |
| dtantsur | :) | 09:45 |
| dtantsur | I guess we should update all PATCH implementations to do this (and hope that nothing breaks) | 09:45 |
| dtantsur | well, ideally, we need to support retries when etag no longer matches... | 09:45 |
| arne_wiebalck | dtantsur: maybe I am not getting how it is supposed to work, but if we retry when the etag does not match, isn't that the same as ignoring the etag? | 09:48 |
| * arne_wiebalck is completely new to etags :) | 09:48 | |
| dtantsur | well, the idea is that you refresh your local copy, see if the change still applies and retry | 09:48 |
| arne_wiebalck | dtantsur: I agree, though, using the etag on patch would be a good first step | 09:49 |
| arne_wiebalck | dtantsur: I see | 09:49 |
| * arne_wiebalck is happy to try patches :-D | 09:50 | |
| arne_wiebalck | another thing I was wondering if this would have been caught by the interop profiles rpioso | 09:50 |
| iurygregory | since this is something that redfish allows I think it should.. | 10:09 |
| iurygregory | if this information is in the redfish spec I mean =) | 10:10 |
| arne_wiebalck | iurygregory, what I meant was: the interop profiles are intended to make sure Ironic can interact with an endpoint; in this case it seems the endpoint is redfish compliant, but Ironic fails to talk to it, so I wonder if the profiles we have/are preparing would flag an issue if run against this hardware | 10:28 |
| iurygregory | the profile is able to detect that the model will use patch? if yes it should fail | 10:29 |
| arne_wiebalck | isn't Ironic using patch to update the boot target? | 10:33 |
| iurygregory | oh my mind was in vmedia workflow :D | 10:34 |
| opendevreview | Verification of a change to openstack/ironic master failed: [iRMC] Convert the type of irmc_port to int https://review.opendev.org/c/openstack/ironic/+/817154 | 11:22 |
| janders | see you next week Ironic o/ | 12:51 |
| janders | have a great weekend everyone | 12:51 |
| dtantsur | see you janders | 12:52 |
| iurygregory | bye janders o/ | 13:08 |
| dtantsur | TheJulia: morning, I'm attempting a reincarnation of your old idea: https://storyboard.openstack.org/#!/story/2009676 | 13:41 |
| TheJulia | braaaaiiins | 13:48 |
| TheJulia | dtantsur: sounds good to me | 13:49 |
| TheJulia | and sounds like the fastest path for single process | 13:49 |
| TheJulia | given locking | 13:49 |
| TheJulia | I *seem* to think the poc patch I had worked aroudn the eventlet nightmare | 13:50 |
| TheJulia | fwiw, if you can find it | 13:50 |
| dtantsur | yep, found it already | 13:51 |
| TheJulia | \o/ | 13:52 |
| TheJulia | okay | 13:52 |
| TheJulia | where was my brain yesterday | 13:52 |
| TheJulia | half in the land of nova-compute | 13:52 |
| TheJulia | and ussuri/train ci | 13:52 |
| TheJulia | PAIN | 13:52 |
| TheJulia | dtantsur: iurygregory: any opinions on killing all builds of tinycore on for ussuri/train? | 13:55 |
| TheJulia | tl;dr, can't build the images anymore, qemu.org has a new ssl cert as of ?Oct 25? which is not recognized as valid | 13:55 |
| dtantsur | sweeet | 13:56 |
| TheJulia | and there is no way for us to patch that code | 13:56 |
| TheJulia | because it was never branched then | 13:56 |
| dtantsur | can we just kill support for ussuri? | 13:56 |
| TheJulia | and infra has flat out rejected it | 13:56 |
| TheJulia | I still need to be able to get patches down to train | 13:56 |
| TheJulia | OR I could only carry patches downstream and we could abandon upstream's train branch | 13:56 |
| TheJulia | abandon | 13:57 |
| TheJulia | I guess worth discussing next week | 14:05 |
| TheJulia | since my only option is to excise tinyipa from ussuri/train branches at this point | 14:05 |
| dtantsur | I'm still for allowing to remove branches in-between | 14:06 |
| dtantsur | JFYI | 14:06 |
| TheJulia | Well, I'd be cool with that | 14:06 |
| TheJulia | less to backport/support | 14:06 |
| TheJulia | and I could still excise tinyipa from train on the repos | 14:07 |
| TheJulia | Added to next week's agenda | 14:09 |
| TheJulia | so my nova patches of great evily goodness *seem* to work, at least in what I can reproduce in CI | 14:09 |
| dtantsur | great evily goodness sounds awesome :) | 14:10 |
| TheJulia | They explicitly update instance.host and compute_node.host, which should prevent orphaning of instances and compute nodes getting dropped | 14:10 |
| TheJulia | as the hash ring rebalances *insert evil laugh here* | 14:10 |
| TheJulia | Of course, the downside to this is I've got checks for "wait, this is inconsistent, hold off! We're going to rebalance again! checks. | 14:11 |
| * TheJulia ponders breakfast | 14:14 | |
| dtantsur | food++ | 14:14 |
| TheJulia | There is a chance I may not be around during the meeting on Monday. It is a travel day for me to get home since we're leaving first thing tomorrow morning and breaking everything in to ~5 hour drives | 14:16 |
| dtantsur | ack | 14:18 |
| dtantsur | safe travels! | 14:18 |
| TheJulia | Thanks! It is the beginning of "windy" season, but the forecasts all look good | 14:19 |
| * TheJulia goes and finds breakfast | 14:20 | |
| iurygregory | TheJulia, let me read all the conversation (I went out for my 1:1 and I just got back :D ) | 15:13 |
| iurygregory | TheJulia, ++ to kill (we do have some dib jobs in ussuri train right?) so we would be fine I guess | 15:20 |
| TheJulia | yeah, we do have some dib jobs | 15:21 |
| TheJulia | just I'm going to have to change metalsmith, ironic, ipa, and ipa-b jobs | 15:21 |
| TheJulia | which is not a big deal | 15:21 |
| TheJulia | but still | 15:21 |
| iurygregory | I can help with that =) | 15:22 |
| iurygregory | so we are totally broken in ussuri train right? | 15:22 |
| TheJulia | Well, I put it on the agenda for next week | 15:22 |
| TheJulia | yeah, we are | 15:22 |
| TheJulia | I already set downstream expectation that it will likely take a week or two to get back to a working train branch | 15:22 |
| TheJulia | which would largely just be switching the jobs over at this point | 15:23 |
| TheJulia | The intertubes hate me today... 1.5 hours expected to download a file | 15:24 |
| TheJulia | 7-8kb/sec sustained | 15:25 |
| TheJulia | err, up to 10! | 15:25 |
| iurygregory | omg =( | 15:40 |
| TheJulia | NobodyCam: launchpad bug 1853009 | 16:04 |
| TheJulia | ... as well. | 16:04 |
| TheJulia | breakfast was a bad idea | 16:21 |
| dtantsur | how so? :( | 16:22 |
| TheJulia | it is very much not settling | 16:23 |
| dtantsur | oh | 16:24 |
| * arne_wiebalck just watched htop for an Ironic driven CPU burn-in on a 256-core machine :-D | 16:37 | |
| * arne_wiebalck needs a bigger screen | 16:38 | |
| opendevreview | Dmitry Tantsur proposed openstack/sushy master: Support for creating and deleting virtual media TLS certificates https://review.opendev.org/c/openstack/sushy/+/817792 | 16:44 |
| arne_wiebalck | dtantsur: for the etag issue, I would think that either patch() checks the resource for it and adds the header; another approach would be to pass the header in for each patch() call after checking for an etag | 16:55 |
| opendevreview | Dmitry Tantsur proposed openstack/sushy-tools master: CertificateService: implement CertificateLocations https://review.opendev.org/c/openstack/sushy-tools/+/817559 | 16:55 |
| arne_wiebalck | the former looks cleaner, the latter easier | 16:55 |
| dtantsur | arne_wiebalck: the latter is what we do for virtual media, right? | 16:55 |
| arne_wiebalck | dtantsur: yes, exactly | 16:55 |
| arne_wiebalck | dtantsur: it would avoid to change the interface to patch() | 16:56 |
| dtantsur | I assume we already have a way to pass headers? | 16:56 |
| arne_wiebalck | dtantsur: yes | 16:56 |
| arne_wiebalck | dtantsur: it is part of patch()`s innterface | 16:56 |
| arne_wiebalck | *interface | 16:56 |
| dtantsur | arne_wiebalck: I don't think it should be done inside the generic patch() call because it does not necessarily know if the request is still valid (and thus can be retried) | 16:56 |
| arne_wiebalck | dtantsur: ok | 16:56 |
| arne_wiebalck | dtantsur: using a wildcard unblocks Ironic in my testing (not a solution, I just wanted to see if that works) | 16:58 |
| dtantsur | okay, good to know | 16:58 |
| arne_wiebalck | dtantsur: I will try passing the etag next | 16:58 |
| arne_wiebalck | where next is probably next week :) | 16:59 |
| dtantsur | makes sense :) | 17:04 |
| arne_wiebalck | bye everyone o/ | 17:41 |
| dtantsur | have a nice weekend folks o/ | 18:03 |
| *** Guest5508 is now known as melwitt | 19:26 | |
| *** melwitt is now known as Guest5716 | 19:27 | |
| *** Guest5716 is now known as melwitt | 19:32 | |
| *** melwitt is now known as jgwentworth | 19:34 | |
| *** outbrito_ is now known as outbrito | 19:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!