| opendevreview | Julia Kreger proposed openstack/ironic master: CI: minor fix to irmc driver clean_up_instance testing https://review.opendev.org/c/openstack/ironic/+/887239 | 00:31 |
|---|---|---|
| TheJulia | would anyone object to us just reverting the 30 second time limit on tests, or force merging anything to change it? | 00:39 |
| iurygregory | I don't | 00:41 |
| TheJulia | maybe my perception is a red herring, but we see wild performance shifts | 00:42 |
| TheJulia | and the timeout itself seems blocked on freaky weirdness with irmc tests (one of which, seems weird, the other... I think is just a lacking mock) | 00:42 |
| iurygregory | maybe reverting the changes we added specific values to see how CI will behave? | 00:59 |
| TheJulia | perhaps | 01:03 |
| TheJulia | I'm wondering if we just disable the unit with driver | 01:03 |
| iurygregory | makes sense to me | 01:04 |
| TheJulia | iurygregory: https://review.opendev.org/c/openstack/ironic/+/886985 | 01:08 |
| TheJulia | it is v+1 from CI | 01:08 |
| TheJulia | if that fails, I'll disable driver libs job | 01:08 |
| TheJulia | but only makes sense for someone else to wf it | 01:08 |
| iurygregory | on it, let me grab my phone :D | 01:09 |
| TheJulia | Thanks! | 01:27 |
| iurygregory | yw | 01:30 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Change migrations timeout to be >60 seoncds https://review.opendev.org/c/openstack/ironic/+/886985 | 01:32 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Change migrations timeout to be >60 seoncds https://review.opendev.org/c/openstack/ironic/+/886985 | 01:45 |
| iurygregory | seems like now I can access gerrit \o/ | 01:49 |
| iurygregory | the change failed .-. | 01:49 |
| iurygregory | again the IRMCVirtualMediaBootTestCase test >.< | 01:51 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: minor fix to irmc driver clean_up_instance testing https://review.opendev.org/c/openstack/ironic/+/887239 | 01:51 |
| TheJulia | It is just bizarre | 02:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Execute tests by class, not randomly https://review.opendev.org/c/openstack/ironic/+/887166 | 02:03 |
| opendevreview | Merged openstack/ironic master: CI: minor fix to irmc driver clean_up_instance testing https://review.opendev.org/c/openstack/ironic/+/887239 | 02:35 |
| TheJulia | Serious, whiskey tango | 02:53 |
| TheJulia | Ya know, they are all py310. I don’t have 3.10 | 03:01 |
| JayF | I have py310 on my local box, are you doing anything special to repro? | 03:22 |
| JayF | maybe toss it in an etherpad and I'll try it in the morn | 03:22 |
| JayF | or literally anywhere | 03:22 |
| ashinclouds[m] | Well, just off the driver lib one fails so readily | 03:23 |
| JayF | hell, I have py311 too | 03:24 |
| JayF | lets break and fix all the things | 03:24 |
| JayF | better than fixing and fixing and fixing and it's still broken | 03:24 |
| JayF | I'll get a driver libs env setup tomorrow and give it a shot | 03:24 |
| ashinclouds[m] | It is just a rod command, fwiw | 03:25 |
| ashinclouds[m] | Rod | 03:25 |
| TheJulia | Tox | 03:25 |
| * TheJulia glares at autocorrect of the other app | 03:25 | |
| * TheJulia returns to fire while awaiting ice cream | 03:26 | |
| TheJulia | https://usercontent.irccloud-cdn.com/file/7Vyr1Jgf/1688009194.JPG | 03:26 |
| JayF | yeah I'd JFDI if I wasn't on my windows desktop, about to cut off for the night | 03:31 |
| opendevreview | Huy Mai proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 05:27 |
| rpittau | good morning ironic! o/ | 07:01 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 07:40 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 08:00 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 08:13 |
| rpittau | ok, at least it's a failure and not a timeout | 08:19 |
| rpittau | oh I see, it's the problem with the foreign keys pymysql.err.OperationalError: (3730, "Cannot drop table 'nodes' referenced by a foreign key constraint 'bios_settings_ibfk_1' on table 'bios_settings'.") | 08:24 |
| rpittau | JayF, TheJulia, not sure where you were with this ^ | 08:41 |
| opendevreview | Merged openstack/ironic-python-agent stable/train: Use utf-16-le if BOM not present https://review.opendev.org/c/openstack/ironic-python-agent/+/861200 | 09:21 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add the initial skeleton of the agent inspect interface https://review.opendev.org/c/openstack/ironic/+/877814 | 10:06 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Very basic in-band inspection with the "agent" interface https://review.opendev.org/c/openstack/ironic/+/885450 | 10:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 11:13 |
| iurygregory | good morning | 11:20 |
| iurygregory | rpittau, I don't think we saw this (we saw the one for allocation and node) | 11:20 |
| iurygregory | I'm still waking up, but I can push a patch for this | 11:23 |
| iurygregory | unless you want to do it =) | 11:23 |
| * iurygregory brb more coffee required | 11:23 | |
| rpittau | iurygregory: thanks, go ahead, I will approve it | 11:36 |
| * iurygregory is awake now | 11:58 | |
| iurygregory | rpittau, oh I think this was a different thing, I was thinking it was a warning, but is an actual error right? | 11:58 |
| rpittau | yep | 11:59 |
| iurygregory | do you have a link from the failure? | 11:59 |
| iurygregory | https://github.com/openstack/ironic/commit/402c32094b8003db6186f17742a97925405657a6 | 11:59 |
| rpittau | here https://review.opendev.org/c/openstack/ironic/+/885507 | 11:59 |
| iurygregory | I thought this approach from the commit would help | 11:59 |
| rpittau | that's for alloactions, I think this is different | 11:59 |
| iurygregory | yeah, it was just a warning, I though we had the same thing | 12:00 |
| iurygregory | but is an actual error | 12:00 |
| dtantsur | Why, WHY does our BadRequest exception has HTTP code 500??? | 12:31 |
| iurygregory | '-' | 12:31 |
| dtantsur | same Unauthorized. WUT. | 12:32 |
| dtantsur | fortunately, nothing uses the latter | 12:32 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix the HTTP code of the BadRequest exception https://review.opendev.org/c/openstack/ironic/+/887271 | 12:38 |
| dtantsur | this is funny, I wonder what is affected ^^ | 12:38 |
| rpittau | dtantsur: why removing Unauthorized? | 12:44 |
| dtantsur | rpittau: it's not used and has a wrong code | 12:44 |
| dtantsur | (should have mentioned in the commit message, hold on) | 12:44 |
| rpittau | yeah, I asked as it was not mentioned :) | 12:44 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix the HTTP code of the BadRequest exception https://review.opendev.org/c/openstack/ironic/+/887271 | 12:46 |
| dtantsur | rpittau: should be better now ^^ | 12:46 |
| rpittau | thanks! | 12:49 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Correct two mistakes in the /continue_inspection API https://review.opendev.org/c/openstack/ironic/+/887202 | 12:51 |
| dtantsur | Heads-up (cc JayF): metal3 now uses our ramdisk images from tarballs.o.o. Please be gentle :) | 13:09 |
| iurygregory | metal3-integration is broken? .-. | 13:27 |
| dtantsur | I've seen one failure one pass | 13:27 |
| iurygregory | it failed on https://review.opendev.org/c/openstack/ironic/+/883062,node-1 introspection completed but the node is not available .-. | 13:29 |
| iurygregory | planing to recheck, but will avoid since I still need feedback =) | 13:30 |
| rpittau | iurygregory: I haven't seen failures in other patches | 13:39 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 13:44 |
| iurygregory | rpittau, https://zuul.opendev.org/t/openstack/builds?job_name=metal3-integration&project=openstack/ironic | 13:48 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 13:48 |
| iurygregory | the last two failures are identical, haven't look at the other ones | 13:48 |
| JayF | fwiw those tox-with-driver-libs tests are passing pretty regularly on my box | 15:15 |
| JayF | I'm rerunning them over and over but, yeah | 15:15 |
| JayF | that was on 3.11, doing again on 3.10 | 15:17 |
| rpittau | JayF: not passing in CI, we're gtting random failures from migration tests | 15:20 |
| rpittau | example https://9744b8fb79d09b5cacd8-8985f81df699324d9133cf83275f9aa5.ssl.cf1.rackcdn.com/885507/6/check/ironic-tox-unit-with-driver-libs/b96489b/job-output.txt | 15:20 |
| JayF | yeah | 15:21 |
| TheJulia | lets get one of the tox machines held | 15:21 |
| JayF | I'm not sure how much capacity I'll have to help with that today | 15:21 |
| TheJulia | specifically from the driver test | 15:21 |
| TheJulia | I have meetings, but I can help in the background | 15:21 |
| JayF | woke up sick but trying to make it thru :| | 15:21 |
| TheJulia | if I have a machine where I can proroduce maybe | 15:21 |
| JayF | I just need the sudafed to kick in so my head stops throbbing | 15:21 |
| TheJulia | eek :( | 15:21 |
| rpittau | I've seen failures in py38, tried to reproduce locally to no avail | 15:21 |
| rpittau | so failure seems legit | 15:23 |
| rpittau | and it's not only on driver-libs | 15:24 |
| JayF | I wonder if my change made it worse | 15:25 |
| JayF | looking at the failures now vs before | 15:25 |
| TheJulia | we're chasing issues where things are dramatically worse based upon when in the day cycle | 15:47 |
| TheJulia | when CI is more heavily loaded, things fail more horrifically and frequently | 15:47 |
| TheJulia | later in the day, not so much | 15:47 |
| dtantsur | noisy neighbors? | 15:48 |
| JayF | I mean, yeah, very, but that doesn't mean our tests shouldn't run at any speed | 15:49 |
| JayF | I think a fix that goes the direction of "our jobs aren't running fast enough" is likely to get nowhere :/ | 15:49 |
| TheJulia | agree | 15:49 |
| TheJulia | I think we need to slow them down | 15:49 |
| dtantsur | We could potentially split migration tests into a new job | 15:50 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/887166 seems like a reasonable first step to isolate things down by class | 15:50 |
| TheJulia | we *could* also do that... I think | 15:50 |
| TheJulia | We likely should | 15:50 |
| * TheJulia pulls up the stestr docs | 15:51 | |
| JayF | I think that's not a bad idea, but I also am highly suspicious it'd make any difference | 15:51 |
| dtantsur | at the very least, it will allow us to make it non-voting (and avoid merging any db changes) until we figure out | 15:51 |
| rpittau | TheJulia: I don't mind merging that, I'll rebase mine | 15:51 |
| dtantsur | but if we somehow end up choking up mysql, it may help as well | 15:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 15:53 |
| TheJulia | so "stestr run --exclude-regex -- WhateverTheMySQLClassNameIs" | 15:54 |
| TheJulia | and we can do the inverse sort of like the driver one | 15:54 |
| dtantsur | Or literally move the files to another subdirectory | 15:56 |
| TheJulia | so the test runner can't find it? | 15:56 |
| dtantsur | at least that's how some projects distinguish unit and functional tests | 15:57 |
| TheJulia | ... hmmm | 15:57 |
| TheJulia | I'm not sure how that would look pratically, but I'm trying to sort the regex now | 15:57 |
| dtantsur | like https://opendev.org/openstack/openstacksdk/src/branch/master/tox.ini#L56 | 15:57 |
| dtantsur | see you tomorrow folks o/ | 15:59 |
| TheJulia | okay, I'd kind of like us to work towards fixing this eventually and moving files seem like we're admitting some level of defeat | 15:59 |
| TheJulia | but... dunno | 15:59 |
| * TheJulia is running an exclude test now | 15:59 | |
| rpittau | good night! o/ | 16:00 |
| dtantsur | I feel like these are actually a different kind of tests | 16:00 |
| dtantsur | we don't even run them normally locally | 16:00 |
| JayF | I'm going to step away for a bit, I'm fairly miserable and am going to try and get un-miserable so I can be useful later today o/ | 16:00 |
| * dtantsur goes for real | 16:00 | |
| TheJulia | I did at home... | 16:00 |
| dtantsur | oh get better JayF! | 16:00 |
| * TheJulia waits for tox to return | 16:00 | |
| opendevreview | Merged openstack/ironic master: Execute tests by class, not randomly https://review.opendev.org/c/openstack/ironic/+/887166 | 16:12 |
| * TheJulia dances | 16:12 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: Unit tests: Isolate mysql test migrations https://review.opendev.org/c/openstack/ironic/+/887297 | 16:16 |
| TheJulia | so the other issue boils back to what rpittau was encountering, the unwind breaking down | 16:16 |
| TheJulia | which is what Jay and I were looking at yesterday | 16:16 |
| TheJulia | none of that explains the irmc virtual media failures | 16:16 |
| TheJulia | which are the most bizzare failures known to human kind | 16:17 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Unit tests: Isolate mysql test migrations https://review.opendev.org/c/openstack/ironic/+/887297 | 16:23 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/887299 | 16:27 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 16:32 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 16:40 |
| opendevreview | Merged openstack/ironic master: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 16:48 |
| iurygregory_ | woot | 16:58 |
| *** iurygregory_ is now known as iurygregory | 16:58 | |
| iurygregory | TheJulia, we are still having irmc test failures even after your fix? .-. | 16:59 |
| TheJulia | well, attempted one of the tests, the other one everything was mocked | 17:00 |
| TheJulia | it makes zero sense that they both still fail sporatically | 17:00 |
| iurygregory | (╯°□°)╯︵ ┻━┻ | 17:00 |
| TheJulia | yes. Very yes. | 17:01 |
| iurygregory | TheJulia, RHOSP 16.1 is train right? | 17:23 |
| TheJulia | aye | 17:23 |
| iurygregory | have you seen any case where the information from inspection is wrong? like the serial and by_path ? | 17:25 |
| TheJulia | by path is dependent upon initialization order on some machines, but that is basically the case | 17:25 |
| TheJulia | and that is driver and bus order id | 17:25 |
| TheJulia | it varies | 17:26 |
| iurygregory | the fact the person is saying that idrac shows a different serial is making me wonder if they have outdated firmware | 17:31 |
| iurygregory | R750... .-. | 17:32 |
| TheJulia | different device serial numbers? | 17:59 |
| iurygregory | yeah, the one we report from all disks https://paste.opendev.org/show/bCYOGcOfCKdkkGxSN8gW/ | 18:54 |
| TheJulia | WUT https://8b0ae7fc14ab169ab4ff-19e17e5db2abd0e280dd7dd93f01d60e.ssl.cf2.rackcdn.com/887297/2/check/ironic-tox-unit-with-driver-libs/83243cc/testr_results.html | 20:01 |
| TheJulia | iurygregory: so, I'd almost ask for someone to power th emachine down, take photos of the disk and lets see who iw right | 20:03 |
| TheJulia | we mgiht be seeing an adapter | 20:03 |
| TheJulia | dunno! | 20:03 |
| iurygregory | TheJulia, yeah I told the person to try to access the machine and check | 21:30 |
| iurygregory | not WUT but WTF re https://8b0ae7fc14ab169ab4ff-19e17e5db2abd0e280dd7dd93f01d60e.ssl.cf2.rackcdn.com/887297/2/check/ironic-tox-unit-with-driver-libs/83243cc/testr_results.html | 21:31 |
| TheJulia | yeah | 21:39 |
| TheJulia | the only way I could see that happen is if the db base test class was not done initalizing | 21:40 |
| TheJulia | dunno, maybe something like this.... | 21:53 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM check and re-trigger db init for testing https://review.opendev.org/c/openstack/ironic/+/887343 | 21:53 |
| TheJulia | laying down for a little bit to let Ci think on that one | 22:01 |
| *** dtantsur_ is now known as dtantsur | 23:50 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
