| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Explicitly close out test connection https://review.opendev.org/c/openstack/ironic/+/887343 | 00:56 |
|---|---|---|
| TheJulia | I'm curious if ^^^ is our orphaned connections | 00:56 |
| TheJulia | if anyone can reproduce the deadlocked state locally still, please give it a spin, if it works the base patch, I'll move it up to master branch and we can retest | 00:57 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Explicitly close out test connection https://review.opendev.org/c/openstack/ironic/+/887343 | 01:12 |
| TheJulia | eh, minor small tiny itty bitty fix | 01:13 |
| TheJulia | repeat! | 01:13 |
| TheJulia | oh, and it passed | 01:17 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Explicitly close out test connection https://review.opendev.org/c/openstack/ironic/+/887343 | 01:38 |
| * iurygregory looks | 02:26 | |
| TheJulia | Of course the 30 second timeout | 02:58 |
| iurygregory | where? .-. | 03:00 |
| ashinclouds[m] | Py38 failed on the db migrations | 03:00 |
| TheJulia | Basically we’re on the right path it seems… | 03:01 |
| iurygregory | in 887343 ? | 03:01 |
| iurygregory | oh py39 failed | 03:02 |
| iurygregory | https://3737a6b1ea4fc714d01b-164d5a3be9b73b43cf7bfb5d1ec3d41f.ssl.cf1.rackcdn.com/887343/4/check/openstack-tox-py39/7da4970/testr_results.html | 03:02 |
| TheJulia | Yeah, we likely out to recheck it again, but we need to get the timeout fix merged along with likely this fix. Splitting MySQL migrations into its own likely would help | 03:02 |
| TheJulia | We just need to unblock ourselves to make progress | 03:03 |
| TheJulia | Oh, sorry, thought it was 38, just finished dinner and glanced on my phone | 03:03 |
| iurygregory | I'm thinking in just "remove" the timeout we added to see | 03:03 |
| TheJulia | Sounds good to me! | 03:03 |
| iurygregory | let's see the results we get from the recheck you did on https://review.opendev.org/c/openstack/ironic/+/886985 | 03:04 |
| TheJulia | Yeah, might have better luck finally | 03:06 |
| iurygregory | ironic-tox-unit-with-driver-libs and openstack-tox-py310 failed | 03:23 |
| iurygregory | TestMigrationsMySQL.test_walk_versions StringException on py310 | 03:24 |
| TheJulia | Heh | 03:24 |
| iurygregory | and driver failed with irmc | 03:24 |
| iurygregory | ;-; | 03:24 |
| TheJulia | So split still I guess | 03:25 |
| iurygregory | I'm starting to thing we should squash commits to see if they would work together lol | 03:25 |
| iurygregory | or use a chain of patches before squashing | 03:25 |
| iurygregory | otherwise we will be with the chicken egg problem | 03:26 |
| TheJulia | yeah | 03:27 |
| TheJulia | so which two or three?! | 03:30 |
| iurygregory | I'm considering 2 | 03:31 |
| TheJulia | seems like the timeout is one, we could do the mysql split out and the aformentioned chnage together, that way we isolate the mysql stuff for now, we hopefully address some root causes, that leaves the irmc variable, but I *suspect* irmc issues might be more how the base db helper just ends up | 03:31 |
| * TheJulia takes the corgi overlord for a quick outing | 03:31 | |
| iurygregory | 886985 -> 887343 | 03:32 |
| iurygregory | 887297 makes a lot of sense also, but if the two together works we probably don't need :thinking: | 03:32 |
| iurygregory | but the idea to run tests by class makes a lot of sense | 03:33 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: CI: Explicitly close out test connection https://review.opendev.org/c/openstack/ironic/+/887343 | 03:41 |
| iurygregory | let us see how CI will behave for 887343 when the parent is 886985 =) | 03:41 |
| iurygregory | avoid adding 887297 now because it will require a lot of CI (since we are adding a new job and we run all our jobs) | 03:42 |
| TheJulia | yeah | 03:43 |
| iurygregory | driver failed IRMCVirtualMediaBootTestCase | 03:59 |
| iurygregory | '-' | 03:59 |
| TheJulia | perhaps we disable that test class for now? | 03:59 |
| iurygregory | (╯°□°)╯︵ ┻━┻_StringException | 03:59 |
| iurygregory | makes sense to me, the failure is just weird tbh | 04:01 |
| TheJulia | no value no exception | 04:01 |
| TheJulia | my theory has been with the irmc stuff that it has something to do with db states, given a third one failed with db-ish failure | 04:03 |
| iurygregory | yeah | 04:03 |
| iurygregory | maybe the one splitting migrations to other job would help? | 04:04 |
| iurygregory | or maybe we are missing something in their class | 04:04 |
| TheJulia | the split does seem to help, at least it moves out a wildcard job of sorts | 04:05 |
| TheJulia | we've got two sort of different db issues going on here it seems | 04:05 |
| TheJulia | one in general with mysql which we see, and then db unit tests | 04:06 |
| TheJulia | well, db enabled unit tests | 04:06 |
| iurygregory | yup | 04:08 |
| iurygregory | going to sleep now, will try to look at things in the morning | 04:14 |
| iurygregory | o/ | 04:14 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Disable irmc virtual media test_prepare_instance_with_secure_boot https://review.opendev.org/c/openstack/ironic/+/887353 | 04:24 |
| rpittau | good morning ironic! o/ | 07:10 |
| rpittau | so irmc unit tests with secure boot are not really happy, now test_clean_up_instance_with_secure_boot is failing | 07:24 |
| rpittau | TheJulia, iurygregory, the idea of squashing some commits is not bad, I would probably just do a big "fix CI" commit with the 3 patches 886985 887343 887353 and disable the other irmc test that is failing for the time being | 07:25 |
| iurygregory | rpittau, agree | 11:00 |
| iurygregory | do you want to do it? I can approve | 11:00 |
| iurygregory | good morning Ironic | 11:00 |
| drannou | Hello | 12:05 |
| drannou | Quick question: did you ever think/talk about disk encryption management on Ironic ? | 12:05 |
| dtantsur | drannou: I've definitely heard this topic before. Nothing specific has been proposed, to my best knowledge. | 12:16 |
| dtantsur | You can definitely build something site-specific using deploy steps. | 12:16 |
| rpittau | iurygregory: I can do it | 12:30 |
| iurygregory | ack | 12:31 |
| drannou | dtantsur: I'm think about "something", may be IPA, that would "manage" SED feature | 12:43 |
| dtantsur | drannou: you may need to elaborate on "SED features", I'm personally not familiar with them. What would IPA do? | 12:45 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix multiple things in CI https://review.opendev.org/c/openstack/ironic/+/887373 | 12:46 |
| rpittau | iurygregory, TheJulia ^ | 12:46 |
| * TheJulia hopes | 12:47 | |
| TheJulia | Also, good morning! | 12:48 |
| opendevreview | Huy Mai proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 12:48 |
| TheJulia | Note to self: don’t press the “all on” button in the rv again | 12:49 |
| rpittau | lol | 12:56 |
| rpittau | good morning TheJulia :) | 12:56 |
| iurygregory | how many rechecks we should do? :D | 13:03 |
| rpittau | gosh I hope not even one | 13:06 |
| TheJulia | So I noticed the opportunistic test classes in oslo.db got deprecated… | 13:07 |
| TheJulia | But in 13.0.0 | 13:08 |
| TheJulia | We’re like 12.3.1 | 13:08 |
| rpittau | yep | 13:08 |
| rpittau | uc still need to be udpated | 13:08 |
| rpittau | aaaand it's green | 13:08 |
| rpittau | TheJulia: hope you don't mind I squashed everything in a single patch | 13:09 |
| iurygregory | it's green but let's do some rechecks to see | 13:10 |
| TheJulia | rpittau: not at all | 13:17 |
| TheJulia | at this time of day, just workflow | 13:17 |
| TheJulia | we see a higher failure rate during EU daytime | 13:17 |
| TheJulia | at least, that has been my perception | 13:17 |
| rpittau | could be, higher load ? | 13:19 |
| TheJulia | That is my gut feeling, but we don't seem to *really* graph anything that would allow us to really corrilate time of day to cloud provider to failure rate | 13:19 |
| TheJulia | A very long time ago, pre-cloud, we were able to time the arrival of school busses near a court trial site, to their internet access going completely sideways at the trial site | 13:20 |
| TheJulia | turns out, add ~15 minutes after a bus arrived, and the bandwidth just disappeared | 13:20 |
| TheJulia | i.e. kids got home, and fired up netflix | 13:21 |
| TheJulia | similar, in early cloud, we were actually able to graph out ripples in AWS US-EAST-1 as the API degraded and you couldn't provision new servers around those time windows | 13:21 |
| TheJulia | best guess, someone was scaling stuff up as each US time zone's kids would start to get home, hardest hit was always US-EAST | 13:22 |
| TheJulia | The sort of maddening issues which drive engineers to drink. A lot. | 13:22 |
| * rpittau grabs a beer | 13:26 | |
| dtantsur | wow | 13:33 |
| JayF | If you need me for something today, please ping directly. I doubt I'll make it a full working day as I have a severe sinus infection | 13:55 |
| Sandzwerg[m] | get well soon | 14:03 |
| opendevreview | Merged openstack/ironic master: Fix multiple things in CI https://review.opendev.org/c/openstack/ironic/+/887373 | 14:15 |
| iurygregory | \o/ | 14:16 |
| rpittau | wonderful | 14:18 |
| iurygregory | we probably want to still isolate mysql tests right? :thinking: https://review.opendev.org/c/openstack/ironic/+/887297 | 14:19 |
| iurygregory | rechecked https://review.opendev.org/c/openstack/ironic/+/883062 - the failure from metal3 seems unrelated | 14:20 |
| rpittau | my internet connection today is slower than a drunk snail | 14:21 |
| iurygregory | drunk snail LOL | 14:22 |
| JayF | Sandzwerg[m]: thanks | 14:38 |
| TheJulia | rpittau: did the snail get into the wine?!? | 14:39 |
| JayF | nice job landing that multi-fix everyone, I wish I could've felt up to helping out some | 14:39 |
| rpittau | TheJulia: no, but it got to our basil plant | 14:39 |
| TheJulia | iurygregory: yeah, I'm not sure we've entirely pinned down precisely what is going on, it could also be related to the base db test case, I... would need to sit down and focus on it to see how it wires up the db again and if they play together, or not | 14:40 |
| TheJulia | rpittau: ugh | 14:40 |
| iurygregory | gotcha, let's see how CI will be behaving with the fixes | 14:41 |
| TheJulia | the irmc failures just have me stumped | 14:41 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Utilize the JSON-RPC port https://review.opendev.org/c/openstack/ironic/+/879215 | 14:50 |
| TheJulia | so we have 3 patches now running it looks like on the current state of master, on top of isolating mysql down specifically | 14:51 |
| TheJulia | I guess we wait and see | 14:51 |
| TheJulia | unit tests all green on four jobs, \o/ | 15:09 |
| iurygregory | \o/ | 15:15 |
| rpittau | time to leave for the weekend then :D | 15:17 |
| rpittau | have a great weekend everyone! o/ | 15:17 |
| TheJulia | ironic.tests.unit.common.test_images.IronicImagesTestCase.test_image_to_raw <-- just crashed out locally :\ | 16:20 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 16:21 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Utilize the JSON-RPC port https://review.opendev.org/c/openstack/ironic/+/879215 | 16:27 |
| TheJulia | actually had issues in the change I needed to fix | 16:27 |
| TheJulia | \o/ | 16:27 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 16:37 |
| TheJulia | i still need to see if I can combine the handlers on that, after lunch maybe. Running into town real quick | 16:37 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add additional logging on iLO power failure https://review.opendev.org/c/openstack/ironic/+/885549 | 19:50 |
| TheJulia | and we have reached the point of the day where the cat declares it is 5pm | 19:58 |
| TheJulia | regardless of the time | 19:58 |
| JayF | I have been reading scrollback, and thank you all for the effort to fix up the unit tests | 19:58 |
| JayF | even if my sinuses don't feel better, my brain does :D | 19:58 |
| iurygregory | CI seems to be behaving fine in master, finally got +1 from zuul again in https://review.opendev.org/c/openstack/ironic/+/883062 \o/ | 21:16 |
| JayF | I have a hypothesis that the more times we say things are okay, the more likely they are to fail. So I'm just taking the victory in silence 🤐😂😂 | 21:18 |
| iurygregory | well, we still need to deal with stable branches... | 21:20 |
| iurygregory | =X | 21:20 |
| NobodyCam | good afternoon Ironic folks. and ofc... | 21:27 |
| NobodyCam | TGIF! | 21:27 |
| iurygregory | hey hey NobodyCam | 21:31 |
| iurygregory | TGIF | 21:31 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 21:36 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Enable vendor interfaces to be called as steps https://review.opendev.org/c/openstack/ironic/+/879089 | 21:36 |
| TheJulia | whee, looks like we have to fix the migrations for sqlalchemy 2.0 too | 21:55 |
| TheJulia | https://f4a7bc5fd40b3ad7edfa-284200bc20166be0f82616d8d2b3e580.ssl.cf5.rackcdn.com/886020/7/check/ironic-tox-unit-with-sqlalchemy-2x/2a4f696/testr_results.html | 21:55 |
| TheJulia | luckilly, just one thing | 21:55 |
| iurygregory | good | 21:56 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add wait step https://review.opendev.org/c/openstack/ironic/+/879061 | 22:01 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/ironic/+/886020 | 22:06 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix db migration test for sqlalchemy 2.0 https://review.opendev.org/c/openstack/ironic/+/887432 | 22:06 |
| TheJulia | I think I'm basically done for the day, ping me if I'm needed but after a few late evenings I need to unwind | 22:07 |
| iurygregory | I will only ping for reviews on monday :D, today is friday | 22:10 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Firmware Interface https://review.opendev.org/c/openstack/ironic/+/885276 | 22:11 |
| iurygregory | have a great weekend everyone o/ | 23:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!