| opendevreview | Steve Baker proposed openstack/ironic master: Introduce conductor touch while offline https://review.opendev.org/c/openstack/ironic/+/897375 | 02:26 |
|---|---|---|
| rpittau | good morning ironic! o/ | 06:33 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Bump workers_pool_size to 300 and remove queueing of tasks https://review.opendev.org/c/openstack/ironic/+/897071 | 06:52 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Delay heartbeat after a failure and handle Unavailable https://review.opendev.org/c/openstack/ironic-python-agent/+/897055 | 07:23 |
| opendevreview | Derek Higgins proposed openstack/sushy-tools master: Add new ironic driver https://review.opendev.org/c/openstack/sushy-tools/+/887046 | 08:28 |
| opendevreview | Derek Higgins proposed openstack/sushy-tools master: Add new ironic driver https://review.opendev.org/c/openstack/sushy-tools/+/887046 | 10:46 |
| iurygregory | good morning | 11:53 |
| zigo | Hi there! | 13:24 |
| zigo | I can't build Ironic in Bookworm backports, with many (39 times) failures of this type: | 13:24 |
| zigo | https://paste.opendev.org/show/b5I08UffMzG6BKpJrJtK/ | 13:24 |
| zigo | Any idea? | 13:24 |
| zigo | TheJulia JayF ^ | 13:24 |
| zigo | FYI, the Ironic backport is the last thing I need to do to complete the Bobcat release ! :) | 13:24 |
| TheJulia | Driving like 350 miles atm | 13:25 |
| TheJulia | I’ll look when I can stop | 13:25 |
| rpittau | zigo: hi! let me see if I can find anything | 13:26 |
| zigo | Thanks. | 13:27 |
| zigo | rpittau: It built fine in Unstable, so probably the issue is in a dependency that I didn't update ? | 13:28 |
| rpittau | py3.11 in CI is actually passing | 13:28 |
| rpittau | yeah | 13:28 |
| rpittau | I was thinking the same | 13:28 |
| rpittau | I'm running locally now | 13:28 |
| rpittau | zigo: btw the CI runs on bookworm for 3.11 :) | 13:29 |
| zigo | Nice ! :) | 13:30 |
| rpittau | you may want to have a look here https://8c14ddfe56056e4e35ff-0ca364809dbe3b9250075209fe309a8e.ssl.cf5.rackcdn.com/897071/2/check/openstack-tox-py311/3d3a415/ | 13:30 |
| zigo | rpittau: What should I look for?!? | 13:38 |
| rpittau | installed packages, but I think I can provide you directly the list, jsut gimme a sec | 13:39 |
| zigo | If the result isn't obvious to you, then I can probably get the "pip freeze" in Sid myself and do a diff, but I'm not sure I'll manage to double-guess like this either... | 13:40 |
| zigo | I'll try anyways ! :) | 13:40 |
| rpittau | zigo: there you go https://paste.opendev.org/show/baMGjNph16u16i6tQsWF/ | 13:41 |
| zigo | Cheers. | 13:41 |
| masghar | TheJulia: Dont IRC and drive :) | 13:43 |
| dtantsur | ^^ this ^^ | 13:43 |
| zigo | Here's the diff: https://paste.opendev.org/show/bTtJKI1xoY6UNmqHlcYs/ | 13:45 |
| zigo | masghar: Don't ping someone that's driving ! :) | 13:46 |
| masghar | Didnt think about that uh oh | 13:46 |
| rpittau | zigo: maybe urllib ? teh validate_link function makes use of it and there were some fixes from 1.26.12 and 1.26.15 | 13:53 |
| JayF | OpenInfra Live starting on youtube in about 5 minutes; going over highlights from the Bobcat release :) | 13:55 |
| JayF | zigo: if rpittau doesn't get you taken care of, please file a bug with the details and lets see. | 13:55 |
| zigo | rpittau: I can easily try upgrading urllib, ok ! | 13:56 |
| zigo | Will try in a few minutes ... | 13:56 |
| TheJulia | zigo: :) | 14:13 |
| TheJulia | zigo: good news? | 14:15 |
| zigo | Not yet, in a never-ending meeting ... | 14:15 |
| TheJulia | Joy, sounds like your in good hands | 14:20 |
| TheJulia | I’m going to get a cup of coffee and get back on the road for a little while | 14:20 |
| zigo | I'm going away, will try tomorrow... | 14:26 |
| iurygregory | tks JayF =) | 14:28 |
| JayF | These updates are the easiest 30 minutes of my job every year :D | 14:29 |
| JayF | even if I'm not PTL next cycle just ask and I can play spokesmodel :P | 14:29 |
| iurygregory | why go is so complicated to debug? D: | 14:31 |
| iurygregory | jesus ... | 14:31 |
| JayF | I find python 10000x more difficult to debug, personally | 14:31 |
| JayF | in terms of, when I have to plug it into a debugger, I mean | 14:32 |
| iurygregory | normally the errors are more clear for me in python | 14:32 |
| JayF | I will say most of my golang work was ... simpler problem sets | 14:32 |
| iurygregory | I see the error in Go, I go to that file an check all the calls but I don't see where I should change things XD | 14:32 |
| JayF | I did a little work in Armada but not a lot of heavy lifting | 14:32 |
| JayF | if it's OSS, wanna share the error and a link to the repo? | 14:33 |
| JayF | maybe fresh eyes help even if I'm not a golang-y? | 14:33 |
| JayF | s/a/as/ | 14:33 |
| iurygregory | https://github.com/gophercloud/gophercloud/pull/2795 :D | 14:33 |
| dtantsur | Well, Python rarely has segfaults | 14:33 |
| JayF | lol | 14:33 |
| JayF | dtantsur: you didn't run our big horrible agent patch on top of Juno :D | 14:34 |
| JayF | dtantsur: python segfaults were the #1 cause of ir-conductor losses in the Rax onmetal env | 14:34 |
| dtantsur | iurygregory: there is at least one typo | 14:35 |
| dtantsur | LOL | 14:36 |
| rpittau | L408 :) | 14:36 |
| dtantsur | Speaking of badges: "Python segfaulter" goes to JayF | 14:36 |
| iurygregory | O.o | 14:36 |
| JayF | dtantsur: probably should award that one to jroll more than me :D | 14:36 |
| JayF | dtantsur: I'm not sure I had a lick of code in Ironic at that point; I was the IPA dude :D | 14:36 |
| iurygregory | rpittau, ohhhhhhhhhhh | 14:37 |
| iurygregory | jesus, I need more coffee! | 14:37 |
| JayF | hey, at least it's not like | 14:37 |
| JayF | some actual problem | 14:37 |
| JayF | typos are my favorite kind of bugs in WIP code; because it means my mental model of the system isn't broken | 14:38 |
| iurygregory | ok 3/4 done with L408 :D | 14:38 |
| JayF | > convenience.go:253: Failure in requests_test.go, line 195: at .Firmware.Reason expected "", but got "Driver ipmi does not support firmware (disabled or no implemented)." | 14:39 |
| JayF | the remaining one, I assume? | 14:39 |
| iurygregory | found :D | 14:39 |
| iurygregory | yeah | 14:39 |
| iurygregory | I forgot to update the node to provide this message | 14:39 |
| JayF | \o/ | 14:39 |
| iurygregory | now time to test things on the bifrost setup :D | 14:43 |
| clarkb | a long time ago (I seem to recall debugging this from my HP office in Seattle) a python version (an early 3.x iirc) had a segfault in the garbage collector that you could hit if you constructed references between objects correctly. We had to convince ubuntu that this was a severe enough bug to do a fix against on an LTS | 14:47 |
| JayF | IDK if we ran 2.7 or 3.x for those | 14:49 |
| JayF | but the timing is right for it to have been 3.x | 14:49 |
| opendevreview | Merged openstack/ironic master: Bump workers_pool_size to 300 and remove queueing of tasks https://review.opendev.org/c/openstack/ironic/+/897071 | 15:02 |
| jroll | JayF: I had long forgotten about that, thanks for the flashbacks :D | 15:35 |
| rpittau | good night! o/ | 15:35 |
| JayF | jroll: gonna be honest, I assumed your nick in here was a zombie bouncer lol | 15:40 |
| jroll | it was until it wasn't! :) | 15:41 |
| JayF | jroll: in any event; I hope things are still going well at stripe(?) | 15:41 |
| jroll | yep! good overall, hope things are good over here for everyone as well | 15:43 |
| JayF | Bobcat (2023.2) released yesterday | 15:43 |
| dtantsur | gradually conquering the world, as always | 15:43 |
| JayF | and we're somewhere in the ballpark of Ironic's 10th birthday | 15:43 |
| dtantsur | oh wow | 15:43 |
| JayF | everytime I say that I think another hair in my beard greys | 15:43 |
| dtantsur | that's why I avoid having a beard :D | 15:43 |
| JayF | if I do that, my head looks like Kirby as an infant :P | 15:44 |
| dtantsur | ROFL | 15:46 |
| stevebaker[m] | good morning | 19:22 |
| TheJulia | Good morning | 19:23 |
| TheJulia | Although afternoon now and was driving since like 4:30 AM | 19:23 |
| zigo | I still have all of these failures, even after backporting urllib3 and requests as suggsted rpittau: https://paste.opendev.org/show/bOJn14KImMDHgEdeYP2z/ | 19:49 |
| zigo | So, the issue isn't urllib3 and reuqests ... | 19:49 |
| * zigo tries without tmpfs ... | 19:50 | |
| JayF | zigo: those errors, I think, are angry about stdout? | 20:06 |
| JayF | zigo: are you putting these in some kind of limited environment with no stdout/err attached, perhaps? Or redirecting in a weird way? | 20:07 |
| zigo | Hum... | 20:07 |
| zigo | I'm building in Jenkins ... | 20:07 |
| JayF | bingo | 20:07 |
| JayF | try it locally, I bet it works | 20:07 |
| zigo | It worked on my laptop yeah... | 20:07 |
| JayF | I'm not saying it's not a bug, but it'd be nice to confirm it's environmental | 20:07 |
| zigo | I mean, I didn't try with the same env. | 20:07 |
| zigo | With same repo and all, on bookworm. | 20:07 |
| JayF | I think this is literally print() erroring | 20:08 |
| zigo | But I could build on my laptop with sbuild. | 20:08 |
| zigo | Ok. | 20:08 |
| clarkb | that exception is raised if a non blocking socket would block for the requested operation | 20:08 |
| zigo | Any clue how I could fix? :) | 20:08 |
| JayF | I'm getting there :D | 20:08 |
| zigo | 2>&1 ? :) | 20:08 |
| clarkb | probably need to check why stdout or whatever print() is writing to is set non blocking | 20:08 |
| JayF | So first of all, please bug this behavior | 20:08 |
| JayF | because we sohuld not fail tests in this scenario IMO | 20:09 |
| zigo | I'll try building on my laptop with the same env (sbuild, bookworm, and the jenkins repos). | 20:09 |
| JayF | secondly, https://opendev.org/openstack/ironic/src/branch/master/ironic/tests/unit/api/base.py#L115 is the code that is angry I think | 20:09 |
| zigo | Then I'll tell you if I see the same behavior, ok ? | 20:09 |
| JayF | zigo: worst case; I think if you patch out all the print() from that file, you'd get a test pass | 20:10 |
| zigo | Ok, and since that's a debug thingy, in the tests, then I wont care, right? | 20:10 |
| JayF | exactly | 20:10 |
| zigo | Thanks. Will try all of that. | 20:10 |
| zigo | First, build on my laptop ... :) | 20:11 |
| * JayF wonders if the easiest path to this is env var for "print api requests" that defaults to true | 20:11 | |
| zigo | It wont take long. | 20:11 |
| JayF | zigo: there are 14 prints in Ironic unit tests; not all are in thoe files | 20:11 |
| JayF | and many should likely be removed | 20:12 |
| JayF | but anything of that nature would, definitionally, be unbackportable as a CI-only fix, I think? | 20:12 |
| JayF | okay I need to get lunch before I get trapped down this rabbithole lol | 20:12 |
| JayF | fixing zigo's unit tests way more interesting than finishing my talk slides but the talk has a solid deadline :P | 20:13 |
| clarkb | normally you don't print() to something that is non blocking though | 20:14 |
| * zigo is building ... | 20:15 | |
| zigo | JayF: Same issue on my laptop, let me try removing the print. | 20:18 |
| zigo | JayF: No print, no bug, you naied it !!! :) | 20:23 |
| zigo | Thanks. | 20:23 |
| clarkb | ironic doesn't do the stdout/stderr capturing in the test suite as far as I can tell so that isn't swapping out fds | 20:23 |
| zigo | Now, I still don't understand why building under Unstable doesn't show the issue ... | 20:23 |
| zigo | fds as in file descriptors? | 20:24 |
| zigo | Anyways, I'm *VERY* happy, that makes it a Debian release of OpenStack the 2nd day of the upstream release for me! :) | 20:25 |
| zigo | (this was my very last issue building packages...) | 20:25 |
| clarkb | zigo: yes file desciptors. Some projects swap out stdout and stderr with buffers to capture prints and attach them to the subunit stream. Otherwise the prints will be all interleaved between tests can be very messy | 20:26 |
| clarkb | but I don't see ironic doing that. It is weird that if this is actually stdout that it would be non blocking for some reason | 20:26 |
| zigo | Oh, looks like I also had the issue in unstable, at least it shows in the buildd: https://buildd.debian.org/status/fetch.php?pkg=ironic&arch=all&ver=1%3A23.0.0-1&stamp=1696497516&file=log | 20:36 |
| * zigo fixes it... :) | 20:37 | |
| zigo | Fun fact: the number of failures differ on each build ... | 20:38 |
| JayF | clarkb: zigo: Yeah honestly I have no idea how this could be triggered tbh; but it's clearly a thing | 20:53 |
| opendevreview | Steve Baker proposed openstack/ironic master: Introduce conductor touch while offline https://review.opendev.org/c/openstack/ironic/+/897375 | 21:44 |
| opendevreview | Steve Baker proposed openstack/ironic master: Implement drain shutdown support https://review.opendev.org/c/openstack/ironic/+/897496 | 21:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!