| opendevreview | Merged openstack/ironic master: Unit tests: Isolate mysql test migrations https://review.opendev.org/c/openstack/ironic/+/887297 | 01:13 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: Add DB API for Firmware and Object https://review.opendev.org/c/openstack/ironic/+/883062 | 03:17 |
| rpittau | good morning ironic! o/ | 05:46 |
| rpittau | TheJulia: I'm not sure about the root cause, it could be dnsmasq, I'm going to remove the pin to focal on top of your CI patch to see if it works | 05:47 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: CI: move snmp pxe job to jammy https://review.opendev.org/c/openstack/ironic/+/888112 | 05:49 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add the initial skeleton of the agent inspect interface https://review.opendev.org/c/openstack/ironic/+/877814 | 06:45 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Very basic in-band inspection with the "agent" interface https://review.opendev.org/c/openstack/ironic/+/885450 | 06:45 |
| opendevreview | Ebbex proposed openstack/bifrost master: Fix schema[meta] linter warnings https://review.opendev.org/c/openstack/bifrost/+/874855 | 06:46 |
| dtantsur | Folks, would really appreciate reviews on https://review.opendev.org/q/topic:inspector+status:open please. | 06:51 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector https://review.opendev.org/c/openstack/bifrost/+/887934 | 06:56 |
| JayF | Going to note that removing the "os=False" from eventlet does pass all our tests, it seems: https://review.opendev.org/c/openstack/ironic/+/887996 | 08:30 |
| JayF | not sure if we should do it; I don't remember the historical reason we did it but it may not apply to modern eventlet | 08:30 |
| JayF | and everytime we have a "it just stopped" kind of bug, I wonder if this has come back to bite us, because that's exactly what we risk when we disable eventlet in some modules | 08:32 |
| * JayF is now curious to lookup what is patched in os for eventlet | 08:32 | |
| frickler | rpittau: just being curious, where was the next ptg announced? didn't see anything on any of the ML | 08:34 |
| eandersson | TheJulia: Did you try specifying the version when installing? In my example I didn't have to do that because I was installing a package that didn't exist in the main repo | 08:44 |
| eandersson | Actually nvm I tried and I can't get it working | 08:52 |
| opendevreview | Merged openstack/ironic master: CI: Use focal dnsmasq https://review.opendev.org/c/openstack/ironic/+/888058 | 09:08 |
| JayF | frickler: https://lists.openstack.org/pipermail/openstack-discuss/2023-July/034363.html | 09:10 |
| JayF | so I looked at the OS module, I think it was pulled out b/c of the image streaming bug where it would hang while image streaming | 09:10 |
| JayF | b/c it looks like file I/O is one of the only green-patched places in eventlet | 09:10 |
| JayF | but... that's in IPA not in Ironic, so we might want to consider my patch? | 09:11 |
| dtantsur | iscsi deploy legacy? | 09:11 |
| JayF | perhaps | 09:11 |
| JayF | IMO it's generally safer to patch everything than exclude stuff | 09:11 |
| dtantsur | in other news, bifrost voting CI jobs green without inspector \o/ https://review.opendev.org/c/openstack/bifrost/+/887934 | 09:11 |
| dtantsur | (this work is far from being finished, just the first PoC) | 09:11 |
| JayF | so I sorta feel like my patch to revert should be "default merge" if there's not a good reason to otherwise | 09:11 |
| JayF | https://zuul.opendev.org/t/openstack/build/1fe493bd63ce4f4fb8730308e7f8e2b8 hmm I wonder if this needs jammy dnsmasq | 09:13 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: DNM: Eventlet science https://review.opendev.org/c/openstack/ironic/+/887996 | 09:14 |
| opendevreview | yatin proposed openstack/ironic master: [DNM] Attempt source install devstack https://review.opendev.org/c/openstack/ironic/+/888121 | 09:14 |
| JayF | ah that failure was mentioned as rpittau as a known bug in inspection in metal3 | 09:16 |
| JayF | that's good news | 09:16 |
| JayF | (in a different patch, to be clear) | 09:16 |
| opendevreview | yatin proposed openstack/ironic master: [DNM] Attempt source install dnsmasq https://review.opendev.org/c/openstack/ironic/+/888121 | 09:17 |
| JayF | I suspect TheJulia would be over the moon, but how would anyone else feel about moving our whiteboard into the wiki? | 09:40 |
| JayF | Just to make it more discoverable | 09:40 |
| JayF | I feel like at the point a whiteboard is like, 6+ years old maybe we've gone too far lol | 09:41 |
| JayF | s/whiteboard/etherpad/ | 09:41 |
| JayF | Trivial, but on a spec so I'd rather not land with a single core: https://review.opendev.org/c/openstack/ironic-specs/+/882760 | 09:49 |
| frickler | JayF: ah, thx, that came much later than the announcement here, though | 09:50 |
| JayF | ah, interesting. maybe it was on socials? | 09:50 |
| frickler | regarding wiki.openstack.org, that's also only still barely alive, so I would suggest not to move anything towards it | 09:51 |
| JayF | What is the proper place for content we want visibility on, but a low abrrier to update? | 09:51 |
| dtantsur | I also seem to vaguely remember that the wiki does not allow new users (for spam reasons) | 09:59 |
| frickler | currently etherpad is your best option I'd say. unless you want to invest in deploying and operating a new wiki | 10:02 |
| JayF | ah, too bad, but good to know | 10:42 |
| JayF | thank you for the information | 10:42 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Firmware Interface https://review.opendev.org/c/openstack/ironic/+/885276 | 10:56 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Allow md5 to be disabled from the conductor https://review.opendev.org/c/openstack/ironic-python-agent/+/882367 | 11:06 |
| opendevreview | Merged openstack/ironic master: Fix db migration tests for sqlalchemy 2.0 https://review.opendev.org/c/openstack/ironic/+/887432 | 11:12 |
| rpittau | TheJulia: the snmp pxe job failed on top of your patch so I guess it's something different from dnsmasq | 11:16 |
| opendevreview | Jay Faulkner proposed openstack/sushy master: Requests must always have a read/connect timeout https://review.opendev.org/c/openstack/sushy/+/888131 | 12:08 |
| dtantsur | I wonder how much will break the same way https://bugzilla.redhat.com/show_bug.cgi?id=2221967 | 12:18 |
| opendevreview | Jay Faulkner proposed openstack/sushy master: Requests must always have a read/connect timeout https://review.opendev.org/c/openstack/sushy/+/888131 | 12:21 |
| opendevreview | Merged openstack/ironic master: Fix the HTTP code of the BadRequest exception https://review.opendev.org/c/openstack/ironic/+/887271 | 12:21 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add additional logging on iLO power failure https://review.opendev.org/c/openstack/ironic/+/885549 | 13:07 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Drop databases in unit tests instead of contents https://review.opendev.org/c/openstack/ironic/+/887234 | 13:08 |
| JayF | My downstream friends WFH today in the UK, if it's not obvious bwahahaha | 13:13 |
| JayF | dtantsur: if we do break, lmk. I can be nerd sniped for that one b/c I also use sphinx for armada python client docs | 13:14 |
| dtantsur | ack | 13:19 |
| JayF | I might take over sphinx-jekyll-builder at some point :/ | 13:19 |
| JayF | I suspect it only needs trivial fixes to keep working, and that one is not even compatible with sphinx 6.x | 13:19 |
| TheJulia | JayF: oh good sushy change, I'm surprised we've not caught that before | 13:19 |
| JayF | TheJulia: that one is credited to johnthetubaguy, he commented it on that ilo freeze bug | 13:20 |
| JayF | TheJulia: and based on the codepaths, this could 100% cause the behavior we saw | 13:20 |
| opendevreview | Merged openstack/bifrost master: remove iniparse system packages requirement https://review.opendev.org/c/openstack/bifrost/+/874520 | 13:21 |
| TheJulia | indeed, as well | 13:25 |
| TheJulia | mystical magical packet loss for 30 seconds | 13:25 |
| JayF | rpittau is apparently the newest approver in metal3 \o/ | 13:26 |
| JayF | congrats buddy! | 13:26 |
| rpittau | JayF: just for the website :) | 13:27 |
| rpittau | I'm a reviewer in various components since a while now | 13:27 |
| JayF | when I got core on IPA only | 13:27 |
| JayF | you think I told my mom that? | 13:27 |
| rpittau | :D | 13:27 |
| JayF | take the win and celly, don't downplay it yourself | 13:27 |
| JayF | approval access is always about trust over tech ability; and you've proven yourself trustworthy to two separate communities | 13:28 |
| rpittau | I'm going to celebrate to Disneyland tomorrow :D | 13:28 |
| JayF | if $popularBloggers are to be belived, it's like antimatter meeting matter for openstack and k8s to cooperate | 13:28 |
| JayF | and now we have at least 2 overlaps in this project alone | 13:28 |
| JayF | for core/approvers | 13:28 |
| rpittau | it is good indeed, we're restyling the site with the help of an Outreachy intern and I really needed the approver status! | 13:29 |
| dtantsur | ++ | 13:31 |
| dtantsur | I'd love to see more Ironic community people in https://github.com/metal3-io/baremetal-operator/blob/main/OWNERS#L3 as well | 13:31 |
| JayF | I don't think it'll be me; I feel like I need a dictionary when I talk to k8s people still | 13:32 |
| dtantsur | heh, I know the feeling | 13:34 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Allow md5 to be disabled from the conductor https://review.opendev.org/c/openstack/ironic-python-agent/+/882367 | 14:27 |
| *** tzumainn__ is now known as tzumainn | 14:31 | |
| JayF | metalsmith ipa-src-legacy job failed, hm | 14:35 |
| JayF | gonna recheck b/c that one passed last itme there | 14:36 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Allow md5 to be disabled from the conductor https://review.opendev.org/c/openstack/ironic-python-agent/+/882367 | 14:43 |
| opendevreview | Merged openstack/ironic stable/yoga: [ci] [stable-only] Cinder fixed; make BFV job vote https://review.opendev.org/c/openstack/ironic/+/884221 | 15:03 |
| opendevreview | Merged openstack/ironic master: CI: Change migrations timeout to be >60 seoncds https://review.opendev.org/c/openstack/ironic/+/886985 | 15:03 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 15:29 |
| rpittau | bye everyone, see you next monday! o/ | 15:48 |
| TheJulia | next monday?!? ugh | 16:16 |
| opendevreview | Verification of a change to openstack/ironic master failed: add default conductor group capability https://review.opendev.org/c/openstack/ironic/+/855705 | 16:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/ironic/+/886020 | 17:51 |
| * TheJulia has a moment of realization | 17:58 | |
| TheJulia | ... I think I've figured out our lock issues with metal3 | 18:03 |
| TheJulia | bahahahahahahaha | 18:10 |
| TheJulia | i found it | 18:10 |
| TheJulia | okay, it is invocations of get_nodeinfo_list, which is triggered by all periodics, and a bunch of our test code. They return and work with what is thought a list of returning values... | 19:15 |
| TheJulia | well, when called from iter_nodes in base manager | 19:15 |
| TheJulia | challenge is it is a result set as a row, and the code pattern yields the row back as a single entry to a periodic | 19:16 |
| TheJulia | in other words, connection still open, result rows still in use | 19:16 |
| JayF | could unit tests have been broken by that, too? unlikely? | 19:17 |
| TheJulia | I suspect unlikely | 19:17 |
| JayF | that's still a good find | 19:19 |
| JayF | that probably has caused headaches in places with very, very bad unresponsive bmcs | 19:19 |
| TheJulia | I have one more meeting today, which will likely take an hour, and then will hopefully focus on this | 19:19 |
| TheJulia | eh, so since it is a read only transaction, it shouldn't cause any issues, but still gets wrapped with a transaction | 19:19 |
| TheJulia | just the result set gets abandoned eventually | 19:20 |
| JayF | ack | 19:20 |
| JayF | so longer held open connections | 19:20 |
| JayF | which is not good but not harmful in reasonable configurations | 19:20 |
| TheJulia | exactly | 19:20 |
| JayF | ooooh TheJulia could this be why the mysql profiling broke for us | 19:21 |
| TheJulia | however, with sqlite... it *is* harmful | 19:21 |
| TheJulia | because you can't write then | 19:21 |
| JayF | and why we had to disable it?!? | 19:21 |
| TheJulia | it breaks in walking the data structures, this is far away from that in the code | 19:21 |
| JayF | I'm not sure I understand but I trust ya | 19:23 |
| opendevreview | Ebbex proposed openstack/bifrost master: remove pymysql system packages requirement https://review.opendev.org/c/openstack/bifrost/+/874519 | 20:25 |
| opendevreview | Ebbex proposed openstack/bifrost master: remove nginx system packages requirement https://review.opendev.org/c/openstack/bifrost/+/874521 | 20:25 |
| opendevreview | Ebbex proposed openstack/bifrost master: remove setuptools system packages requirement https://review.opendev.org/c/openstack/bifrost/+/874522 | 20:25 |
| opendevreview | Ebbex proposed openstack/bifrost master: Refactor the use of include_vars https://review.opendev.org/c/openstack/bifrost/+/874523 | 20:25 |
| opendevreview | Ebbex proposed openstack/bifrost master: Refactor use of include_vars https://review.opendev.org/c/openstack/bifrost/+/855807 | 20:31 |
| TheJulia | https://bugs.launchpad.net/ironic/+bug/2027405 | 20:44 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DB: Fix result set locking with periodics https://review.opendev.org/c/openstack/ironic/+/888188 | 20:50 |
| TheJulia | stevebaker[m]: JayF ^ | 20:50 |
| TheJulia | we will want to check the metal3 log, but that should greatly improve errors | 20:51 |
| TheJulia | And by improve I mean reduce | 20:56 |
| JayF | I want to review that with morning eyes | 21:01 |
| stevebaker[m] | Reviewed, lets wait for the jobs | 21:09 |
| TheJulia | Likely best for multiple runs | 21:35 |
| TheJulia | not one db lock exception in that change's metal3 logs | 21:54 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DB: Fix result set locking with periodics https://review.opendev.org/c/openstack/ironic/+/888188 | 22:12 |
| TheJulia | passed again :) | 23:14 |
| * TheJulia dances | 23:14 | |
| TheJulia | no locking errors | 23:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!