| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: Special case lenovo UEFI boot setup https://review.opendev.org/c/openstack/ironic/+/910447 | 00:04 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic stable/zed: Special case lenovo UEFI boot setup https://review.opendev.org/c/openstack/ironic/+/910448 | 00:06 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/xena: Special case lenovo UEFI boot setup https://review.opendev.org/c/openstack/ironic/+/910316 | 00:17 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: Special case lenovo UEFI boot setup https://review.opendev.org/c/openstack/ironic/+/910317 | 00:17 |
| *** jph6 is now known as jph | 00:21 | |
| JayF | TheJulia: landing the version of that on master | 00:25 |
| JayF | oh, I mean | 00:26 |
| JayF | +2 but no +A | 00:26 |
| JayF | gate is broken so no sense in +A | 00:26 |
| JayF | but someone can feel free to when it's fixed | 00:26 |
| TheJulia | yeah | 00:26 |
| TheJulia | I backported it so someone could hopefully pick it up off of wallaby and give it a spin | 00:26 |
| JayF | I +2'd back to stable/zed | 00:27 |
| JayF | I'll wait for that report to hit UM+others | 00:27 |
| TheJulia | ack | 00:27 |
| * TheJulia has reached zero brain | 00:27 | |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Invoke tests with fake interfaces https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/909939 | 00:42 |
| TheJulia | depends-on on added for jayf's patch. | 00:42 |
| TheJulia | Witht hat, I'm going to go call it a day | 00:42 |
| TheJulia | oh heh, the changes don't stack cleanly | 01:30 |
| TheJulia | oh well | 01:30 |
| opendevreview | Merged openstack/ironic master: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/888121 | 01:43 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Invoke tests with fake interfaces https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/909939 | 03:41 |
| TheJulia | hjensas: 909939 fixes the tests, but I'll have to unbrick the gates to get it to merge. I guess all our downstream notes can get updated with the same cause. | 04:30 |
| opendevreview | Kaifeng Wang proposed openstack/python-ironicclient master: Client support port name https://review.opendev.org/c/openstack/python-ironicclient/+/896067 | 06:45 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 06:48 |
| rpittau | good morning ironic! o/ | 07:58 |
| rpittau | we have 3 patches in ironicclient to merge before releasing, if anyone can have a look today would be great | 09:01 |
| rpittau | https://review.opendev.org/c/openstack/python-ironicclient/+/908788 | 09:01 |
| rpittau | https://review.opendev.org/c/openstack/python-ironicclient/+/906794 | 09:01 |
| rpittau | https://review.opendev.org/c/openstack/python-ironicclient/+/906796 | 09:01 |
| rpittau | TheJulia, JayF: not sure who we can ping for https://review.opendev.org/c/openstack/tempest/+/908727 | 09:02 |
| rpittau | TheJulia: about https://review.opendev.org/c/openstack/ironic/+/910444 not sure if you tried with the latest pkg from jammy, it is indeed 2.90 https://paste.openstack.org/raw/bMvZMYa2mKsXzmWJbcwu/ | 09:53 |
| zigo | Is there (was there) a problem with the Ironic gate? How come my patch failed gating twice with unrelated issues? https://review.opendev.org/c/openstack/ironic-python-agent/+/910209 | 10:02 |
| zigo | Oh, read TheJulia, so the gate really is broken ... :/ | 10:03 |
| rpittau | zigo: there are multiple issues, trying to sort the unit tests at the moment | 10:20 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Fix commands order in partition tests https://review.opendev.org/c/openstack/ironic-python-agent/+/910480 | 10:27 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Fix commands order in partition tests https://review.opendev.org/c/openstack/ironic-python-agent/+/910480 | 10:34 |
| frickler | rpittau: gmann and kopecmartin are the only remaining tempest people afaict. added them to your patch, you can also ping them in #openstack-qa if needed | 10:38 |
| rpittau | frickler: thanks a lot! :) | 10:39 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Fix unit tests after ironic-lib changes https://review.opendev.org/c/openstack/ironic-python-agent/+/910480 | 11:22 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Add inspection PXE filter service https://review.opendev.org/c/openstack/ironic/+/907991 | 11:30 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Add inspection PXE filter service https://review.opendev.org/c/openstack/ironic/+/907991 | 11:36 |
| opendevreview | Merged openstack/python-ironicclient master: [codespell] Fixing Spelling Mistakes https://review.opendev.org/c/openstack/python-ironicclient/+/906794 | 12:20 |
| opendevreview | Merged openstack/python-ironicclient master: [codespell] Adding Tox Target for Codespell https://review.opendev.org/c/openstack/python-ironicclient/+/906795 | 12:20 |
| opendevreview | Verification of a change to openstack/python-ironicclient master failed: [codespell] Adding CI target for Tox Codespell https://review.opendev.org/c/openstack/python-ironicclient/+/906796 | 12:26 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 13:12 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Update tinyipa to tinycore 15.x https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/910169 | 13:37 |
| TheJulia | good morning | 14:12 |
| TheJulia | rpittau: I'd just keep the job non-voting, it has been left failing for cycles in the past, its not a big deal in my book | 14:13 |
| rpittau | TheJulia: ack | 14:13 |
| opendevreview | Merged openstack/python-ironicclient master: Force constraints when installing a package during tox test https://review.opendev.org/c/openstack/python-ironicclient/+/908788 | 14:27 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/910444 is for stable/2023.2 and passes CI | 14:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: re-enable grenade test job https://review.opendev.org/c/openstack/ironic/+/910516 | 14:35 |
| TheJulia | heh, ubuntu pulled their new dnsmasq into focal | 14:41 |
| dtantsur | so kinds of them | 14:41 |
| TheJulia | since it is quietly exiting, I'm going to pull the source install workaround down to focal as well | 14:42 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910518 | 14:46 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Fix unit tests after ironic-lib changes https://review.opendev.org/c/openstack/ironic-python-agent/+/910480 | 14:53 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: update dnsmasq to 2.90 via source https://review.opendev.org/c/openstack/ironic/+/910521 | 15:00 |
| TheJulia | ugh https://9cb23ed83c623b0d0348-b8e5e06ecccd85e574a730a33d1fddb0.ssl.cf1.rackcdn.com/910518/1/check/ironic-tox-unit-with-driver-libs/805f681/testr_results.html | 15:38 |
| * dtantsur blinks | 15:39 | |
| dtantsur | have they just switched to async/await? Oo | 15:39 |
| TheJulia | The version is 5.0.0 | 15:40 |
| TheJulia | it is not being constrained | 15:40 |
| TheJulia | it should... be constrained | 15:40 |
| JayF | pysnmp is a driver lib, is it in u-c? | 15:41 |
| JayF | or do we need to constrain in driver-requirements.txt | 15:41 |
| dtantsur | we might consider using u-c always | 15:41 |
| JayF | yeah it's in u-c | 15:42 |
| JayF | https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L399 | 15:42 |
| JayF | three snmp libs in there | 15:42 |
| JayF | https://opendev.org/openstack/ironic/src/branch/master/tox.ini#L31 | 15:43 |
| JayF | we don't use u-c for driver-libs jobs | 15:43 |
| JayF | there's the answer | 15:43 |
| TheJulia | it is not in u-c on that branch | 15:43 |
| TheJulia | this is stable/2023.1 | 15:43 |
| rpittau | we probably need this https://review.opendev.org/c/openstack/ironic/+/908783 | 15:43 |
| JayF | either way, we don't use constraints for driver-libs | 15:43 |
| JayF | rpittau: we don't pass -c on the driver-libs job | 15:44 |
| JayF | it's more basic than that | 15:44 |
| TheJulia | yeah, we need to directly constrain it on that branch | 15:44 |
| JayF | I'd suggest for stable/ branches, updating requirements | 15:44 |
| JayF | and fixing master going forward | 15:44 |
| JayF | but I'll +2 any reasonable fix | 15:44 |
| TheJulia | master is not broken by this right now | 15:44 |
| JayF | oh! | 15:44 |
| JayF | {[testenv]deps} is inheritance, isn't it? | 15:45 |
| JayF | is that a new thing? New to me, anyway, and awesome | 15:45 |
| rpittau | not very new | 15:45 |
| TheJulia | on a plus side, I seem to be able to reproduce it locally :) | 15:45 |
| * TheJulia waits for tox to timeout | 15:45 | |
| TheJulia | err, no, it passed | 15:47 |
| TheJulia | ugh | 15:47 |
| TheJulia | oh, it is via proliantutils | 15:48 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: stable-only: pin proliantutils to prevent break https://review.opendev.org/c/openstack/ironic/+/910528 | 15:53 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Don't import sushy conditionally, it's a requirement https://review.opendev.org/c/openstack/ironic/+/910529 | 15:56 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910518 | 15:56 |
| JayF | dtantsur: that's a fun change, I like it :D | 16:02 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910518 | 16:03 |
| JayF | I just landed https://review.opendev.org/c/openstack/ironic/+/888297 as a single core, only change from last (2x+2) patchset is spelling fix | 16:06 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/901090 needs another core review | 16:07 |
| rpittau | approved | 16:08 |
| JayF | dtantsur: I went ahead and landed https://review.opendev.org/c/openstack/ironic/+/902801 (reserved workers pool), I don't like it, but I don't have better ideas or time to implement them | 16:08 |
| JayF | I am afraid my original -1 might have scared off other cores from approving it, too :) | 16:09 |
| TheJulia | I tagged https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/909939 as a prio item for reviews... It makes the api tests explicit instead of relying upon a default new interface starting as "fake" && "noop", meaning before the change, you couldn't just run the api tests against a deployment with any operational defaults | 16:12 |
| dtantsur | JayF: thanks! | 16:26 |
| TheJulia | ugh, looks like we're going to need to recheck the stable/2023.2 fix. Metal3-integration failed on it | 16:35 |
| TheJulia | timeout from quay.io | 16:35 |
| JayF | if metal3 itself doesn't do stable | 16:36 |
| JayF | we should probably drop those jobs from stable branches | 16:36 |
| JayF | (yeah?) | 16:36 |
| TheJulia | .... | 16:36 |
| TheJulia | You've jumped to an unrelated conclusion | 16:36 |
| TheJulia | the root cause of the failure is the job timed out talking to quay.io | 16:36 |
| JayF | No, I'm pulling in information from the slack context of a chat dtantsur and I had a couple weeks ago | 16:36 |
| TheJulia | well, then yeah, we likely shouldn't have the jobs on stable branches | 16:36 |
| JayF | Yeah, I'm basically musing related to that if we should run those jobs or not | 16:37 |
| JayF | dtantsur: does it make sense to run metal3 jobs on ironic? | 16:37 |
| JayF | dtantsur: er, stable brnaches | 16:37 |
| dtantsur | Not much, at least not at this point | 16:37 |
| dtantsur | and yes, quay is more or less reliable, but not 100% | 16:38 |
| rpittau | I agree on not running metal3 jobs on stable branches, doesn't really make sense for now | 16:39 |
| JayF | ack; I'll push changes to that effect now() | 16:41 |
| rpittau | I'll have a look tomorrow if noone else get to it, time to leave | 16:41 |
| rpittau | good night! o/ | 16:41 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add note regarding metal3 ci job in CI config for stable runs https://review.opendev.org/c/openstack/ironic/+/910536 | 16:43 |
| TheJulia | JayF: fyi^ just so we know we can prune them out later on | 16:43 |
| JayF | Does anyone know who panhongyin is in Gerrit? | 16:44 |
| JayF | They've been providing more and more reviews but I've not met them | 16:44 |
| JayF | and want to thank them for being active | 16:44 |
| JayF | TheJulia: https://review.opendev.org/c/openstack/ironic/+/910436 needs to be abandoned, I think? (changing package url for dnsmasq) -- if so just say yeah and I'll do it | 16:46 |
| TheJulia | yup, thanks | 16:46 |
| JayF | aight, I just did a run through priority reviews | 16:48 |
| JayF | Sorry it's been about a week since I did a through one before today | 16:48 |
| dtantsur | TheJulia: you can also tell Zuul to only run a job on master | 16:55 |
| dtantsur | I think we've done it for some jobs somewhere.. | 16:55 |
| dtantsur | yeah https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/zuul.d/project.yaml#L44-L51 | 16:55 |
| TheJulia | I think you can, but we already have to prune jobs as time passes anyway | 16:55 |
| dtantsur | yep, but this will be automagical | 16:56 |
| TheJulia | so the pre-emptive grant of permission is not a bad thing | 16:56 |
| TheJulia | we're really bad at pruning stuff, imho | 16:57 |
| clarkb | note that isn't really automagically because https://opendev.org/openstack/ironic-python-agent-builder/src/branch/stable/2023.2/zuul.d/project.yaml#L40-L47 also exists | 16:58 |
| opendevreview | Merged openstack/python-ironicclient master: [codespell] Adding CI target for Tox Codespell https://review.opendev.org/c/openstack/python-ironicclient/+/906796 | 16:58 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Force constraints when installing a package during tox test https://review.opendev.org/c/openstack/ironic-python-agent/+/908787 | 16:58 |
| clarkb | if you remove the job from master then all of a sudden that old job config will apply to master | 16:59 |
| TheJulia | Heh, so pruning the config is likely for the best then | 17:00 |
| dtantsur | TIL! | 17:03 |
| TheJulia | no, today we learned! | 17:04 |
| TheJulia | :) | 17:04 |
| dtantsur | True :) | 17:04 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.2: ci: stable-only: remove metal3-integration ci job https://review.opendev.org/c/openstack/ironic/+/910537 | 17:09 |
| TheJulia | heh, metal3-integration has failed on another ojb | 17:11 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add inspection PXE filter service https://review.opendev.org/c/openstack/ironic/+/907991 | 17:14 |
| *** dking is now known as Guest1316 | 17:15 | |
| *** Guest1316 is now known as dking | 17:16 | |
| dtantsur | I assume we haven't done anything to fix https://bugs.launchpad.net/ironic/+bug/2052468 have we? | 17:19 |
| TheJulia | afaik no | 17:19 |
| TheJulia | I proposed a ptg topic on the subject, fwiw | 17:19 |
| dtantsur | TheJulia: ack. Next, could you confirm/deny https://bugs.launchpad.net/ironic/+bug/2053068/comments/1 ? | 17:21 |
| * dtantsur OUCH @ https://bugs.launchpad.net/ironic/+bug/2054594 | 17:22 | |
| TheJulia | yes, that is true | 17:23 |
| TheJulia | oh jeeze | 17:23 |
| dtantsur | the length issue? yeaaah | 17:24 |
| TheJulia | The length can be extended, just ugh | 17:24 |
| TheJulia | I geuss it is something like <federation><user> concatonated together | 17:25 |
| dtantsur | I wonder why we picked 32 | 17:26 |
| TheJulia | length of a uuid | 17:27 |
| TheJulia | didn't expect longer user ids | 17:28 |
| TheJulia | I'll start on a fix later today most likely, need to draft an agenda and send a few emails first | 17:28 |
| dtantsur | Could not import extension openstackdocstheme (exception: No module named 'distutils') | 17:31 |
| dtantsur | Incredible! | 17:31 |
| opendevreview | Dmitry Tantsur proposed openstack/python-ironicclient master: Add missing commands to the documentation https://review.opendev.org/c/openstack/python-ironicclient/+/910540 | 17:32 |
| dtantsur | is anyone aware of any movements around https://bugs.launchpad.net/ironic/+bug/1732534 ? | 17:39 |
| TheJulia | I'm unsure it can be solved in ironic at all | 17:41 |
| TheJulia | perhaps some documentation? | 17:41 |
| clarkb | useradd on linux also limits usernames to 32 chars long | 17:41 |
| opendevreview | Verification of a change to openstack/ironic stable/2023.2 failed: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910444 | 17:42 |
| JayF | Does someone have a suggestion for cid on places to learn more about base level tech that Ironic is built on? e.g. DHCP/PXE/Redfish/IPMI/etc | 17:48 |
| dtantsur | Ironic source? :D </kidding> | 17:55 |
| opendevreview | Verification of a change to openstack/ironic stable/2023.2 failed: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910444 | 17:59 |
| clarkb | JayF: I learned a lot of that stuff in a lab environment on old sun workstation hardware that my university basically didn't want anymore. So we had a large closet that we hacked on stuff in. | 18:00 |
| clarkb | tl;dr I would suggest something similar of possible. Can probably virtualize much of it | 18:00 |
| clarkb | start with a route and a switch, add a piece of hardawre that you can dhcp and pxeboot and take it from there | 18:01 |
| opendevreview | Verification of a change to openstack/ironic master failed: Detect ilo6 and redirect to redfish https://review.opendev.org/c/openstack/ironic/+/888297 | 18:06 |
| JayF | clarkb: That requires more local resources than he has access to | 18:07 |
| JayF | dtantsur: Where do you think the questions came from :D | 18:07 |
| JayF | dtantsur: I got to butcher an explanation to cid on how driver composition works, I'm glad it's not recorded, you'd probably find it upsetting ;) | 18:08 |
| JayF | like a kid drawing the mona lisa as a stick figure :P | 18:08 |
| dtantsur | LOL | 18:09 |
| cid | lol. didn't sound like that to me. | 18:09 |
| dtantsur | meanwhile, I've added an "Ancient bugs" section to the bug dashboard, and it's kinda depressing... | 18:09 |
| JayF | I basically said node.driver gives you a menu of options to pick from | 18:09 |
| JayF | with defaults | 18:09 |
| dtantsur | 2014-04-21 23:19:57 is the oldest open one | 18:09 |
| JayF | and you can use the interfaces to tweak it | 18:09 |
| JayF | lol | 18:09 |
| JayF | probably from me | 18:09 |
| JayF | that date is perfect for it to be an early jay bug | 18:10 |
| dtantsur | Aeva https://bugs.launchpad.net/ironic/+bug/1310843 | 18:10 |
| JayF | and I wrote a lot of doozies lol | 18:10 |
| * dtantsur hopes he hasn't caused conflicts for in-flight dashboard patches | 18:10 | |
| clarkb | JayF: cid: it doesn't have to be much. If you have a home router it can probably be convinced to dhcp and pxe other things. Probably the biggeest issue is finding something to pxeboot but that can just be a VM even | 18:11 |
| clarkb | and then tcpdump all the things | 18:11 |
| cid | I heard at least 10 new terms today on the meet. do you have any resources. Like, I can't pretend to know what "tcpdump" all the things means. | 18:14 |
| clarkb | cid: tcpdump is a network capture tool that you can run on linux/unix type systems. Maybe windows too I'm not sure. But it allows you to see the actual network traffic flowing on the network which can be useful to better understand network protocols. Another higher level tool that is probably more user friendly is wireshark | 18:15 |
| JayF | Microsoft has a tool akin to tcpdump which is stellar on windows | 18:15 |
| JayF | Wireshark also has a gui for windows which can do captures | 18:16 |
| clarkb | nice. I usually start with tcpdump to figure out what I want to capture then dump that to a file and look at it in wireshark for more in depth diggig/debugging | 18:16 |
| cid | I have used wireshark once | 18:18 |
| cid | By the way JayF: I didn't get a mail for the meet | 18:20 |
| JayF | tcpdump is just a CLI tool that does wireshark-y things | 18:23 |
| JayF | in fact, wireshark will often take a dump file from tcpdump as input to "replay" | 18:23 |
| JayF | but I'd even say ... you don't even really have to go that far | 18:24 |
| JayF | you have a devstack w/Ironic | 18:24 |
| JayF | set to pxe boot fake baremetal nodes... | 18:24 |
| JayF | you could potentially just tcpdump on the bridge while the pxe booting is happening and just watch it | 18:24 |
| JayF | I'm not 100% sure which interface, but figuring that part out could be part of the fun | 18:24 |
| clarkb | ++ | 18:25 |
| clarkb | and really the reason to tcpdump is to "see" things in action to get a better undersatnding of the network protocols. Not strictly necessary but can be really useful when things go wrong | 18:25 |
| JayF | clarkb: tbh, I actually have an adage of network captures /usually/ are wild goose chases | 18:32 |
| JayF | only exception being if you're working on /actual network software/ (e.g. the troubleshooting Julia was doing around OVN, of course you have to packet cap that stuff) | 18:33 |
| clarkb | for me it makes things concrete in a way that makes them more undstandabale | 18:33 |
| clarkb | a diagram for dhcp flow is super abstract but seeing the broadcasts and responses and renewals helps me | 18:33 |
| JayF | look at too many packet captures and you realize how awful some dhcp clients are lol | 18:35 |
| * TheJulia twitches | 18:38 | |
| TheJulia | sign, looks like standalone is broken now | 18:57 |
| JayF | I just rechecked my job, it was broken in a strange way | 19:00 |
| JayF | I was thinking (hoping?) it was a bit flip in transfers | 19:00 |
| JayF | but if it's reproducing in other places :( | 19:00 |
| TheJulia | looks like it might just be dnsmasq in general | 19:00 |
| TheJulia | at least, that is my *guess* at the moment | 19:01 |
| JayF | Would you be +1 to marking it n-v while we troubleshoot? | 19:01 |
| TheJulia | dunno, looks like the redfish variant failed | 19:03 |
| TheJulia | (.... I thought I removed one of them... maybe that change just hasn't merged yet) | 19:03 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add redfish https boot CI job https://review.opendev.org/c/openstack/ironic/+/901090 | 19:03 |
| JayF | heh | 19:04 |
| JayF | nice timing, gerrit | 19:04 |
| JayF | that's adding a job, not removing one | 19:04 |
| TheJulia | hmmm | 19:05 |
| TheJulia | nothing definitive, no direct sign of dnsmasq | 19:05 |
| JayF | in the ironic-standalone failure I saw, I looked in console logs | 19:06 |
| TheJulia | dnsmasq respawning https://f4a78187a8f66e46939f-e2f3a8f1da38bd85104d6de65559a608.ssl.cf1.rackcdn.com/902801/2/gate/ironic-standalone/626a7a4/controller/logs/screen-q-dhcp.txt | 19:06 |
| JayF | and last ramdisk boot failed with errors around not getting proper size back | 19:06 |
| JayF | we really need to find that .deb file | 19:07 |
| JayF | somewhere | 19:07 |
| JayF | and host it to get our gate happy for now | 19:07 |
| JayF | and dig dnsmasq without a time crunch | 19:07 |
| TheJulia | I don't even know where to start with https://f4a78187a8f66e46939f-e2f3a8f1da38bd85104d6de65559a608.ssl.cf1.rackcdn.com/902801/2/gate/ironic-standalone/626a7a4/controller/logs/ironic-bm-logs/node-2_console_log.txt | 19:08 |
| JayF | it's sorta the behavior I'd expect | 19:09 |
| JayF | if it's respawning | 19:09 |
| TheJulia | well, any new process launch would change the log | 19:09 |
| TheJulia | it is almost like qemu is kind of going "uhhh reset!" | 19:09 |
| TheJulia | I can't help but feel like we might find something like: https://forums.fedoraforum.org/showthread.php?323033-kernel-Dazed-and | 19:11 |
| JayF | TheJulia: we don't need to fix the dnsmasq stuff somewhere else? e.g. bifrost, right? | 19:14 |
| JayF | This looks devstack-y so I assume not, but we've validated it's actually running the version we think, right? | 19:14 |
| TheJulia | bifrost leverages static config | 19:14 |
| TheJulia | the issue here is rooted in dnsmasq configuration getting updated | 19:14 |
| JayF | okay, confirmed the basic assumption, we're running the compiled version | 19:15 |
| JayF | going to check neutron-dhcp-agent for changes | 19:15 |
| TheJulia | ... we could hold the next failure | 19:16 |
| JayF | trying to rule out other possibilities before diving into C, which I don't know well | 19:16 |
| JayF | the key is being able to reproduce | 19:16 |
| JayF | if I can get a reproducer, specifically locally, I can solve this problem | 19:16 |
| JayF | either Jay-I or I can get resources from GR-OSS | 19:16 |
| opendevreview | Merged openstack/ironic master: Handle jsonschema empty error message update https://review.opendev.org/c/openstack/ironic/+/909592 | 19:18 |
| opendevreview | Merged openstack/ironic master: Force constraints when installing a package during tox test https://review.opendev.org/c/openstack/ironic/+/908783 | 19:18 |
| opendevreview | Merged openstack/ironic-inspector master: Force constraints when installing a package during tox test https://review.opendev.org/c/openstack/ironic-inspector/+/908784 | 19:18 |
| TheJulia | whoaw | 19:19 |
| TheJulia | things merged | 19:19 |
| TheJulia | \o/ | 19:19 |
| JayF | yeah, this is the other thing I was suspicious of | 19:19 |
| JayF | gate is as busy today as it ever is | 19:19 |
| JayF | if there were more rare edge cases in dnsmasq breakage, we'd see them here | 19:19 |
| TheJulia | or the excess pressure is causing "other" things happening which are related | 19:19 |
| JayF | yep | 19:19 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Remove downgrade_dnsmasq; 2.90 is upstream now https://review.opendev.org/c/openstack/ironic/+/910445 | 19:20 |
| JayF | fixed the version number instead of rechecking | 19:20 |
| JayF | it did pass on your recheck | 19:20 |
| JayF | if we can verify this works at least as well as the custom compiled version, at least we've made some progress (and can operate testing on upstream ubuntu) | 19:20 |
| JayF | so I do think one thing we can/should do | 19:23 |
| JayF | is enable debugging symbols on our build (if possible) | 19:23 |
| JayF | try to save the core file generated | 19:24 |
| JayF | and see why it's busted | 19:24 |
| TheJulia | dtantsur: so https://review.rdoproject.org/r/c/openstack/ironic-distgit/+/51902/2/openstack-ironic.spec although I'm wondering if it should recommended since it is purely for bios booting | 19:24 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Update nova instance instructions to use demo user https://review.opendev.org/c/openstack/ironic/+/910545 | 19:31 |
| JayF | cid: ^ as promised | 19:31 |
| cid | that was fast | 19:31 |
| JayF | Doc updates are mostly a quick edit and push; once you get the workflow downpat it doesn't take so long | 19:32 |
| cid | true | 19:33 |
| cid | so you can provide a fix without create a bug first? | 19:33 |
| cid | *creating | 19:33 |
| JayF | So there are two basic things that a bug does for us: 1) allows us to track details of an issue until we have time to track it down (or while we're tracking it down) or 2) allows a place for us to show users where things have changed | 19:34 |
| TheJulia | so going back to the merged patches, https://review.opendev.org/c/openstack/ironic/+/909592?tab=change-view-tab-header-zuul-results-summary :( | 19:34 |
| JayF | ah | 19:34 |
| JayF | in some projects, it also does a third: creating documentation the bug has been fixed | 19:35 |
| JayF | In Ironic, we use release notes for that last bit | 19:35 |
| JayF | so unless we need to track something longer term, or it's serious enough to want a bug to point at (e.g. a security issue), we mainly are concerned that things that need a release note get one | 19:35 |
| JayF | to summarize: we almost never ask for someone to explicitly create a bug unless it's for a serious issue | 19:35 |
| cid | copy that | 19:36 |
| JayF | TheJulia: we don't use any dnsmasq lua features, right? | 19:42 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add a reserved workers pool (5% by default) https://review.opendev.org/c/openstack/ironic/+/902801 | 19:44 |
| TheJulia | I'd have to look it up | 19:44 |
| TheJulia | I highly doubt it | 19:44 |
| JayF | ack, that's my assumption | 19:44 |
| JayF | I'm digging thru dnsmasq repo right now | 19:44 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [DNM/Science] Build master dnsmasq https://review.opendev.org/c/openstack/ironic/+/910546 | 19:47 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [DNM/Science] Build master dnsmasq https://review.opendev.org/c/openstack/ironic/+/910546 | 19:50 |
| opendevreview | Verification of a change to openstack/ironic master failed: Log upon completion of power sync https://review.opendev.org/c/openstack/ironic/+/891334 | 20:29 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: [ci] Temporarily disable standalone job voting https://review.opendev.org/c/openstack/ironic/+/910548 | 20:32 |
| JayF | TheJulia: ^ I understand if you wanna nack that :) | 20:33 |
| TheJulia | I'm semi-tempted to recheck some of the failing items this evening before/while at/or after tonight's hockey game | 20:35 |
| JayF | I think that's a good data point, but still doesn't get us past the fact we can't merge things | 20:36 |
| JayF | and that we can have a reasonable amount of confidence it's because the job is broken, not because the code is broken | 20:36 |
| JayF | the time pressure to resolve a gate issue one day before milestone-3 is going to make it hard to root cause and fix this properly | 20:37 |
| JayF | that's my basic motivation | 20:37 |
| TheJulia | what worries me is stuff merged with only some tests being run | 20:45 |
| JayF | those were only unit test changes | 20:48 |
| JayF | so they only ran unit tests | 20:48 |
| TheJulia | I thought one had actual code | 20:48 |
| TheJulia | anyway, distracted | 20:48 |
| JayF | I didn't check them all | 20:48 |
| JayF | but my spot checks all were unit test only | 20:48 |
| * TheJulia goes back to gogole doc in other window | 20:48 | |
| JayF | TheJulia: https://review.opendev.org/c/openstack/ironic/+/908959 merge conflict, your change to remove ironic-standalone job altogether | 20:54 |
| JayF | TheJulia: I will rebase | 20:55 |
| TheJulia | oh yeah... there it is! | 20:55 |
| TheJulia | thanks! | 20:55 |
| JayF | oooooh it's stackede behind https://review.opendev.org/c/openstack/ironic/+/908955 | 20:56 |
| TheJulia | oh, doesn't need to be | 20:56 |
| JayF | yeah | 20:56 |
| JayF | I think I'm going to wait for my thing to land | 20:56 |
| JayF | then start landing these beehind it | 20:56 |
| JayF | that's the better path | 20:56 |
| TheJulia | I was just doing some cleanup to remove some excess stuff and it all ended up on a chain | 20:56 |
| JayF | hmm, I could squash em | 20:56 |
| TheJulia | sounds good | 20:56 |
| JayF | that's probably better tbh | 20:57 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Update link to ipmitool repository https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/910216 | 21:00 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Multiple CI updates/improvements https://review.opendev.org/c/openstack/ironic/+/908955 | 21:02 |
| JayF | this is going to merge conflict with my other landing change | 21:03 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Multiple CI updates/improvements https://review.opendev.org/c/openstack/ironic/+/908955 | 21:04 |
| JayF | now it's stacked | 21:05 |
| JayF | so it can be landed | 21:05 |
| JayF | rpittau: This is a change you effectively already had a +2 on: https://review.opendev.org/c/openstack/ironic/+/908955 and is CI, you can probably land it as a single core | 21:05 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Update ipmitool version to 1.8.19 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/910344 | 21:07 |
| TheJulia | JayF: ^^^ cirros didn't get a dhcp adddress :( | 21:09 |
| TheJulia | I think we need to just ask for infra to hold the next standalone job failure | 21:09 |
| TheJulia | since it has the highest odds of failing | 21:09 |
| TheJulia | given so many different dhcp reconfigurations occur there | 21:10 |
| JayF | I struggle to answer what I'd do with such a machine | 21:10 |
| JayF | I might take the approach of spinning up a VM and trying to beat up on dnsmasq and get a tight reproducer | 21:10 |
| TheJulia | 1) verify the right version is being invoked 2) hunt for core files 3) try to reproduce exactly | 21:10 |
| JayF | if I can do that, the whole shape of this problem changes | 21:10 |
| JayF | 1 is not something I had considered, is good | 21:11 |
| TheJulia | so neutron does log it | 21:11 |
| JayF | infra-root: Can someone please hold the next failure of `ironic-standalone` from literally anywhere | 21:11 |
| * JayF wonders if that highlight works in all channels | 21:12 | |
| TheJulia | neutron is logging "dnsmasq[63710]: started, version 2.87 cachesize 150" | 21:12 |
| TheJulia | so #1, is the one we built | 21:12 |
| JayF | a core file really is what we need, I think | 21:12 |
| JayF | if we can identify where it breaks we can go hunting for the bug | 21:12 |
| JayF | I really suspect we're somehow getting it to reload config twice and it's stomping on itself somehow | 21:12 |
| TheJulia | Feb 28 20:10:02.134411 np0036907166 dnsmasq[63741]: exiting on receipt of SIGTERM | 21:13 |
| JayF | but that's not justified by any evidence other than spidey-sense lol | 21:13 |
| TheJulia | ohhhhh | 21:13 |
| JayF | :-? | 21:13 |
| opendevreview | Verification of a change to openstack/ironic stable/2023.2 failed: ci: Source install dnsmasq-2.87 https://review.opendev.org/c/openstack/ironic/+/910444 | 21:13 |
| fungi | JayF: it does, yep | 21:19 |
| JayF | for access > ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILR/OLTS/VWHzE3vgBFCaTNBg2+MRCENOmDr9oEMxzhZ jay@jvf.cc | 21:19 |
| JayF | I suspect TheJulia would want a key on there too | 21:20 |
| JayF | but I can do that if it's a hassle :) | 21:20 |
| fungi | JayF: i have to specify a project, so any openstack-ironic change which fails a job named ironic-standalone will get that build's node(s) held | 21:22 |
| fungi | er, any openstack/ironic change | 21:23 |
| TheJulia | any openstack/ironic change should do | 21:23 |
| TheJulia | just ironic-standalone.* | 21:23 |
| JayF | TheJulia: what was the deal with the sigterm discovery? | 21:25 |
| JayF | TheJulia: conclusions, I mean | 21:25 |
| TheJulia | I'm starting to wonder if we have a comedy of errors | 21:25 |
| JayF | just making sure before I go deep on this I'm going deep on the right thing | 21:25 |
| JayF | If so it needs a better sense of humor :| | 21:26 |
| TheJulia | so restarted 12 times, only sigtermed twice on https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d5e/910546/2/check/ironic-standalone/d5ebf5a/controller/logs/screen-q-dhcp.txt | 21:27 |
| JayF | I have a 2pm-2:30pm meeting, if you wanted to look at this on a call after that I can | 21:28 |
| TheJulia | enable and restart methods getting called by neutron | 21:28 |
| JayF | would not hate making sure my knowledge is synced before I get in deep to trying to break dnsmasq in a VM | 21:28 |
| TheJulia | 2-3 is a meeting for me, and 3:30 is mr corgi's pickup from the surgeon | 21:28 |
| JayF | oh, I hope things went well with your pup, my FIL's dog recently had that issue+surgery and is doing better as a result | 21:28 |
| TheJulia | Surgeon called summer and said "so many stones... so very very many stones | 21:29 |
| JayF | hopefully they'll be a lot better getting it fixed | 21:31 |
| fungi | TheJulia: JayF: wait, to be clear, zuul matches on an exact job name. i told it to hold the next failure of a job named "ironic-standalone" | 21:34 |
| fungi | is that an actual job? | 21:34 |
| TheJulia | we have an ironic-standalone and an ironic-standalone-redfish job | 21:34 |
| TheJulia | both are unhappy | 21:34 |
| JayF | e.g. ironic-standalone https://zuul.opendev.org/t/openstack/build/b4274e6e026b4cb592ad9e211eb6917b : FAILURE in 1h 19m 09s | 21:34 |
| fungi | okay, i can create a separate autohold for ironic-standalone-redfish if you want | 21:35 |
| JayF | Kangie: ^ Someone in #gentoo-chat said you might have ideas on how to troubleshoot this. Mainly I'm trying to figure out how, locally, in an attempt to reproduce the dnsmasq respawning issues, I might restrict I/O performance on a libvirt+kvm VM in order to emulate high contention in the gate (any other advice you have on this would be good, too) | 21:35 |
| JayF | Kangie: basically we're seeing dnsmasq respawn, frequently, during config updates under high load, where we suspect "high load" in reality == bad I/O performance | 21:36 |
| Kangie | It looks like there might be IO limits in a qemu branch | 21:36 |
| JayF | and I just reproduced the issue on HEAD of master of dnsmasq | 21:36 |
| Kangie | https://repo.or.cz/w/qemu/qemu-dev-zwu.git/shortlog/refs/heads/io_limits_latest | 21:36 |
| JayF | ooh good stuff | 21:36 |
| Kangie | Sadly a decade or more old | 21:36 |
| JayF | oh, not good stuff LOL | 21:36 |
| Kangie | But if the patches still apply or got upstreamed.... | 21:37 |
| JayF | yeah, I'll use that to search around | 21:37 |
| Kangie | I guess other things - network filesystem | 21:37 |
| JayF | am also slightly tempted to... oh, that's a good idea | 21:37 |
| Kangie | Mount it on a network FS and restrict bandwidth | 21:37 |
| JayF | I was going to literally just put a spinning disk in the machine | 21:37 |
| JayF | and run bonnie++ on the same partition | 21:37 |
| Kangie | (or point it at a clustered filesystem and mildly load it... :P) | 21:37 |
| JayF | network fs + traffic shaping is a really, really stellar idea | 21:38 |
| JayF | mainly because I know how to do all those things already | 21:38 |
| JayF | lol | 21:38 |
| JayF | Thanks; I knew you'd have some inspiration for me! | 21:38 |
| Kangie | Hey | 21:38 |
| Kangie | https://blogs.igalia.com/berto/2015/08/14/io-limits-for-disk-groups-in-qemu-2-4/ | 21:38 |
| Kangie | IO limits seem to have landed :D | 21:38 |
| JayF | okay, this is excellent :D | 21:39 |
| Kangie | I got lucky on my first google. :) | 21:39 |
| Kangie | Lemme know how it goes for you please! | 21:39 |
| JayF | I sometimes vapor-lock and don't even know what to look for | 21:39 |
| JayF | I would say hopefully I don't need it | 21:40 |
| Kangie | Hahah true | 21:40 |
| JayF | but my hope dispenser is dry and empty lolo | 21:40 |
| Kangie | I get some hardware to pilot all my new node management stuff | 21:40 |
| Kangie | I'm thinking ironic to image nodes and a Linux head node stack based on containers | 21:40 |
| JayF | well I'll state, for posterity, I don't think we've ever had this issue reported in the real world :D | 21:40 |
| JayF | "linux head node stack" I know all those words separately | 21:41 |
| Kangie | Any caveats about running ironic in a VM? | 21:41 |
| JayF | but can't make sense of them together | 21:41 |
| JayF | Only real caveat to running Ironic-anything-anywhere is that conductors need specific networking | 21:41 |
| Kangie | Like if I wanted to bind an interface in and have ironic do dhco and stuff | 21:41 |
| Kangie | Dhcp* | 21:41 |
| Kangie | Great. That might be a deployment option | 21:41 |
| JayF | depending on how you have it deployed, what services/etc and what you're using, you might also end up with neutron-dhcp-agent running your dhcp | 21:41 |
| JayF | I don't know how much stack you're going to stack | 21:42 |
| JayF | but we should have this chat sometime when my brain isn't mushy from looking at CI logs for hours :) | 21:42 |
| Kangie | Haha. I'll tidy up the slides that a guy gave a good talk on and share em at some point | 21:42 |
| JayF | In fact, I can put aside some time for you if you wanted to chat on it sync | 21:42 |
| JayF | just lmk | 21:42 |
| Kangie | I had to take photos from the back row and the company hasn't responded to my emails | 21:42 |
| Kangie | No rush, I'll be snowed under with procurement until April. | 21:43 |
| Kangie | Good luck with your VM issues | 21:43 |
| JayF | time to riir dnsmasq /s | 21:43 |
| JayF | somehow seems less daunting than tracking down a bug that it seems like our CI is the only thing in the world that can reproduce | 21:43 |
| JayF | we get that a lot (I know nova has that issue r/n in CI with a kernel bug) | 21:43 |
| clarkb | if only we could convince people that these problems are actually problems and likely show up elsewhere people just don't bother to report it | 21:50 |
| JayF | Who is "people" in this case? Most folks working on OpenStack are corporate people, paid to make OSS better. I suspect that's not the case for the author of dnsmasq | 21:52 |
| JayF | like, I agree with the sentiment, I just feel like in the case of dnsmasq ... we are the people | 21:52 |
| JayF | even if we are ill-equipped to be lol | 21:52 |
| fungi | i think he means ironic users are also seeing these problems and just not reporting them to you | 22:01 |
| fungi | and the feeling that they only ever occur in ci jobs is wishful thinking | 22:02 |
| fungi | or nova users in the case of the aforementioned kernel oopses | 22:03 |
| fungi | rather, the ci failures are likely indicative of bugs which also occur in production somewhere and nobody's brought it to the project's attention | 22:04 |
| fungi | we often see bugs in our providers whose signatures/symptoms closely match bugs we "only saw in ci" a few cycles earlier | 22:05 |
| clarkb | fungi: yes exactly the infra team has found over and over again that issues in ci go into prod and then we find them there too | 22:07 |
| clarkb | its just that people don't report them up which makes people think they can avoid fixing them | 22:07 |
| clarkb | and I was specifically thinking of the libvirt/kernel stuff | 22:07 |
| fungi | though not in these cases at least, since they're severely impacting the projects' abilities to test and merge changes | 22:08 |
| clarkb | they push back a lot on us and its fair to a point to get enough info to make things actionable but then they go "oh this isn't the very latest version of libvirt can't help you" | 22:08 |
| clarkb | nevermind millions of hypervisors are running that version | 22:08 |
| fungi | definitely more aggravating when bugs we "occasionally saw in ci" turn into bugs we "frequently see in production" after they make it into a major release and people start upgrading to it | 22:10 |
| TheJulia | i think I have an idea of what is going on | 22:23 |
| JayF | I'd be really interested to see what :D | 22:38 |
| TheJulia | so, with 2.87, it *appears* when we get a second release, dnsmasq can loose its mind | 22:40 |
| TheJulia | it then seems to go poof looking at the dnsmasq systemd journal | 22:40 |
| JayF | 'a second release' meaning what, exactly? | 22:40 |
| TheJulia | looking for that window again :) | 22:40 |
| JayF | DHCPRELEASE x2 for the same thing? | 22:40 |
| JayF | or something else? | 22:40 |
| TheJulia | https://www.irccloud.com/pastebin/gK0vFtNt/ | 22:41 |
| JayF | would we know via logs, for instance, if there was a simultaneous thing happening like a HUP | 22:41 |
| JayF | or do we know this is isolated from any intentional config changes | 22:41 |
| TheJulia | https://www.irccloud.com/pastebin/GkOhsBpS/ | 22:42 |
| TheJulia | but, I don't see that when we go to 2.90 | 22:42 |
| JayF | but instead we see other breakages | 22:43 |
| TheJulia | yeah | 22:43 |
| TheJulia | what if it is another symptom instead of the actual cause | 22:43 |
| * TheJulia looks at a 59 second gap | 22:43 | |
| JayF | hmmm is that consistent across *all* the services? | 22:44 |
| TheJulia | what is that your asking about | 22:45 |
| JayF | e.g. if the VM itself hit a 59 second gap in execution for $busy_reasons, that would be a key insight into what's going on I suspect | 22:45 |
| JayF | I'm saying was teh 60 second gap dnsmasq | 22:45 |
| JayF | or was it *ALL OF THE THINGS* (or at least most) | 22:45 |
| TheJulia | no, the gap more looks like just an artifact of my greps | 22:45 |
| JayF | trying to differentiate between "the whole VM is out to lunch" and "dnsmasq is out to lunch" | 22:45 |
| JayF | ack | 22:45 |
| JayF | right now I'm looking at the code in neutron dhcp agent | 22:46 |
| JayF | that validates that dnsmasq is done with a hup | 22:46 |
| JayF | basically pulling on the thread of "where are there potential races in low performance cases" | 22:46 |
| TheJulia | I'm starting to suspect individual cores are going out to lunch | 22:46 |
| clarkb | you should see that as cpu steal in top and similar | 22:47 |
| clarkb | whats the new tool called copilot? | 22:47 |
| clarkb | though if it is just a deadlock then taht will be less visible | 22:47 |
| TheJulia | apparently I'm on a call scheduled for an hour and everyone is talking how it will be 2 hours 8| | 22:48 |
| JayF | fungi: clarkb: I honestly hope you don't see "sweeping failures under the rug" in Ironic's culture (if you do, lets talk about what that looks like in a DM; I'm interested to know your perspective). We try pretty hard to isolate and fix the issues where they show up, since we have such a huge range of unknown built-in by the nature of there being so much crazy hardware in the world | 22:53 |
| fungi | JayF: i don't think so, no. it's more of a well-honed kneejerk reaction whenever people say things that might sound like they think a bug exists only in ci environments and not in the real world | 22:55 |
| clarkb | no its usually people outside ofopenstack actually | 22:56 |
| fungi | turns out we do a surprisingly good job of emulating real-world environments and bugs people encounter in their ci jobs are almost certainly possible to hit in real deployments | 22:57 |
| clarkb | openstack has the ci issue. Identifies a problem in a dependency. Those maintainers say "thats not ap roblem because its a ci issue" | 22:57 |
| clarkb | the implication being the problem is in our test environment and not in the software being tested | 22:57 |
| clarkb | and it is often difficult to get people to understand that I'm not breaking their software it was already broken | 22:57 |
| fungi | yeah, i do think projects outside openstack have a much higher chance of using non-representative ci systems and assume by default bugs encountered in tests are not real bugs | 22:58 |
| clarkb | for example: we have gotten pusback from qemu/libvirt/kernel types because of things like "no one should actually use qemu" nevermind its useful for all kinds of stuff | 22:59 |
| fungi | we tried from the beginning to test openstack as close to real usage as feasible in a ci system | 22:59 |
| clarkb | and then when we say we can't use nested virt because it crashes all the time they push on that and say "oh well wheere are the bug reports" we can't actually provide theem beacuse we don't have insight into the host kernel | 22:59 |
| clarkb | and when we by some miracle manage that we get told everything is too old and can't be helped | 22:59 |
| clarkb | at every turn we're doing things wrong and yet I did nothing to modify the software or use it beyond its intended scope | 23:00 |
| JayF | I see the most infuriating variant of this frequently in gentoo community: someone pushes a PR fixing a real issue, on real machines people use that they care enough to put PRs up for (things like; this is invalid C and only works on gcc version X) ... and they just get crapped on because the author often says "use GCC version X" and just doesn't care about the broken folks | 23:01 |
| TheJulia | so! we're running low, but not horribly low on ram. we just start to tickle swap even though we have ~1+G free. At least according to the logs. I'm going to try and do a patch on dialing down concurrency a little so we only run one job but there is time to run the cleanup separately | 23:05 |
| TheJulia | oh! I see a difference in our job config | 23:06 |
| TheJulia | 1 cpu on one which is more fail happy, 2 on redfish. Just... "interesting" | 23:06 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Adjust standalone job concurrency https://review.opendev.org/c/openstack/ironic/+/910552 | 23:07 |
| fungi | JayF: i feel like that attitude from some upstreams is exactly why distros end up carrying so many downstream patches in their packaging | 23:16 |
| fungi | in debian they even have a term for them, "hostile upstreams" | 23:17 |
| fungi | some are just notoriously anti-packaging in general and have essentially been flagged as "do not contact" | 23:18 |
| fungi | others tolerate the idea of downstream packaging but don't want to be bothered with any bugs outside their one documented and tightly-scoped deployment model/platform/environment | 23:20 |
| TheJulia | Even packaging friendly teams can sometimes do things which can really shoot packagers in the foot. | 23:37 |
| TheJulia | I suspect turning dstat on might be a good idea on the ironic standalone jobs, but still a pile of super weird | 23:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!