| rpittau | good morning ironic! o/ | 08:01 |
|---|---|---|
| rpittau | TheJulia, JayF, I've approved the release mappings change, thanks for that, and apologies for not adding that before... | 08:03 |
| rpittau | I took a note to add to verify release mappings to our release docs, it's been forgotten too many times, it needs to be part of the release process | 08:03 |
| opendevreview | Merged openstack/ironic master: Update release mappings https://review.opendev.org/c/openstack/ironic/+/929839 | 09:27 |
| masghar | Interestingly, I am getting the same results with python 3.11: ironic.tests.unit.common.test_format_inspector.TestFormatInspectors.test_vmdk_bad_descriptor_mem_limit and test_vmdk_bad_descriptor_mem_limit_stream_optimized are failing with error 127 | 10:03 |
| masghar | Is there a pip install dependencies command I should be running before? | 10:03 |
| dtantsur | masghar: I saw this in our downstream fork CI. I just removed these tests from our fork. | 10:04 |
| masghar | I just have a clean Centos 9 with ironic | 10:04 |
| dtantsur | but it's cool that you can reproduce it locally: I could not on my Fedora | 10:04 |
| dtantsur | try running the same commands its running just in the shell? | 10:04 |
| masghar | dtantsur: you can just remove the tests? =D | 10:04 |
| dtantsur | masghar: well.. format_inspector is imported code from ??nova??. We don't care about its tests too much. | 10:05 |
| dtantsur | realistically, we should remove it before the next release and replace with access to some oslo library | 10:05 |
| dtantsur | masghar: yeah, this is the source: https://opendev.org/openstack/oslo.utils/src/branch/master/oslo_utils/imageutils/format_inspector.py | 10:05 |
| dtantsur | maybe JayF and TheJulia will appreciate help with replacing our copy of the module with oslo.utils | 10:06 |
| masghar | I see that this code was added to handle the CVE last month | 10:11 |
| masghar | Alright, I will ignore it in favor of my current task now, and hopefully circle back to this later | 10:12 |
| masghar | Thanks dtantsur! | 10:12 |
| dtantsur | masghar: correct. Since we needed to backport the CVE fix to all branches in existence, we could not rely on a new oslo.utils version. | 10:13 |
| dtantsur | for the master branch though, we totally can | 10:13 |
| * dtantsur is pondering Just Do It | 10:13 | |
| masghar | Yeah for the master branch we can clean things up I think | 10:14 |
| opendevreview | Will Szumski proposed openstack/bifrost master: Fix checksum parsing for Rocky Linux https://review.opendev.org/c/openstack/bifrost/+/929753 | 10:21 |
| masghar | dtantsur: so I switched user to root, and all is passing now | 10:24 |
| masghar | I have a feeling my regular user (with passwordless sudo access) didnt have permissions to create files in /tmp, which is what the qemu-img command was trying to do | 10:25 |
| masghar | I switched back to my old user and its passing now as well | 10:33 |
| dtantsur | sigh | 10:34 |
| masghar | The only difference I can think of is that I now also have tox installed for root, and now its using the root/system-wide tox by default? | 10:35 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Replace image_format_inspector with its oslo.utils version https://review.opendev.org/c/openstack/ironic/+/929904 | 10:44 |
| dtantsur | TheJulia: dunno if you had it planned already, but since you already have a lot on your plate, so ^^^ | 10:44 |
| opendevreview | Mark Goddard proposed openstack/tenks master: Change default boot mode to UEFI https://review.opendev.org/c/openstack/tenks/+/827553 | 10:49 |
| opendevreview | Merged openstack/tenks stable/2.0: CI: Remove CentOS Stream 8 jobs https://review.opendev.org/c/openstack/tenks/+/928775 | 11:14 |
| dtantsur | anyone else wants to read https://review.opendev.org/c/openstack/ironic-specs/+/926654 (disable_power_off spec)? Has 2x +2 already, I'd rather merge it by EOW if possible. | 11:29 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: The adoption test should not depend on the iscsi deploy https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927029 | 11:36 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: Check inspection data and abortion in the standalone tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927928 | 11:36 |
| dtantsur | rpittau: FYI https://review.opendev.org/c/openstack/ossa/+/928005 is blocked on your approval | 11:48 |
| TheJulia | dtantsur: I had not planned it directly, but thanks! | 12:26 |
| rpittau | dtantsur: ack, approved | 12:37 |
| rpittau | masghar: it should still use at least tox 4.4.0 as it's the minversion required in tox.ini | 12:52 |
| masghar | rpittau: yep my tox was the most recent, I don't think it was <4.4.0 | 13:37 |
| opendevreview | Merged openstack/bifrost master: Restart systemd services on failure https://review.opendev.org/c/openstack/bifrost/+/884198 | 13:44 |
| tkajinam | dtantsur, hi ! I wonder if you have any update about the plan to retire ironic-inspector ? I guess the new inspection feature within ironic will no longer be experimental in 2025.1 but I'd like to know if that's the right expectation. | 14:01 |
| tkajinam | (It's not urgent but I want to understand the timeline to determine when I spare time to update puppet-ironic. | 14:01 |
| dtantsur | tkajinam: the new inspection is no longer experimental now, and ironic-inspector is deprecated already | 14:02 |
| tkajinam | dtantsur, , ah, ok. I was looking at https://github.com/openstack/ironic/blob/master/doc/source/admin/inspection/pxe_filter.rst?plain=1#L9 . probably this can be removed now. | 14:04 |
| tkajinam | (and a few other descriptions which say the feature is experimental | 14:05 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: doc: Promote built-in introspection from experimental https://review.opendev.org/c/openstack/ironic/+/929936 | 14:08 |
| dtantsur | tkajinam: the PXE filter itself is sorta experimental | 14:13 |
| dtantsur | my hope was that someone who actually cares about PXE filters comes and provides feedback.. but it has not happened | 14:17 |
| cardoe | JayF: https://review.opendev.org/c/openstack/ossa/+/928005 I think is in your queue | 14:17 |
| JayF | That change is from me | 14:22 |
| JayF | I can't core review my own change | 14:22 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: doc: Promote built-in introspection from experimental https://review.opendev.org/c/openstack/ironic/+/929936 | 14:23 |
| dtantsur | JayF: you can remove W-1 though | 14:24 |
| JayF | Oh, that is true but I need to check to make sure I've done all the things that are needed to do for ironic | 14:25 |
| JayF | I forgot I was waiting for Riccardo to come back | 14:25 |
| JayF | I can assure you that whether that patch is merged or not, as a member of the VMT I'll make sure that ironic is treated the same way as a VMT project would be even if technically something comes up before that landed | 14:26 |
| rpittau | thanks for that patch JayF :) | 14:31 |
| rpittau | JayF, dtantsur, TheJulia: FYI I just submitted the final release for ironic for dalmatian, I guess we're good but please let me know if there's something missing | 14:32 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: Check inspection data and abortion in the standalone tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927928 | 14:37 |
| dtantsur | fix for missing test coverage: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927029 | 14:38 |
| dtantsur | rpittau: I'd like to get https://review.opendev.org/c/openstack/ironic/+/929904 in but since it's failing the CI... I guess no | 14:39 |
| rpittau | dtantsur: I can hold the ironic release until next week | 14:39 |
| dtantsur | rpittau: maybe it's not worth it really | 14:39 |
| dtantsur | is there a way to fit https://review.opendev.org/c/openstack/bifrost/+/929770 into bifrost final? | 14:40 |
| rpittau | if it lands before EOW yes | 14:41 |
| rpittau | I've +2 ed it | 14:41 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Work around missing microversion headers on / https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928120 | 14:43 |
| dtantsur | ouch | 14:44 |
| dtantsur | ouch-ouch | 14:44 |
| dtantsur | Error while preparing to deploy to node f040ad05-f081-427b-be20-3031e181e2e6: Image eecbaef9-2e15-4abd-a433-a9e4a133659d is unacceptable: Converted to raw, but format is now gpt: ironic.common.exception.ImageConvertFailed: Image eecbaef9-2e15-4abd-a433-a9e4a133659d is unacceptable: Converted to raw, but format is now gpt | 14:44 |
| dtantsur | TheJulia: I think the new version of the format inspector is broken for us ^^^ | 14:44 |
| rpittau | going to include the ironic-tempest-plugion fix in the current release patch | 14:45 |
| dtantsur | mmm, it might be easy to fix, okay | 14:46 |
| dtantsur | TheJulia: nm, it comes from our code. but it's interesting that now the image format can be "gpt" or "raw" | 14:50 |
| dtantsur | I wonder if we need to add it to allowed formats, hmmm | 14:51 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Replace image_format_inspector with its oslo.utils version https://review.opendev.org/c/openstack/ironic/+/929904 | 14:52 |
| TheJulia | dtantsur: they likely added a gpt option as well | 15:03 |
| TheJulia | dtantsur: there was chatter about doing so, I haven't looked at the latest code | 15:04 |
| TheJulia | that... raw no longer means raw thing is really not great for us | 15:04 |
| TheJulia | because we will have to learn that gpt also means raw | 15:04 |
| clarkb | well and qcow2 can also have gpt partitioning within it? | 15:05 |
| clarkb | seems odd to coalesce things like that rather than say return a tuple raw,gpt qcow2,mbr | 15:05 |
| TheJulia | Indeed | 15:06 |
| JayF | dtantsur: rpittau: I am -.5 to merging a change to use the oslo.utils version before we release; if for no other reason than knowing we've done a lot of testing on the existing way, and it seems late to intorduce that | 15:10 |
| JayF | and I think chatter in here is pretty good evidence that waiting might be wise | 15:11 |
| * JayF does not want to have to backport an ironic fix via oslo utils | 15:11 | |
| JayF | TBH with those changes re: raw/gpt it almost makes me nervous to use it /at all/ unless we cross-gate | 15:15 |
| JayF | (which basically means I'm saying we should cross-gate :D) | 15:16 |
| rpittau | JayF: I see your point, it makes sense | 15:17 |
| rpittau | we can introduce the change at the beginning of the next cycle and we'll have much more time to test it | 15:17 |
| JayF | that's exactly what I was thinking | 15:18 |
| JayF | and is why I haven't been prioritizing doing the IPA equivalent change | 15:18 |
| JayF | wanted to do a little refactor there too but didn't want to land it until start of epoxy | 15:18 |
| rpittau | I will unlock the ironic release, I don't see other patches that are blocking it | 15:19 |
| rpittau | dtantsur: sorry, the bifrost final release already been done at the end of August as requested to happen before September 13th https://review.opendev.org/c/openstack/releases/+/927153 | 15:36 |
| rpittau | we can try to squeeze another minor release before end of tomorrow, a bit rushy :) | 15:37 |
| JayF | question, sorry to be the fun police, but 929770 looks nice, but the release note says explicitly it can be breaking to existing VMs | 15:39 |
| JayF | does that mean it'd need to be a *major* bump? | 15:39 |
| JayF | I only ask to frontrun what might be a similar question from release team when that PR comes up | 15:39 |
| rpittau | JayF: that's only if you migrate the vm from an older distro though | 15:40 |
| JayF | oh, so it's more like | 15:41 |
| JayF | if you upgrade the ubuntu on the box bifrost is running | 15:41 |
| JayF | AND you upgrade bifrost (which you'd have to do to do ^^^) | 15:41 |
| JayF | you will have to pat the VMs on the head to be happy again | 15:41 |
| JayF | if that's right, I'll +2A that right now | 15:41 |
| rpittau | yep | 15:41 |
| rpittau | JayF: one more small fix if you have the time, thanks! https://review.opendev.org/c/openstack/bifrost/+/929769 | 15:45 |
| JayF | anything else you need me to poke at? | 15:45 |
| rpittau | actually yes :P | 15:47 |
| rpittau | last one, I promise https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927029 | 15:47 |
| JayF | I don't mind, it's literally my job :D | 15:48 |
| JayF | lmao | 15:48 |
| rpittau | :D | 15:48 |
| JayF | that change is sad and funny and sad | 15:48 |
| rpittau | heh :/ | 15:48 |
| rpittau | good night! o/ | 15:56 |
| TheJulia | just going to note, it is okay to backport fixes | 15:59 |
| TheJulia | It is not the end of the world | 15:59 |
| TheJulia | stuff happens :) | 15:59 |
| opendevreview | cid proposed openstack/virtualpdu master: WIP: Vendor pysnmp-lextudio into virtupdu https://review.opendev.org/c/openstack/virtualpdu/+/928823 | 16:12 |
| JayF | oooh I've been waiting to see how this goes | 16:17 |
| JayF | the rough equivalent of putting that project on life support, but for our use case that's probably OK | 16:17 |
| dtantsur | this is how you learn how much was dependent on one person.... | 16:18 |
| JayF | yep; also see the thread on dhcp/dnsmasq on the list: one dude in the UK | 16:19 |
| JayF | we need to do a better job of reconizing impacts while folks are around, before they are gone/burned out/etc | 16:19 |
| dtantsur | JayF: just read it.. yeah. I'll try to be a part of this conversation: the DHCP limitations around HA have been a recurring topic in the metal3 community. | 16:21 |
| JayF | the one thing I'm going to bring up in the PTG | 16:22 |
| JayF | is like, they are talking about pluugging it into the agent, which is good | 16:22 |
| JayF | but I also wonder with the way kea has it's own set of distributed features, if there's a way to orient it where kea replaces the agent if you want | 16:22 |
| JayF | IDK if that's reasonable/possible yet, but I wanna ask the question | 16:22 |
| dtantsur | it's a good question | 16:22 |
| dtantsur | especially if a lot of code will need to be written for Neutron.. it's the code we won't benefit in metal3/bifrost | 16:23 |
| JayF | right now, we're allocating time for cid to do *both* an Ironic and neutron backend for kea | 16:25 |
| JayF | even though in the short term only the neutron agent is directly valuable to GR | 16:25 |
| dtantsur | please absolutely do reach out to me with questions like "what the hell does a standalone deployment actually need from DHCP?" | 16:26 |
| JayF | I don't have to ask those questions, I think | 16:26 |
| JayF | we just have to implement an existing interface | 16:26 |
| JayF | this is one of the things that's nice about this, hopefully we're just implementing like 4-6 methods that look roughly the same across ironic and neutron | 16:27 |
| dtantsur | yeah, it's probably more than enough | 16:27 |
| JayF | *hopefully* 🤞 | 16:27 |
| dtantsur | module unmanaged inspection :-P | 16:27 |
| dtantsur | * modulo | 16:27 |
| TheJulia | I think an ideal case is if we need to do any record prep/management in ironic, we teach ironic what to do to send it across if needed, if neutron doesn't do translation, and our own for metal3/bifrost becomes a lot easier | 16:31 |
| TheJulia | neutron integrated makes sense for those huge deployments though | 16:31 |
| JayF | honestly if we can't implement a kea backend that is transparent to swap out with dnsmasq | 16:31 |
| JayF | as at least one of the options | 16:31 |
| TheJulia | there is a weird matrix path there we just need to be mindful of and not ignore slightly different cases | 16:32 |
| JayF | I'll consider the project unsuccessful from my POV | 16:32 |
| TheJulia | (which we're kind of good at, really) | 16:32 |
| JayF | (as mentioned in mailing list, that may end up being kea+unbound) | 16:32 |
| TheJulia | cool cool | 16:33 |
| TheJulia | also, from one point of view, it is about learning | 16:33 |
| TheJulia | there is tons of logic in doing that learning | 16:33 |
| clarkb | I meant to do a bit more digging to make sure my unbound setup isn't just bad, but it doesn't do online reloads of its records iirc | 16:33 |
| TheJulia | and our skin internallh in ironic is mainly an option number to alias difference | 16:33 |
| JayF | clarkb: it works for me locally in opnsense context | 16:33 |
| JayF | clarkb: DHCP hostnames automatically update in unbound | 16:34 |
| clarkb | JayF: ya with an unbound outage | 16:34 |
| JayF | so there's *some* working dynamic update mechanism | 16:34 |
| TheJulia | and dnsmasq core dumps on reload so.... | 16:34 |
| JayF | wait, are you saying it restarts each time!? | 16:34 |
| TheJulia | pick your poison! | 16:34 |
| JayF | I don't wanna believe you but I know you don't make crap up LOL | 16:34 |
| clarkb | I'm running pfsens so it is similar, but I also run pfblockerng whcih puts a ton of records in unbound and I get like a 2 minute outage from unbound to update records | 16:34 |
| dtantsur | dnsmasq does have inotify support | 16:34 |
| clarkb | so I've completely disabled dynamic allocations and dhcp hands out static leases whcih I have statically configured in unbound to minimize the downtime | 16:34 |
| JayF | This would explain so much on my network | 16:35 |
| clarkb | its possible that pfsense is using unbound incorrectly/poorly but its a major pain point for me locally | 16:35 |
| johnsom | Hmm, I am pretty sure unbound does hitless reloads as well.... | 16:35 |
| JayF | bceause I have a LOT of iot devices | 16:35 |
| clarkb | JayF: part of the issue to be fair is that I probably have a couple 100k records in bound due to pfblockerng | 16:35 |
| clarkb | a lot of iot devices is still on the order of 100s and restarts might be quick and painless | 16:36 |
| clarkb | johnsom: and ya it is entirely possible that pfsense is using unbound poorly. I should do more digging | 16:36 |
| JayF | clarkb: yeah, I see no evidence it's hard restarting, but I'll keep an eye out for blips in the future | 16:37 |
| johnsom | There is even a "reload_keep_cache" unbound-control option that saves the cache | 16:38 |
| JayF | clarkb: one thing I've done at old jobs, to keep that problem at bay with RBL lookups, was have a two tiered DNS: first tier was local stuff that got updated, second tier caching layer was everything else + RBLs | 16:38 |
| johnsom | For the ironic folks, I am starting to look at the concept of "private DNS servers" I.e. servers that handle tenant network requests. OVN has very limited support for this today. I am wondering if you have any feedback on "per-network" dns resolution or "per tenant" or "as many private DNS servers as they want" choices. | 16:41 |
| clarkb | ok ya looks like unbound-control should be able to do it. I wonder if pfsense is just restarting the actual service instead because why do the online thing. | 16:41 |
| TheJulia | dtantsur: if memory serves, the dnsmasq bug is related to the add/remove/update of records regardless of the trigger inotify or hup, but I think we were just focusing on hup because we coudl reproduce it in the code path on semi-demand (besides, neutron would hup it like 20-30 times as part of some of ironic's tempest suite runs | 16:41 |
| johnsom | The systemd service also has reload implemented, not sure on the cache saving option thought | 16:41 |
| clarkb | johnsom: as a non ironic user but an openstack user we've explicitly avoided any cloud provided dns because its almost always broken... | 16:42 |
| JayF | johnsom: I'm not sure Ironic cares that much about DNS, aside from if users are putting hostnames in image urls and the like | 16:42 |
| TheJulia | johnsom: would some PTG time help? | 16:42 |
| JayF | ++ PTG time even if it's a quickie | 16:42 |
| TheJulia | JayF: I think we care about $thing being reachable from the physical wire | 16:42 |
| * JayF doesn't know what he might not know | 16:42 | |
| johnsom | Sure, happy to join | 16:42 |
| TheJulia | johnsom: could you add a note to our etherpad? | 16:43 |
| JayF | TheJulia: I guess I didn't consider that "it works at all" was a BM-specific feature | 16:43 |
| clarkb | and its on my todo list to look into converting to dns over tls because ov the ovn mitm "attack"/feature | 16:43 |
| JayF | TheJulia: but I guess we can't take anything for granted, we learned that re: OVN+MTU :( | 16:43 |
| TheJulia | JayF: different approaches, thinking back to the pmtu stuffs | 16:43 |
| johnsom | clarkb Yeah, the OVN situation is.... interesting. | 16:43 |
| clarkb | I guess my feedback is that please don't subvert my decision to use external dns if you implement things closer to the cloud tenant too | 16:43 |
| TheJulia | JayF: and the lack of arping dhcp responses | 16:43 |
| TheJulia | JayF: "hey, this is grub, I want to arp my sever to verify.... helloooo.... helllooooooo out there *falls over" | 16:44 |
| JayF | You have probably told me before and I blocked it outta my mind due to pure absurdity | 16:44 |
| johnsom | TheJulia Is this the right etherpad? It's blank. https://etherpad.opendev.org/p/oct2024-ptg-ironic | 16:44 |
| TheJulia | johnsom: no, one moment | 16:44 |
| JayF | https://etherpad.opendev.org/p/ironic-ptg-october-2024 | 16:45 |
| JayF | I'll go fix ptg | 16:45 |
| TheJulia | That one! | 16:45 |
| johnsom | Thanks | 16:45 |
| johnsom | I have the same issue with the ptg links. lol | 16:45 |
| JayF | ptg website is fixed | 16:46 |
| TheJulia | thanks JayF! | 16:46 |
| * JayF put a warning in the wrong etherpad | 16:57 | |
| JayF | cid: https://bugs.launchpad.net/ironic/+bug/2078361 would be a good bug to pick up if you have capacity | 17:03 |
| * cid goes looking | 17:04 | |
| JayF | I'll note there may be other docs-build related bugs which would be good to poke at, too, might be something that can be knocked out together | 17:05 |
| cid | I have had 2078361 in a pinned tab for a while since I triaged it. | 17:07 |
| JayF | yeah, that's also one that may interact with a to-be-merged requirements change | 17:07 |
| JayF | we just need to make sphinx be happy in most cases | 17:07 |
| opendevreview | Merged openstack/bifrost master: Support OVMF with 4M flash storage https://review.opendev.org/c/openstack/bifrost/+/929770 | 17:17 |
| opendevreview | Verification of a change to openstack/ironic-tempest-plugin master failed: The adoption test should not depend on the iscsi deploy https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927029 | 17:25 |
| opendevreview | Merged openstack/bifrost master: Fix missed occurrence of SETUPTOOLS_USE_DISTUTILS https://review.opendev.org/c/openstack/bifrost/+/929769 | 17:38 |
| opendevreview | cid proposed openstack/ironic master: [WIP] Add inspection rules https://review.opendev.org/c/openstack/ironic/+/918303 | 17:53 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/929171 and it's backported bretheren could use a land in unmaintained branches (the MTU fix from a couple weeks abck) | 18:06 |
| TheJulia | done | 18:29 |
| JayF | ty | 18:30 |
| JayF | btw, I told the security team I'm ready for that ossa patch to merge | 18:31 |
| JayF | I haven't reconfigured launchpad because after spending a half hour poking at it I couldn't find the setting | 18:31 |
| JayF | leading me to assume: 1) I can't find it (always possible) or more likely, 2) it needs perms I don't have | 18:31 |
| JayF | but i have an ask out for help | 18:31 |
| TheJulia | What is the setting? | 18:43 |
| JayF | #4: https://security.openstack.org/repos-overseen.html#requirements | 18:45 |
| TheJulia | hmm, dunno | 19:52 |
| TheJulia | I thought there was at one time | 19:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!