| rpittau | good morning ironic! happy friday! o/ | 07:52 |
|---|---|---|
| *** dking is now known as Guest2233 | 10:20 | |
| iurygregory | good morning! | 11:30 |
| *** mko is now known as Guest2243 | 12:05 | |
| opendevreview | Takashi Kajinami proposed openstack/ironic-python-agent master: Replace crypt module https://review.opendev.org/c/openstack/ironic-python-agent/+/937175 | 12:28 |
| opendevreview | Merged openstack/ironic master: clean up lints for automated_steps doc plugin https://review.opendev.org/c/openstack/ironic/+/937070 | 13:50 |
| TheJulia | good morning | 14:21 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: The i18n function is used but not imported https://review.opendev.org/c/openstack/ironic/+/937255 | 14:45 |
| cardoe | Am I crazy or is ^ broken and should be backported? | 14:45 |
| opendevreview | Merged openstack/bifrost master: Use python version to set DEFAULT_PIP_ANSIBLE https://review.opendev.org/c/openstack/bifrost/+/936797 | 14:48 |
| rpittau | cardoe: mmm that looks broken indeed and it's not the only place apparently | 14:52 |
| cardoe | rpittau: since you're on... may I ask your opinion on which version of Python to target? 3.6/3.7 or 3.9? ala https://review.opendev.org/c/openstack/sushy/+/934916 | 14:55 |
| rpittau | cardoe: 3.9 I doubt we'll backport that :) | 14:58 |
| rpittau | although dtantsur mentioned the oldest supported for the oldest supported branch, but I don't get why since we're not backporting that change (hopefully) | 14:58 |
| cardoe | Well I think his concern is future changes being applied and needing to be backported and using a style that's 3.9 but then needing to be backported to somewhere older. | 14:58 |
| rpittau | yeah, I got to that right when I was answering his comment :) | 14:59 |
| rpittau | so yep 3.7 | 14:59 |
| opendevreview | Merged openstack/ironic stable/2024.2: Update Node Cache after Successful Clean/Service https://review.opendev.org/c/openstack/ironic/+/937107 | 15:00 |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.10 and will be offline. We have allocated an hour for the outage window lasting until 1700 UTC. | 15:02 | |
| rpittau | looks like gerrit wants us to start the weekend early :) | 15:03 |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.10 and will be offline starting at 1600 UTC. We have allocated an hour for the outage window lasting until 1700 UTC. | 15:05 | |
| iurygregory | weekend early ++ | 15:21 |
| iurygregory | time for me to organize more things before traveling :D | 15:21 |
| priteau | Hello. Is there a known issue with IPA master branch at the moment? We are hitting this error in Kayobe CI: | 15:29 |
| priteau | traps: ironic-python-a[495] trap invalid opcode ip:7efee73f58a2 sp:7ffcb6eaf7a8 error:0 in _rust.abi3.so[7efee7112000+631000] | 15:29 |
| priteau | Illegal instruction | 15:29 |
| priteau | Hum, we are still using TinyIPA in these CI jobs | 15:30 |
| TheJulia | heh | 15:32 |
| TheJulia | Ohhh, I wonder what the invalid opcode is | 15:33 |
| TheJulia | Illegal instruction, fun | 15:33 |
| rpittau | bye everyone, have a great weekend! o/ | 15:36 |
| TheJulia | o/ | 15:36 |
| priteau | There's no change in IPA-builder though, so either it's a change in TinyLinux 15 itself, or a change in the virtualisation layer (but it affects both our rocky-9 and ubuntu-jammy jobs) | 15:41 |
| TheJulia | we do *some* compilation in CI | 15:42 |
| TheJulia | it could be a default change in newer ubuntu which doesn't compile in older calls | 15:43 |
| TheJulia | .... cleanest thing to do would be to ensure the VM CPU flags are up to date | 15:43 |
| TheJulia | this is on VMs your creating? | 15:43 |
| priteau | yes, with openstack/tenks | 15:43 |
| TheJulia | okay | 15:43 |
| * TheJulia pulls up tenks to see what it does with the xml | 15:44 | |
| priteau | We can check next week if it resolves itself. The issue only appeared today. | 15:46 |
| TheJulia | priteau: do you have a job link handy? | 15:48 |
| priteau | https://zuul.opendev.org/t/openstack/build/59488a4ffd76437e90045a03d9f9b65b | 15:48 |
| priteau | VM console is at https://2733627e1a0348401721-fb543e4b225e8fdb51d174f0fee213a6.ssl.cf1.rackcdn.com/909686/47/check/kayobe-overcloud-ubuntu-noble/59488a4/primary/kolla/tenks/tk0-console.txt | 15:48 |
| priteau | And a rocky-9 one: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_824/936784/1/gate/kayobe-overcloud-rocky9/8243b42/primary/kolla/tenks/tk0-console.txt | 15:52 |
| priteau | But maybe don't waste time on this quite yet, in case it's a temporary glitch with the latest build. | 15:54 |
| * TheJulia pulls up the libvirt manual | 15:54 | |
| -opendevstatus- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.10 and will be offline momentarily. We have allocated an hour for the outage window lasting until 1700 UTC. | 16:01 | |
| TheJulia | priteau: I'm worried tenks is sending you down a bad path | 16:02 |
| TheJulia | specifically, per the logs, it appears to create bios booting (which is kind of insane these days) VMs of the "pc" type | 16:03 |
| priteau | We switched to uefi by default some weeks ago, I will check if we override this | 16:03 |
| TheJulia | your machine type should also likely be at least pc-q35 also | 16:04 |
| TheJulia | ... In our CI, we have pc-q35-8.2 (https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f15/896570/12/check/ironic-tempest-uefi-redfish-https/f15fb2e/controller/logs/libvirt/qemu/node-0.xml) | 16:04 |
| priteau | Thanks. I will check all this next week. | 16:04 |
| TheJulia | we're then also enabling the cpu host-model to be passed through | 16:04 |
| TheJulia | <cpu mode="host-model" check="partial"/> | 16:04 |
| priteau | Is this relevant even when using pure qemu (no kvm) | 16:05 |
| TheJulia | basically, your getting a VM with a a default processor which has a limited instruction set | 16:05 |
| TheJulia | It is when you have OSes changing/evolving forcing minimum revisions for compile time artifacts to use newer CPU flags | 16:05 |
| TheJulia | for example, centos stream 9, required x86_64 v2. stream 10 requires v3 | 16:06 |
| TheJulia | v3 == haswell processors (from 11 years ago) | 16:06 |
| TheJulia | the VM you have defined... If I'm understanding libvirt properly is x86_64-v1 | 16:06 |
| TheJulia | so if the host OS compiles v2 or v3 flags, yeah, your going to get invalid opcode errors on v1 | 16:07 |
| priteau | We're inside TinyLinux here though. But your comment is relevant for our attempt to switch to c9s IPA | 16:09 |
| TheJulia | priteau: ultimately, it all boils down to whatever the "virsh capabiities" command splits out on the host | 16:09 |
| TheJulia | and possibly nested virt or not | 16:10 |
| priteau | Thanks for the pointers. I will investigate further next week. | 16:10 |
| TheJulia | even in tinylinux, we're building on noble as the base build host at this point for latest artifacts | 16:10 |
| TheJulia | so noble may have it's own spin | 16:10 |
| priteau | I see | 16:11 |
| TheJulia | yeah, its a bit of an onion | 16:11 |
| priteau | Thank you for checking :) | 16:15 |
| JayF | I'll note that there are machines in opendev CI cluster that do not support x86-64-v3 | 16:32 |
| JayF | And if you were running an IPA image based on an OS that requires V3, that's exactly the error you'd get | 16:32 |
| TheJulia | Yeah, i was just thinking about that in the shower | 16:38 |
| TheJulia | if the compilers lean towards v3/v4 automatically as tuning, there is more risk, but I suspect we would have already seen that for ironic's CI... | 16:38 |
| JayF | I doubt that GCC will ever take that automatically, but as you noted above, some distributions have changed their defaults. | 16:46 |
| JayF | My project to give us a gentoo-based CI ram disk is using the non-optimized binaries from them, so it won't be an issue on that. I'd also be surprised if tiny core was willing to make their binaries incompatible with old hardware. I'd be more worried about Ubuntu or centos. | 16:47 |
| * JayF notes he is on mobile today and on a sick day since his back is refusing to operate properly | 16:47 | |
| TheJulia | JayF: the invalid opcode was in ironic-python-agent's runtime | 16:49 |
| TheJulia | so... something which was installed to support it | 16:49 |
| TheJulia | JayF: Find a ghostbusters movie and watch it from a linear surface | 16:50 |
| TheJulia | or any other preferred movie | 16:50 |
| TheJulia | Ghostbusters is only on my mind because I just used "don't cross the streams" | 16:50 |
| JayF | TheJulia: that looks like a cryptography library was installed that was built with V3 | 17:04 |
| JayF | I have a friend who works on that library, I'm asking if they changed anything | 17:06 |
| cardoe | TheJulia: is x86_64 v3 really haswell? I know Intel v3 was Haswell. I didn't think the GCC ABI standardization followed Intel. | 17:28 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: apply line length rules to the doc directory https://review.opendev.org/c/openstack/ironic/+/937269 | 17:30 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: change ambiguous variable name https://review.opendev.org/c/openstack/ironic/+/937270 | 17:30 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: move imports to top of file for lints https://review.opendev.org/c/openstack/ironic/+/937271 | 17:30 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: enable ruff in pre-commit with some initial lints https://review.opendev.org/c/openstack/ironic/+/937272 | 17:30 |
| TheJulia | cardoe: On the intel side, I think v3 starts mapping to the haswell line | 17:32 |
| TheJulia | I think of it much more in processor lines | 17:32 |
| JayF | There's a very good page on the Gentoo wiki that maps them to consumer processors. | 17:33 |
| TheJulia | I think the modeling follows features | 17:33 |
| JayF | Oh for sure, but that makes it even more confusing when you're trying to map it to actual processor models | 17:34 |
| TheJulia | yeah | 17:35 |
| TheJulia | I gave up trying to keep any portion of the tree in my head mapping wise | 17:36 |
| cardoe | yeah it doesn't map. | 17:37 |
| cardoe | x86-64-v1 = AMD K8 and Intel Prescott | 17:37 |
| cardoe | aka the O.G. 64-bit stuff. AMD64 and EM64T | 17:38 |
| TheJulia | yup | 17:38 |
| TheJulia | Which was a *very* long time ago | 17:38 |
| cardoe | x86-64-v2 = AMD Bulldozer / Intel Nehalem. Which for Intel is the architecture before Sandy Bridge which Intel retconned as Intel Core v1 | 17:39 |
| cardoe | x86-64-v3 = AMD Excavator / Intel Haswell. Which happens to be what Intel calls v3. QEMU targets this now for emulation. But older QEMU for Xen still targets x86-64-v1, which could be the issue with some of the CI builders. | 17:41 |
| cardoe | x86-64-v4 = AMD Zen 4 / Intel Skylake | 17:41 |
| cardoe | So all around old stuff. I suspect you're hitting the old QEMU issue, JayF. | 17:41 |
| JayF | I'm personally not hitting anything, I'm just responding to the CI log posted above. | 17:43 |
| dtantsur | side note: any RHEL/CS 9 derivative will require v3 | 17:46 |
| TheJulia | dtantsur: I thought it was v2 for CS9 and v3 for CS10 | 17:47 |
| dtantsur | ah, Julia already said that | 17:47 |
| dtantsur | TheJulia: I may be wrong. I remembered it was v3 but I have memory of a goldfish. | 17:47 |
| TheJulia | maybe more like a pack of goldfish who speak :) | 17:47 |
| TheJulia | I know there are some dicscussions around CS10 suddenly having v3 as a req and how only OpenMetal and Rackspace Flex offers defaults which end up matching | 17:48 |
| * dtantsur a pack of goldfish in a hoodie | 17:48 | |
| TheJulia | Which is going to create pain in different areas | 17:48 |
| TheJulia | dtantsur: heh | 17:49 |
| dtantsur | I see, yeah. The transition to v2 did require updating our virtual machine settings | 17:49 |
| TheJulia | Yeah, and the CS10 discussion is why I'm slightly more aware of this right now | 17:49 |
| TheJulia | Ultimately, in the short term, we might need to see if we can tune the builds for python attributes for tinycore... or just use a binary build leaning towards v2 until other operators can do the needful to document/update so VMs can get launched with newer flags | 17:50 |
| TheJulia | This is very much a feedback loop challenge | 17:50 |
| TheJulia | and we can't let ourselves fall into traps of past consistency | 17:51 |
| dtantsur | wdym "traps of past consistency"? | 17:51 |
| TheJulia | so if we build artifacts which are geared for just v2, then that will work, but we're also avoiding moving forward | 17:52 |
| TheJulia | so we sort of trap ourselves in that past consistent state rooted in ages past | 17:52 |
| TheJulia | does that make sense? | 17:52 |
| dtantsur | Yeah, but I don't think it's on us whether we build stuff for v2 or v3? | 17:53 |
| dtantsur | Like, if you use IPA-builder with CS10, you will get a v3 build | 17:53 |
| TheJulia | Intel Nehalem was released in November 2008.... | 17:53 |
| TheJulia | yup | 17:53 |
| TheJulia | And there *is* a duality, we're going to have to navigate it more and more | 17:54 |
| dtantsur | Or do you mean migrating our official IPA builds to CS 10? | 17:54 |
| dtantsur | I guess it will be dictated on one hand by the support of CS 9, on the other - how many drivers will be removed in CS 10, on the third (WUT, how many hands to I have) - which hardware will only be supported in CS 10 | 17:54 |
| TheJulia | We will want to eventually, but we're functionally blocked with the state of opendev's ci proiders | 17:54 |
| TheJulia | unless we can lock our jobs to specific clouds | 17:54 |
| dtantsur | and that as well | 17:54 |
| dtantsur | (I think we essentially understand each other, just cannot agree on the exact words to express that, lol) | 17:55 |
| TheJulia | yeah, definitely | 17:55 |
| TheJulia | I kind of see this like bios boot versus uefi boot | 17:55 |
| TheJulia | eventually, everyone needs to be uefi | 17:55 |
| * dtantsur +2 | 17:56 | |
| TheJulia | the road for some to get there is going to be different for everyone | 17:56 |
| TheJulia | .... if they are not already | 17:56 |
| dtantsur | Yet, I still see folks resisting UEFI, sometimes for kinda valid reasons, most commonly - out of habit | 17:56 |
| TheJulia | oh wow | 17:56 |
| TheJulia | do they not understand they are limiting all IO to the first 4GB? | 17:56 |
| JayF | One thing worth consideration is whether or not our customers, meaning openstack ironic users, are all new enough hardware. That should be what drives how quickly we move forward, not the decisions of some distributions IMO | 17:56 |
| * TheJulia is semi-mindblown | 17:57 | |
| JayF | For instance, if cardoe is using ironic plus IPA, we know for a fact that Rackspace has some machines that don't meet the spec | 17:57 |
| dtantsur | TheJulia: I did not ask, to be honest (mostly because TIL) | 17:57 |
| dtantsur | Well, it is 0 days since somebody told me "we need IPMI because of old hardware" (and 1 days since the previous instance) | 17:58 |
| JayF | Exactly 😂😂😠| 17:58 |
| TheJulia | From my point of view, I get it takes time to move hardware, I get people might want to keep much older hardware in production for longer if the tax advantage is there and they still have replacement parts. Distributions are trying to make their choices. Even tools are as well. The challenge is also some clouds out there which default to much older defaults. | 18:00 |
| TheJulia | Like... I know one provider is super high end hardware within the last ?2? years.... yet they don't expose v3 vms by default. | 18:01 |
| TheJulia | so there is that lagging state is part of the feedback loop challnege | 18:01 |
| TheJulia | ... actually that gear is likely 3 years old now | 18:01 |
| cardoe | I mean I'm not using those old gross machines, JayF | 18:01 |
| TheJulia | heh | 18:02 |
| TheJulia | cardoe: I already said flex was definitely v3 friendly | 18:02 |
| TheJulia | :) | 18:02 |
| cardoe | Yeah basically anything using paid Xen will probably not be v3 friendly. | 18:02 |
| TheJulia | exactly | 18:02 |
| TheJulia | This is very similar to getting folks to using UEFI VMs by default | 18:03 |
| TheJulia | dtantsur: I've started pushing back on "want to use old hardware" asks with the reality that can burn resources, and at the end of the day it is a business question of support. | 18:04 |
| TheJulia | Distributions actually setting some of these guardrails in place actually also benefits projects like ours | 18:05 |
| dtantsur | I'm pretty lucky here: most of my customers want redfish virtual media which puts limits on how old the hardware might be | 18:05 |
| TheJulia | "no, your not using that super old hardware with a pile of CPU security issues" | 18:05 |
| TheJulia | yeah | 18:05 |
| dtantsur | true | 18:05 |
| TheJulia | Anyway, I should like... try to focus on/think about code today | 18:06 |
| TheJulia | Once I start my second cup of coffee | 18:06 |
| TheJulia | :) | 18:06 |
| dtantsur | ++ | 18:06 |
| dtantsur | I got two cups of coffee in the afternoon and now speed-running though the ethics and compliance training :D | 18:06 |
| TheJulia | le sigh | 18:06 |
| TheJulia | That is likely going to be next week for me | 18:07 |
| dtantsur | It's not bad this time around. I think I'll be done in under 1.5 hours (maybe even less). | 18:07 |
| TheJulia | Last year I think mine ran 4 hours total | 18:07 |
| TheJulia | because the state I live in requires specific amounts of time per year | 18:07 |
| dtantsur | Thanks god, they cut down on videos | 18:07 |
| dtantsur | Ahhh | 18:08 |
| TheJulia | so I end up with like the management ethics course by default | 18:08 |
| dtantsur | ouch :( | 18:08 |
| TheJulia | yeah | 18:08 |
| TheJulia | it is what it is | 18:08 |
| dtantsur | I have a constructive suggestion for them: do 1.5 hours of training, fill the rest with cat videos | 18:08 |
| TheJulia | hehehehe | 18:08 |
| * dtantsur is capable of watching 2.5 hours of cat videos without any issues | 18:08 | |
| TheJulia | I have an orange cat, he alone is sufficient amusement | 18:09 |
| dtantsur | lucky you! | 18:09 |
| TheJulia | except when he wants to walk across or sit on the laptop | 18:09 |
| dtantsur | cat do what cat must | 18:10 |
| TheJulia | indeed! | 18:10 |
| * TheJulia sips coffee | 18:10 | |
| dtantsur | FYI folks. Our entire team will have an onsite meetup next week. Expect delayed responses and people being in and out of IRC. | 18:25 |
| dtantsur | (in fact, I won't have access to my IRC bouncer and will need to invent something) | 18:27 |
| cardoe | dtantsur: you see my comments about Python 3.6 vs 3.7? | 18:30 |
| dtantsur | cardoe: where? | 18:30 |
| cardoe | https://review.opendev.org/c/openstack/sushy/+/934916 | 18:31 |
| * TheJulia whispers "irccloud...." | 18:33 | |
| TheJulia | :) | 18:34 |
| dtantsur | TheJulia: yeah, I'll probably set it up over the weekend. | 18:34 |
| TheJulia | Have a great weekend and travel safely! | 18:34 |
| * TheJulia tries to figure out how she broke image artifact checksum verification suddenly | 18:35 | |
| dtantsur | cardoe: responded | 18:37 |
| dtantsur | TheJulia: thanks! | 18:37 |
| dtantsur | See you folks o/ | 18:37 |
| TheJulia | err, its the mocking, not my code | 18:40 |
| TheJulia | doh | 18:40 |
| cardoe | I replied to your reply | 18:44 |
| iurygregory | bye everyone, see you in two weeks o/ | 19:36 |
| TheJulia | woot, in a good palce to make more progress on oci image registry support next week | 19:54 |
| TheJulia | my substrate/flow changes now have no failing unit tests | 19:55 |
| TheJulia | now to just begin to bolt in the client stuffs | 19:55 |
| cardoe | So am I able to define extra verifying steps? | 19:56 |
| cardoe | When adding a node? | 19:56 |
| cardoe | I'm REALLY REALLY wanting to figure out how to get rid of our custom node ingestion and do it all within Ironic steps/hooks/etc. | 19:56 |
| cardoe | I wanna try to then document some examples for folks. | 19:56 |
| TheJulia | Dunno | 20:05 |
| TheJulia | I honestly don't remember the verifying steps stuff and why | 20:05 |
| TheJulia | but I suspect a driver could override/inject at a minimum | 20:05 |
| dxterslab | hey guys! do you know how openstack handles source nat for baremetal devices? is this done with ovn and integrating the ovn gw chassis with the physical network. (assume no gpu) | 20:57 |
| dxterslab | i mean assume no dpu | 20:57 |
| JayF | Btw, cryptography did ship a new wheel that depends on a newer glibc recently, so it is possible that's an actual break. | 21:28 |
| cardoe | JayF: you happen to peek at ruff and see any of the other flake8 modules you think would be good for us to use? | 21:36 |
| opendevreview | Doug Goldstein proposed openstack/sushy master: enable flake8 logging checks in ruff https://review.opendev.org/c/openstack/sushy/+/937068 | 21:39 |
| opendevreview | Doug Goldstein proposed openstack/sushy master: enable pyupgrade via ruff to Python 3.7 https://review.opendev.org/c/openstack/sushy/+/934916 | 21:39 |
| TheJulia | pity dxterslab didn't hang out | 21:40 |
| TheJulia | could have answered that | 21:40 |
| TheJulia | JayF: hmm. Quite possible | 21:40 |
| cardoe | Yeah I was actually wanting to discuss that with dxterslab | 21:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!