| *** dmellado7452 is now known as dmellado745 | 00:15 | |
| opendevreview | Jay Faulkner proposed openstack/ironic-tempest-plugin master: Basic API tests for sharding https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/906749 | 00:40 |
|---|---|---|
| JayF | ^ should be ready for review, I'll #ironic-week-prio it tomorrow once it V+1s, but if you wanna sneak preview review it I won't mind | 00:41 |
| TheJulia | Cool, I’ll take a look tomorrow. Thanks Jay! | 00:41 |
| Kangie | Hi all, having finally worked around the... "fun" nature of my employer's network and managed to get bifrost-cli setup from repos in a VM, what's going to be the best way to do a test virtual deployment? Ideally I'd like to run ironic and a few test VMs on the same host on an internal network to get familiar with it before proceeding on prod hardware. Is it sane to set the | 06:15 |
| Kangie | provisioning network as an internal net and use DHCP/PXE? The libvirt guide cautions against it however I suspect that has to do more with managing bare metal servers at the same time than any real limitation | 06:15 |
| Kangie | Otherwise a few VMs and a sushy emulator seems reasonable enough if I'm reading my docs right? | 06:16 |
| rpittau | good morning ironic! o/ | 08:44 |
| dtantsur | Kangie: there is a testenv mode that can be used in complete isolation from your production environment. I'd avoid trying to use the same bifrost to do both: it's possible, but the configuration may be overwhelming at first. | 09:13 |
| iurygregory | good morning | 11:16 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Online migration for inspect_interface inspector->agent https://review.opendev.org/c/openstack/ironic/+/907398 | 12:24 |
| dtantsur | TheJulia, JayF ^^^ | 12:24 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Add inspection PXE filter https://review.opendev.org/c/openstack/ironic/+/907991 | 13:03 |
| dtantsur | TheJulia, hjensas ^^ without purging with a doc addition | 13:04 |
| hjensas | dtantsur: thank, I like the "leave the purging to whatever is managing the service approach" :) | 13:51 |
| TheJulia | good morning | 13:59 |
| opendevreview | Merged openstack/ironic-python-agent master: Trivial: avoid deprecated utcnow https://review.opendev.org/c/openstack/ironic-python-agent/+/907298 | 14:43 |
| iurygregory | https://nvd.nist.gov/vuln/detail/CVE-2023-40547 Shim CVE | 14:45 |
| TheJulia | Yeah, when downloading using http | 14:59 |
| JayF | I saw that. It's nice when the security issues that might impact your users are juuuuuust over the horizon of scoped into stuff we care about lol | 16:04 |
| TheJulia | dtantsur: do you remember how tight the https cert validation is ? | 16:23 |
| dtantsur | Define "tight" please | 16:24 |
| TheJulia | specifically we've run into issues with clocks recently where it seems a firmware timezone is mucking with the time | 16:26 |
| dtantsur | yeah, we've run into that too recently | 16:28 |
| TheJulia | on hpe gear | 16:28 |
| TheJulia | ? | 16:28 |
| dtantsur | not sure any more, sorry. possibly. | 16:28 |
| dtantsur | TheJulia: https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/config.py#L77-L80 | 16:29 |
| TheJulia | ack, thanks | 16:29 |
| dtantsur | we may bump that to 24 hours if we keep seeing timezone problems | 16:29 |
| clarkb | I think there is still a dos in the content length handling in shim, but if you're talking to a nefarious http server you have bigger problems. If you set a terabyte large content-length value you'll clear the new check around buf sizes (because a terabyte is larger than whatever you read) then fail to allocate a buffer for it. | 16:34 |
| clarkb | but also its been many years since I dealt with C so my reading may be wrong | 16:35 |
| JayF | For ironic cases, if you're close enough to own the http server, you're likely close enough to screw with PXE as well | 16:35 |
| JayF | so this only really is something that's impactful for UEFI HTTP boot + vmedia, I think (?) | 16:35 |
| dtantsur | The vmedia case have everything inside the ISO | 16:36 |
| JayF | yeah, well I also realized I'm thinking only about provisioning actions cases | 16:37 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Add inspection PXE filter https://review.opendev.org/c/openstack/ironic/+/907991 | 16:37 |
| JayF | and there may be something in this category around deployment on disk | 16:37 |
| JayF | but we use grub shim from locally installed package on OS, yeah? | 16:37 |
| JayF | so that is not an issue | 16:37 |
| opendevreview | Julia Kreger proposed openstack/ironic master: docs: add additional content for host clock skew https://review.opendev.org/c/openstack/ironic/+/908511 | 16:52 |
| TheJulia | dtantsur: ^ | 16:52 |
| TheJulia | so, httpboot in general, since there are two modes, the vulnerability is when your downloading grub as part of the pure network boot path where basically shim expects to be able to grab grub at $CWD/grubx64.efi | 16:54 |
| TheJulia | the key is you can still feed a stock shim with a signed artifact | 16:55 |
| TheJulia | from disk, doesn't apply of course | 16:55 |
| rpittau | good night! o/ | 17:30 |
| * TheJulia looks at https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-guide.html and twitches from reference to pxelinux.0 | 17:54 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: docs: augment admin troubleshooting docs for system scope context https://review.opendev.org/c/openstack/ironic/+/908203 | 18:21 |
| iurygregory | TheJulia, the most recently clock skew issue was in a Dell Poweredge R650 | 18:33 |
| TheJulia | funky | 18:38 |
| TheJulia | so, likely we do just need to extend the skew | 18:38 |
| JayF | maybe find a way, if properly configured, to have IPA to set the BMC clock on boot, perhaps? | 18:39 |
| TheJulia | dtantsur: so, out of curiosity, could a node get stuck in inspectfail ? I'm thinking not really, but it is a fear screaming in the back of my head | 18:40 |
| TheJulia | JayF: BMC clock is not the issue, although most people mistake it for the host clock | 18:40 |
| dtantsur | TheJulia: should not be possible, you can always "manage" or whatever it | 18:41 |
| TheJulia | Yeah, just worried we're requiring humans to know to get it out of that state, but... yeah | 18:41 |
| dtantsur | Doesn't it apply to all fail states? I might be missing some context | 18:41 |
| TheJulia | in terms of the online upgrade, but I guess it is fine, really | 18:42 |
| TheJulia | I'm likley just over thinking | 18:42 |
| dtantsur | It's a good topic to discuss. Potentially, tear down may fail with the new interface. BUT! I'm now seriously wondering if we have any tear down at all :D | 18:43 |
| * dtantsur should go packing | 18:46 | |
| TheJulia | packing?!? | 18:50 |
| dtantsur | leaving for vacation on Saturday | 18:51 |
| TheJulia | \o/ vacation | 18:53 |
| TheJulia | .... I need a vacation and it is just Feburary 8th.... | 18:53 |
| JayF | https://zuul.opendev.org/t/openstack/build/bf6c998654024902b5281a0d745ea725 | 21:32 |
| JayF | I set max_microversion on my change | 21:32 |
| JayF | but yet still > Version 1.89 was requested but the minor version is not supported by this service. The supported version range is: [1.1, 1.87]. | 21:32 |
| JayF | in 2023.2 jobs (and I suspect zed jobs are failing for similar reasons) | 21:33 |
| * JayF trying to figure out what he's missing | 21:33 | |
| JayF | s/max_/min_/ | 21:34 |
| JayF | hmm | 21:40 |
| JayF | do those jobs need max_microversion set in config? | 21:40 |
| JayF | I think tempest is thinking the max microversion for 2023.2 is 1.87 when in reality it should be 1.89? | 21:41 |
| TheJulia | afaik, no | 21:42 |
| iurygregory | JayF, release mappings is correct for 2023.2? | 21:42 |
| JayF | There's def. something weird going on | 21:42 |
| JayF | That's exactly what I'm looking for | 21:42 |
| iurygregory | because I only see 2023.1 there | 21:42 |
| iurygregory | https://github.com/openstack/ironic/blob/master/ironic/common/release_mappings.py | 21:42 |
| JayF | yikes, that's a big miss | 21:43 |
| iurygregory | maybe is related, not sure | 21:44 |
| JayF | that file is missing a LOT of information | 21:44 |
| JayF | that's for damn sure | 21:44 |
| iurygregory | I will try to look tomorrow | 21:44 |
| JayF | 23.0.0 was our 2023.2 release | 21:44 |
| JayF | and is not in that file at all | 21:44 |
| iurygregory | need to go grab dinner with the family =) | 21:44 |
| JayF | I mean, I'm looking now | 21:44 |
| JayF | so hopefully you'll have a review or it'll be fixed | 21:45 |
| TheJulia | so why 1.89 ? | 21:45 |
| JayF | sharding. | 21:45 |
| iurygregory | sharding was before 1.89 no? | 21:45 |
| JayF | https://docs.openstack.org/api-ref/baremetal/#id236 says 1.82 (!?) | 21:45 |
| TheJulia | yeah, should be 1.82 I think | 21:46 |
| iurygregory | yeah | 21:46 |
| iurygregory | https://github.com/openstack/ironic/commit/8e34d622aff72d7dd286add31e3d7cd366629bc2#diff-5e4dd7b8baeac0d222e2f6d734549d37b1487f67f5f1dd339101de3df3d5ebbb | 21:46 |
| JayF | yeah it's 82 | 21:46 |
| JayF | we still are missing some mappings | 21:46 |
| JayF | and that should be fixed | 21:46 |
| JayF | but is a different break | 21:46 |
| iurygregory | correct | 21:46 |
| TheJulia | sign, https://github.com/openstack/ironic/blob/master/ironic/common/release_mappings.py#L657 needs to also have 2023.2 | 21:46 |
| JayF | yeah, that's what we were talking about | 21:46 |
| JayF | 2023.2 is missing from that file, we have releases missing | 21:46 |
| JayF | going to fix my tempest change then push a fix for that | 21:47 |
| TheJulia | yeah, but that should be entirely disjointed from api microversion behavior | 21:47 |
| TheJulia | That is upgrade aliasing, really | 21:47 |
| TheJulia | A version released only knows master, that is to resolve the list in case of human | 21:47 |
| JayF | yeah the microversion stuff is 100% I have 1.89 in my tempest, I need 1.82 | 21:47 |
| JayF | this is another related issue found while hunting for the root cause here | 21:48 |
| opendevreview | Jay Faulkner proposed openstack/ironic-tempest-plugin master: Basic API tests for sharding https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/906749 | 21:48 |
| TheJulia | oh, you need api_microversion as well as min_microversion, I think | 21:54 |
| JayF | that is not true if other examples are to be believed | 21:54 |
| JayF | it looked like it's doing the right thing, just was requesting a too-new version in 2023.2 case | 21:54 |
| JayF | because I told it a too-new version | 21:55 |
| JayF | so even if that's not right I wanna see the output and understand where the other would come into play | 21:55 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Fix release mappings for 2023.2 https://review.opendev.org/c/openstack/ironic/+/908530 | 21:55 |
| JayF | I made ^ that by looking at a diff of object and api directories between 22.1 and 23.0 | 21:56 |
| TheJulia | https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_basic_ops.py most of these use api_microversion, not min_microversion, fwiw | 21:56 |
| JayF | so hopefully it should be right | 21:56 |
| JayF | scenario != api tests | 21:56 |
| JayF | scenario tests have a different method for skipping afaict | 21:56 |
| TheJulia | ahh | 21:58 |
| TheJulia | looks like the mapping on rpc is wrong | 21:58 |
| opendevreview | Jay Faulkner proposed openstack/ironic-tempest-plugin master: Basic API tests for sharding https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/906749 | 21:58 |
| TheJulia | the branch has 1.58 | 21:58 |
| JayF | how does the branch get a newer RPC version than master? | 21:59 |
| JayF | oh, I get what you're saying | 21:59 |
| JayF | I read that as "release mappings in branch has..." and you meant "actual rpc version on branch is..." | 22:00 |
| TheJulia | yes | 22:00 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Fix release mappings for 2023.2 https://review.opendev.org/c/openstack/ironic/+/908530 | 22:00 |
| opendevreview | Jay Faulkner proposed openstack/ironic-tempest-plugin master: Basic API tests for sharding https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/906749 | 22:48 |
| NobodyCam | good afternoon Ironic folks | 22:56 |
| NobodyCam | is there a reason we block setting owner with node create? | 22:56 |
| NobodyCam | `openstack baremetal node create --driver ipmi --owner None --name cjk-test-deleteMe` | 22:57 |
| NobodyCam | `The attribute(s) "owner" are invalid; they are not needed to create nodes.` | 22:57 |
| JayF | What version? | 23:00 |
| JayF | for both cli and api | 23:00 |
| NobodyCam | antelope | 23:00 |
| ashinclouds[m] | Use a system scoped account | 23:00 |
| NobodyCam | and I'll bet I'm really old on cli... /me checks | 23:00 |
| JayF | I would use latest cli for sure | 23:01 |
| JayF | even on older ironic | 23:01 |
| NobodyCam | ```ckrelle$ openstack --version | 23:01 |
| NobodyCam | openstack 5.5.0``` | 23:01 |
| NobodyCam | latest broke our ussuri regions :( | 23:01 |
| NobodyCam | ...broken in our... | 23:02 |
| TheJulia | What version of python-ironiccclient | 23:02 |
| TheJulia | NobodyCam: are you attempting to create a node with None as an owner to avoid the auto-map to your user's project ? | 23:09 |
| TheJulia | Assuming your using a project admin and not a system scoped admin | 23:09 |
| TheJulia | oh wow, i think i found it | 23:20 |
| TheJulia | yup, we have an explicit list | 23:21 |
| TheJulia | and the way osc stuff is tested, we never hit it | 23:22 |
| NobodyCam | https://www.irccloud.com/pastebin/iZ2xDBPu/ | 23:24 |
| JayF | TheJulia: what's the issue? | 23:24 |
| NobodyCam | and yep having user add nodes for a different project | 23:25 |
| NobodyCam | yes they are admin | 23:25 |
| TheJulia | rofl, one moment! | 23:26 |
| TheJulia | patch inbound | 23:26 |
| opendevreview | Julia Kreger proposed openstack/python-ironicclient master: Fix on-creation ability https://review.opendev.org/c/openstack/python-ironicclient/+/908534 | 23:29 |
| JayF | TheJulia: Should we filter creation fields at client level at all? | 23:29 |
| JayF | TheJulia: is there a reason we shouldn't just let the API complain? | 23:30 |
| JayF | I have some valid answers to this question; I'm curious which one we're going with :D | 23:30 |
| TheJulia | I dunno.... I just found it and kept laughing | 23:30 |
| JayF | I am going to +2 this | 23:30 |
| JayF | but I think I'd also be +2 to something that just ... didn't check that list | 23:30 |
| TheJulia | the last four + firmware_interface were missing | 23:31 |
| JayF | everyone of of those represents a massive failure in software development, review, and processes. | 23:31 |
| JayF | except for the shard one, that one is okay | 23:31 |
| JayF | :P | 23:31 |
| JayF | (that is entirely a joke to be clear) | 23:31 |
| TheJulia | *we* do validation on the api side, and we even have some testing on that, so I don't see why the API shouldn't be the one | 23:31 |
| TheJulia | I just dunno why every single creation has extra logic wrapped around it in the OSC commands | 23:32 |
| TheJulia | maybe it is some convention we inherented that we were sort of blind to unless we tried on create manually | 23:32 |
| TheJulia | the frustrating thing is how the test style works, we never actually hit where the list is validated | 23:32 |
| NobodyCam | Thank you! :bow | 23:33 |
| TheJulia | NobodyCam: by all means pull it down and give it a spin | 23:34 |
| TheJulia | I don't have a running deploy locally, but that is *exactly* where the error is coming from | 23:34 |
| NobodyCam | +++ | 23:34 |
| TheJulia | JayF: funny thing is, you suggested earlier in the week, we might want to set some owners on node create | 23:34 |
| TheJulia | NobodyCam: but seriously, why are you setting it to None?! | 23:34 |
| * JayF all the sudden wonders what kinda havoc a project named "None" could cause in OpenStack | 23:35 | |
| TheJulia | Muahahahahahaha | 23:35 |
| NobodyCam | (matching Ussuri behavior) until we can get things updated else where | 23:35 |
| NobodyCam | LoL | 23:36 |
| TheJulia | so, did you by chance read the troubleshooting.rst file I posted? | 23:36 |
| NobodyCam | :( no | 23:36 |
| * NobodyCam looks | 23:36 | |
| TheJulia | one moment | 23:36 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/908203 | 23:37 |
| TheJulia | so, yes, it is auto-magically record your creating project ID most likely | 23:37 |
| Kangie | dtantsur: I've looked at testenv - does it "just work" in a 'separate VMs' configuration (i.e. I'm deploying directly into a QEMU/KVM VM on infrastructure that I'd prefer to use for the task, not onto the bare metal running the hypervisor)? The docs don't really say but I gather maybe nested virtualisation is how it's supposed to work? I don't mind learning some "fun" extra | 23:37 |
| Kangie | config bits if configuring a few VMs attched to a virtual provisioning network with DHCP is straightforward and closer to what I'll need to do in a real env | 23:37 |
| JayF | > AssertionError: Versions (23, 0) and (2023, 2) are not sequential | 23:37 |
| TheJulia | Setting it to a string None is entirely different, if you enforce only new policies, it will disappear | 23:37 |
| JayF | *sigh* | 23:37 |
| NobodyCam | anything that has `Don't Panic` in it can't be bad | 23:37 |
| JayF | Kangie: you know ansible? | 23:38 |
| JayF | Kangie: it's all ansible. | 23:38 |
| Kangie | I will by the time I'm done with this... | 23:38 |
| JayF | heh, good answer | 23:38 |
| Kangie | I've already reverse engineered a bit to get around restrictive firewalls | 23:38 |
| TheJulia | JayF: the assertionerror makes me want to make a margharita and exercise | 23:39 |
| JayF | Yeah, basically we use a tool called sushy-tools or VirtualBMC to emulate BMCs | 23:39 |
| JayF | redfish and IPMI respectively | 23:39 |
| Kangie | I'm leaning towards sushy and targeting redfish - all our hardware should support it. I hope :D | 23:39 |
| JayF | yeah, so if you're looking for a testenv like that, you'd deploy sushy-tools somewhere (probably network-location close to the hypervisors if it were me) | 23:39 |
| TheJulia | NobodyCam: anyway, if your not enforcing the new policy, then the field shouldn't *really* have any impact | 23:40 |
| JayF | and then use the existing testenv setup as a template to create "fake nodes" for sushy-tools to talk with and emulated redfish BMCs for Ironic to talk to mapped to them | 23:40 |
| JayF | Kangie: I will close this by saying: we tell people not to run sushy-tools in production. If you let sushy-tools mess around with hypervisors in your prod environment, and it breaks a thing, we'll :( and have empathy, but you should be VERY CERTAIN you're comfortable with the changes it'll do before colocating it on a hypervisor with anything | 23:41 |
| NobodyCam | TheJulia: yea I used the set command to fix a couple node, then checked node create help and saw owner so told the enroll folks to use owner in there create and it ran into the issue you have provided a patch for | 23:41 |
| TheJulia | NobodyCam: ahh! | 23:41 |
| NobodyCam | it a project visibility for scheduling services above us | 23:41 |
| TheJulia | Ahh | 23:42 |
| TheJulia | so hmm | 23:42 |
| NobodyCam | in our newly deployed antelope region | 23:42 |
| TheJulia | NobodyCam: so for your RBAC context, let me get some code for you to be aware of | 23:42 |
| TheJulia | ohhh ahhhhhh | 23:42 |
| NobodyCam | we hit the glance rbac restriction | 23:43 |
| TheJulia | NobodyCam: https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/node.py#L2812-L2831 | 23:43 |
| TheJulia | owned_node in that context is if your allowed to create a node | 23:43 |
| TheJulia | depending on your workflow, you might need a custom policy as it applies to that entire path | 23:44 |
| TheJulia | Or, you just create in one project and move depending on your workflow | 23:45 |
| NobodyCam | +++ | 23:45 |
| NobodyCam | Thank you | 23:46 |
| Kangie | Right. Thanks JayF. Another reminder that it would be ideal to have actual development environment hardware. | 23:47 |
| Kangie | I guess the alternative is nested virtualisation. | 23:47 |
| JayF | That is what 99.999% of us do for our dev environments | 23:48 |
| JayF | it's not like you put actual workloads in the machines you boot | 23:48 |
| JayF | by default, we use an OS called "cirros" which is a linux distro basically designed for what we use it for (yep, it booted, done) | 23:48 |
| TheJulia | (I'll note, the power bills for rackmount servers can also be cost prohibitive....) | 23:51 |
| TheJulia | Says the crazy lady with like 16kW of solar | 23:51 |
| NobodyCam | :p | 23:57 |
| NobodyCam | Bitcode mining at TheJulia house | 23:57 |
| NobodyCam | hehehehehe | 23:57 |
| NobodyCam | s/bitcode/bitcoin/ | 23:57 |
| Kangie | TheJulia: I'm in the process of building node management for a new HPC cluster... | 23:58 |
| Kangie | (and retrofitting it to part of our existing cluster) | 23:59 |
| Kangie | I also have a 42U rack in my garage :D | 23:59 |
| JayF | Kangie is another person I dragged over here from the gentoo community with promises of "open source clouds" and "hardware provisioning" | 23:59 |
| TheJulia | Kangie: awesome! | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!