| opendevreview | Taketani Ryo proposed openstack/ironic master: Add a way to set up a tftp without xinetd https://review.opendev.org/c/openstack/ironic/+/892606 | 04:26 |
|---|---|---|
| rpittau | good morning ironic! Happy Friday! o/ | 06:07 |
| arne_wiebalck | Good mrorning rpittau and Ironic! | 06:54 |
| rpittau | hey arne_wiebalck :) | 06:55 |
| rpittau | JayF: I think we need a launchpad project space for ipa-builder :) | 07:24 |
| espenfl | Hi there. I have previously been deploying bare metal nodes nicely with an inventory file using Bifrost. Had a pause of a few months and got back to it now and updated the Bifrost setup to the most recent. In doing so, I get for instance `Unsupported parameters for (openstack.cloud.baremetal_node) module: properties.vendor` when enrolling using the previously working | 08:16 |
| espenfl | inventory. These are for supermicro nodes, so was using `properties.vendor`, which is not passing in `openstack.cloud.baremetal_node`. Upon locking at the Ironic code, it seems it is still supported there. Is this something that needs to be added to `openstack.cloud.baremetal_node` or am I doing something that is no longer supported? Thanks in advance for any comment or | 08:16 |
| espenfl | help. | 08:16 |
| dtantsur | espenfl: hi, I suspect the ansible module has an allowlist of properties. Do you actually need the vendor there? | 08:20 |
| espenfl | I think, that for supermicro, the only thing this does is to sort the difference of the bootdev so that it is set to disk when you set this. We only set pxe and do not change anything else, so in practice I do not really need it I think. Just noticed. Happy to open a PR for it and adding it to the allowlist. Ironic supports it: | 08:23 |
| espenfl | https://docs.openstack.org/ironic/latest/admin/drivers/ipmitool.html, but maybe we do not need it Bifrost side for now? | 08:23 |
| dtantsur | espenfl: rather in the ansible module: https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/plugins/modules/baremetal_node.py#L511 | 08:24 |
| espenfl | dtantsur: Yes. Indeed. I will open a PR towards that then. | 08:25 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 09:35 |
| rpittau | iurygregory: I did some tests with abort inspection with inspector itself and I don't think it works as we expect it works :) | 10:06 |
| rpittau | it looks like the abort command goes through also when the inspection is not running, which can be dangerous as it will try to force the power off of the node | 10:06 |
| rpittau | I don't see changes that impacted that behavior in inspector, but there's a lot between wallaby and now, so it's a mistery how the functional test was working before but no anymore in the inspector client CI | 10:06 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 10:35 |
| iurygregory | rpittau, yeah agree | 10:57 |
| iurygregory | good morning Ironic | 10:57 |
| espenfl | Hi there. Upon enrolling a node with Bifrost (using the bifrost-cli) I get: `"BadRequestException: 400: Client Error for url: ********/nodes, Expected UUID for uuid: "`. I know this specific inventory deployed a few months ago, so what have changed is a more update Bifrsot and Ironic installation. A bit unsure where to start with this. Seems an uuid is not supplied. | 11:24 |
| iurygregory | normally you pass a json file to enroll nodes, maybe check what is there? | 11:25 |
| iurygregory | rpittau, how did you manage to reproduce consistently the inspector client issue? | 11:29 |
| espenfl | iurygregory: Thanks for the reply. Yes, using the same json as before (March I believe). Bifrost should generate an uuid if not given I think. | 11:35 |
| iurygregory | let me check in my env | 11:35 |
| iurygregory | which command did you use? | 11:36 |
| iurygregory | I have a env up I can try to reproduce your issue here | 11:36 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Add inspection (processing) hooks https://review.opendev.org/c/openstack/ironic/+/887554 | 11:39 |
| iurygregory | espenfl, the command you are using is $./bifrost-cli enroll /tmp/baremetal.json right? | 11:43 |
| espenfl | iurygregory: correct | 11:44 |
| iurygregory | yeah I'm getting the same error | 11:47 |
| espenfl | iurygregory: Okey, that is good I guess. I was concerned something was messed up on my side. | 11:55 |
| iurygregory | yup | 11:56 |
| espenfl | It seems this has to be happening in `ansible-collections.openstack.cloud.baremetal_node` and this is maybe some newer addition in the context of Bifrost? Hence, why this was not happening before summer. The uuid should be generated there if not provided it seems. | 12:00 |
| iurygregory | we have been using the collections from some time I think | 12:02 |
| iurygregory | maybe a new release of the collection is requesting this.. | 12:02 |
| rpittau | iurygregory: I just run the functional tests and I got the error | 12:05 |
| iurygregory | with the changes I have in the patch you commented right? | 12:06 |
| rpittau | yep | 12:06 |
| iurygregory | ok, there is probably something crazy in my env :D | 12:06 |
| iurygregory | lol | 12:06 |
| iurygregory | rm -rf now :D | 12:06 |
| rpittau | lol | 12:06 |
| iurygregory | I was chasing the code but seems like we've always returned 202 .-. | 12:07 |
| iurygregory | even our docs doesn't mention error for abort in case inspection hasn't started | 12:07 |
| rpittau | yep | 12:08 |
| iurygregory | https://docs.openstack.org/api-ref/baremetal-introspection/?expanded=abort-introspection-detail | 12:08 |
| rpittau | that's why I have no idea how the test was passing before | 12:08 |
| iurygregory | <insert it's fine gif here> | 12:08 |
| iurygregory | yup! | 12:08 |
| iurygregory | same | 12:08 |
| rpittau | maybe the python version changed something ? | 12:08 |
| rpittau | ¯\_(ツ)_/¯ | 12:08 |
| iurygregory | who knows right? :D | 12:09 |
| iurygregory | we have to play with a few things | 12:09 |
| iurygregory | and good catch with the changes in the tox.ini | 12:09 |
| rpittau | probably need more eyes on it, it's easy to overlook things with so many changes | 12:10 |
| iurygregory | ++ | 12:13 |
| espenfl | Okey, so it seems `openstacksdk` sends an empty `uuid` and `id`. But one is expected. Not sure if `openstacksdk` is supposed to create that id or it is supposed to happen in `ansible-collections`, but I will check a bit more. | 12:32 |
| opendevreview | Pierre Riteau proposed openstack/ironic-inspector master: docs: fix typo https://review.opendev.org/c/openstack/ironic-inspector/+/892785 | 12:46 |
| *** TheJulia is now known as needs_brains_and_sleep | 13:04 | |
| needs_brains_and_sleep | brraaaaaaains! | 13:04 |
| opendevreview | Merged openstack/ironic-inspector master: docs: fix typo https://review.opendev.org/c/openstack/ironic-inspector/+/892785 | 13:13 |
| *** needs_brains_and_sleep is now known as TheJulia | 13:23 | |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Update iRMC driver vendor library version https://review.opendev.org/c/openstack/ironic/+/892790 | 13:55 |
| TheJulia | so it *looks* like the ovn job actually worked | 14:33 |
| TheJulia | well, 2 out of 3 | 14:33 |
| TheJulia | it seems ovn + rescue might be problematic | 14:33 |
| JayF | \o/ | 14:39 |
| TheJulia | seems... snat gets turned back on somehow | 14:39 |
| TheJulia | and then back off | 14:39 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Move metal3 integration job to non-voting temporarily https://review.opendev.org/c/openstack/ironic/+/892794 | 14:47 |
| rpittau | JayF TheJulia dtantsur ^ | 14:47 |
| rpittau | I raised the issue in the metal3 community channel | 14:47 |
| JayF | +2A, ty rpittau | 14:50 |
| * JayF gone for a couple hours | 14:50 | |
| dtantsur | Maybe having it voting was too ambitious.. | 14:57 |
| rpittau | nah, it worked fine so far, and we actually catch multiple issues | 14:58 |
| rpittau | as other integration jobs we need to keep an eye on it | 14:58 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Revert "Move metal3 integration job to non-voting temporarily" https://review.opendev.org/c/openstack/ironic/+/892797 | 15:12 |
| rpittau | bye everyone, have a great weekend! o/ | 15:45 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Exclude .pyc encoding files. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 16:10 |
| dtantsur | Have a great weekend folks o/ I'm leaving for 1.5 weeks, see you then. The IRC bouncer will be off at some point, so email me if anything is needed. | 16:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 16:22 |
| TheJulia | o/ dtantsur | 16:22 |
| TheJulia | so, folks, anyone have opinions on finally creating an neutron aware tftp service? | 16:22 |
| JayF | sounds like a good topic for ptg | 16:25 |
| TheJulia | we've discussed it before, but OVN basically shooting tftp in the foot, we don't have many options :\ | 16:30 |
| JayF | honestly, I'd suggest, instead of making TFTP a separate service, maybe just adding it to their existing DHCP agent/interface? | 16:36 |
| JayF | since dnsmasq you get tftp for "free"? | 16:37 |
| TheJulia | well, we would need to make a service to do that | 16:37 |
| TheJulia | and in OVN, there is no more dnsmasq | 16:37 |
| TheJulia | ovn itself is the dhcp server | 16:37 |
| JayF | does ovn provide tftp? | 16:38 |
| TheJulia | nope | 16:38 |
| JayF | let me ask the question this way | 16:38 |
| JayF | instead of saying "neutron aware tftp service" | 16:38 |
| JayF | what's the actual feature we need? | 16:38 |
| JayF | like what is going to change about interactions to make things like OVN behave better | 16:39 |
| TheJulia | we need to attach a tftp instance onto the actual subnet in use by a bare metal instance | 16:39 |
| TheJulia | and use that to service requests | 16:39 |
| JayF | because right now tftp crosses a network segment in ~most configurations | 16:40 |
| TheJulia | yes | 16:40 |
| TheJulia | and if there is any NAT in that mix, OVN can't forward tftp | 16:40 |
| JayF | would such an implementation be limited to "providing iPXE roms and configs" ? | 16:40 |
| TheJulia | you can't rom UEFI | 16:40 |
| TheJulia | afaik | 16:40 |
| JayF | TheJulia: I wonder if making OVN know how to behave properly with segmentation and MTUs is the better path | 16:40 |
| TheJulia | well, you can load an image, but we have no way to tie that to the card | 16:40 |
| TheJulia | so we loose netboot control | 16:40 |
| TheJulia | JayF: maybe, maybe not, dunno | 16:41 |
| JayF | did you mean "from uefi" or "rom uefi" | 16:41 |
| JayF | I lost the thread at that point | 16:41 |
| TheJulia | so you can make a EFI firmware image that you can load onto a system, it can persist there, but it is not like "i want to boot from the network card oprom" | 16:42 |
| TheJulia | which is what ipxe roms are | 16:42 |
| TheJulia | and there is no way to tie the two together remotely | 16:42 |
| JayF | I am confused because right now, I have local UEFI-based netbooting setup that feeds an modified ipxe rom (netboot.xyz) to efi hosts | 16:42 |
| TheJulia | at best, it might be a entry in the boot order list, but may not be modifiable | 16:42 |
| JayF | I think we are still not talkign about the same thing? | 16:43 |
| TheJulia | another option is HTTPBoot, but that leaves everything that can't do it out | 16:43 |
| TheJulia | and OVN doesn't support such in it's dhcp server yet | 16:43 |
| TheJulia | I'm not sure we're meaning the same thing when we say rom | 16:44 |
| JayF | I'm talking about e.g. | 16:45 |
| JayF | > -rw-r--r-- 1 root wheel 1027072 Jul 2 04:12 ipxe.efi | 16:45 |
| TheJulia | yes, and how do you tell the system to boot that file | 16:46 |
| TheJulia | on a plus side https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py doesn't seem too insane | 16:46 |
| JayF | next-server and filename? | 16:46 |
| TheJulia | can't use next-server | 16:46 |
| TheJulia | that is inherently tftp | 16:46 |
| JayF | okay you are confusing my question then | 16:46 |
| JayF | I was saying, if such a theoretical TFTP neutron interface exists | 16:46 |
| TheJulia | oh | 16:47 |
| JayF | what is the api to it? | 16:47 |
| JayF | Does it always serve up ipxe roms and you just give an ipxe config? | 16:47 |
| JayF | I'm just thinking about how complex dhcp configs get in a hurry; ifs based on boot mode, based on ipxe vs network rom, boot types supported, etc | 16:47 |
| TheJulia | if an interface existed, we would have an agent, say running as 10.1.0.253 or $mystical_magical_metadata IP, which could service packets/connections from that logical network | 16:48 |
| TheJulia | we would use that as the configured server IP instead of the ironic default | 16:48 |
| JayF | so perhaps something like "we'll get this ipxe config booted on the server" would be a semi-reasonable interface, the neutron implementation could just own getting ipxe booted on the node, once ipxe comes up it feeds the provided user config in | 16:48 |
| TheJulia | the client would then be able to retrieve artifacts from it | 16:48 |
| TheJulia | it wouldn't be neutron most likely, they would force it to be out of their tree | 16:49 |
| JayF | This is kinda why I want to mix the dhcp and the tftp magic though | 16:49 |
| JayF | because I think depending on dhcp config it may change how you wanna serve up netbooting | 16:49 |
| JayF | I guess I'd rather think of it as "neutron pxe booting interface" than "neutron tftp" because it gets more interesting | 16:49 |
| TheJulia | so the challenge there is, we would have to tell "oh, here are the artifacts you'll use to setup the tftp server" | 16:49 |
| JayF | I'm saying don't support that. | 16:50 |
| TheJulia | or "here is the proxy to, you'll need to proxy everything via http" | 16:50 |
| JayF | My suggestion would be a payload like: {mac: 000000000000, ipxe_conf: '#!ipxe\n\nyour_conf_here'} | 16:50 |
| JayF | then the neutron service would own everything about getting ipxe booted with that config on the node | 16:51 |
| JayF | leaving the implementation detail of *how* to boot that node to the enhanced dhcp+netboot interface | 16:51 |
| TheJulia | I guess I'm hesitant to try and put anything on to the ovn core's plate because it has taken us 3 years to get dhcp support | 16:51 |
| TheJulia | and even then,, it is not just ipxe config, it is any part of the existing pxe infrastructure which the conductor may have on the filesystem | 16:52 |
| TheJulia | from templates to files because we need to bootstrap the bootloader | 16:52 |
| JayF | do we still support non-localboot post-provisioning? | 16:52 |
| TheJulia | For example, PXE when using grub, we need to chain through two files before we can even go hunting for config | 16:52 |
| JayF | I thought tftp was 100% limited to getting a pxe config (+ramdisk+kernel if you're using tftp for that, but IRL that is not very reliable) | 16:53 |
| JayF | ugh | 16:53 |
| TheJulia | technically, it has been kept in our back pocket | 16:53 |
| JayF | why do we have to support grub+pxe | 16:53 |
| TheJulia | because tons of people just don't want ipxe | 16:53 |
| JayF | get them all in one place and I can tell them they're wrong /s | 16:53 |
| TheJulia | (and grub is actually not half bad, just doesn't have any ability to fallback like ipxe does) | 16:53 |
| TheJulia | also, Secure Boot | 16:53 |
| JayF | with our grub pxe support, it never uses ipxe at all? | 16:53 |
| JayF | that's right, secure boot is the real answer there isn't it | 16:54 |
| TheJulia | it never uses ipxe | 16:54 |
| JayF | stevebaker[m] gave a good talk on that at the summit | 16:54 |
| JayF | I wish the reality of running hardware didn't require Ironic to be so unopinionated as software :( | 16:54 |
| TheJulia | humans generally default to an opinionated mode, generally unless you apply lots of this dried green stuff | 16:55 |
| opendevreview | Merged openstack/ironic master: Move metal3 integration job to non-voting temporarily https://review.opendev.org/c/openstack/ironic/+/892794 | 18:20 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Enable OVN CI https://review.opendev.org/c/openstack/ironic/+/885087 | 20:04 |
| JayF | TheJulia: you wanna help and land https://review.opendev.org/c/openstack/ironic/+/891912 and then we can add a "with OVN" configuration to the devstack-guide? | 20:05 |
| JayF | :D | 20:05 |
| JayF | ah, I see we need a DS patch | 20:06 |
| TheJulia | unfortunately, yes | 20:06 |
| TheJulia | we have to start some stuff up earlier than it likes to | 20:07 |
| TheJulia | JayF: do you have the ptg etherpad link handy? I can't seem to find it | 21:17 |
| JayF | it's in the meeting agenda | 21:17 |
| JayF | https://etherpad.opendev.org/p/ironic-ptg-october-2023 | 21:18 |
| TheJulia | thanks! | 21:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!