| adam_metal3 | Hi Ironic | 05:18 |
|---|---|---|
| rpittau | good morning ironic! o/ | 07:33 |
| adam_metal3 | rpittau, good morning | 07:37 |
| rpittau | hey adam_metal3 :) | 07:37 |
| dtantsur | TheJulia: hi, since you probably have opinions about inspector, could you check https://review.opendev.org/c/openstack/ironic-specs/+/878001 please? Has 2x +2 already, really close to landing. | 07:56 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Move ubuntu jobs to jammy https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/879538 | 09:40 |
| hjensas | dtantsur: I made a comment regarding DHCP filter, even with direct DB access is there not a risk that the frequency we would need to read from the DB would become an issue in large deployments? | 09:53 |
| dtantsur | hjensas: these are valid concerns, I'm just not sure I have better ideas.. | 10:45 |
| dtantsur | we could, of course, make the new script/service expose JSON RPC | 10:46 |
| dtantsur | ... which will be pretty close to having our own DHCP agent :D | 10:50 |
| opendevreview | Merged openstack/ironic stable/zed: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879470 | 10:53 |
| adam_metal3 | Sandzwerg[m], hi, would you be available to send the network_data.json with the working bond config ? | 11:01 |
| hjensas | yeah, RPC or a rest api that ironic could post events to. | 11:02 |
| hjensas | I had look at multilevel dhcp providers idea I mentioned on one of the reviews, it get's complicated because there is code to get IP address via the DHCP provider. AFICT getting the IP is only required for ELILO uefi boot, which is deprecated ... makes me want to rip code out. | 11:04 |
| dtantsur | hjensas: I think elilo support is gone already | 11:07 |
| hjensas | yes, elilo support is gone. There is a reno, but the code added in dhcp provider is still there - and we seem to link grub2 pxe configs to under both IP address and MAC address. (But grub2 should be happy with just the MAC ...) | 11:09 |
| hjensas | dtantsur: I'm thinking instead of our own DHCP agent over dnsmasq, it would be better to look at kea-dhcp (or other options). We could even opt not to use the neutron DHCP provider, just configure kea-dhcp with static IPs from neutron vifs. | 11:11 |
| hjensas | dtantsur: the kea-dhcp option also has the benefit of HA/load-balancing capabilities. What is Metal3's thoughts on HA/load-balancing DHCP, not a requirement or something you would like to see at some point? | 11:16 |
| dtantsur | hjensas: we haven't thought about that (which does not mean a hard no) | 11:17 |
| dtantsur | we need to solve the ironic HA/load-balancing problem first :) | 11:17 |
| dtantsur | and that involves 1) doing something about inspector, 2) doing something about cross-conductor PXE configuration | 11:18 |
| hjensas | ack, oh another question - discovery is a usecase you want to support in Metal3? | 11:32 |
| iurygregory | morning ironic | 11:44 |
| Sandzwerg[m] | adam_metal3: does that help? https://etherpad.opendev.org/p/gtFooG_MOO9tB79DUb6H | 11:50 |
| dtantsur | hjensas: possibly. not on the roadmap now, but there have been requests upstream | 11:52 |
| adam_metal3 | Sandzwer[m], thanks I will check!!! | 11:52 |
| iurygregory | I'm still puzzled that the grenade job didn't failed on my patch | 12:38 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 12:43 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/zed: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879550 | 12:48 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/yoga: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879551 | 12:49 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/xena: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879552 | 12:49 |
| iurygregory | backport of features? .-. | 12:50 |
| dtantsur | especially to xena.. | 13:02 |
| TheJulia | … on video at | 13:03 |
| TheJulia | err | 13:05 |
| TheJulia | silly autocorrect | 13:05 |
| TheJulia | i typed bifrost, hit enter, it changed it | 13:05 |
| * TheJulia goes and makes coffee | 13:05 | |
| opendevreview | Merged openstack/ironic master: Fix requests calls with timeouts https://review.opendev.org/c/openstack/ironic/+/879497 | 13:13 |
| TheJulia | good morning everyone! | 13:15 |
| TheJulia | hjensas: dtantsur: the mac address support was added to be able to eventually remove the by ip behavior which is now the far flung fallback of grub2 after machine hardware uuid (the internal one) and the interface mac address | 13:19 |
| dtantsur | morning TheJulia, ack | 13:19 |
| hjensas | TheJulia: good morning! yes, that is my conclusion on reading grub2 docs as well. | 13:21 |
| TheJulia | ... at one point it didn't support the mac address which seems weird | 13:21 |
| TheJulia | but they added that in | 13:21 |
| TheJulia | (Thank you universe!) | 13:21 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 13:50 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/xena: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879552 | 13:55 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/yoga: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879551 | 13:57 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/zed: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879550 | 13:59 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/xena: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879552 | 14:00 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/yoga: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879551 | 14:00 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost stable/zed: Feature: Enable user_data's file generation https://review.opendev.org/c/openstack/bifrost/+/879550 | 14:01 |
| mnasiadka | Good afternoon Ironic | 14:04 |
| mnasiadka | Have sort of an issue with idrac-redfish node enrollment | 14:04 |
| mnasiadka | ERROR ironic.conductor.verify [req-bd409ce5-20f1-44f5-928b-edaf69fef7a9 bifrost_user - - - -] Failed to get power state for node 55ed12a1-c878-4d33-9653-c751d8309baf. Error: Redfish exception occurred. Error: 'NoneType' object has no attribute 'encode': ironic.common.exception.RedfishError: Redfish exception occurred. Error: 'NoneType' object has no attribute 'encode' | 14:05 |
| mnasiadka | Tried searching for storyboard bugs, but it seems I'm not really good with it ;-) | 14:05 |
| iurygregory | first time I see this one | 14:05 |
| mnasiadka | hmm, let me check, I think I haven't set redfish_user and redfish_password, just blindly set drac_user and drac_password (but docs say it's only for wsman) | 14:09 |
| *** dking is now known as Guest10062 | 14:19 | |
| mnasiadka | ok, that fixed it | 14:31 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 14:36 |
| dtantsur | I actually wonder at which point we can deprecate WSMAN support given that Redfish is the bright future. | 14:47 |
| dtantsur | wdyt TheJulia? | 14:47 |
| TheJulia | huh whaaaattt | 14:54 |
| * TheJulia blinks because her brain has been deep in ironic-grenade | 14:54 | |
| TheJulia | dtantsur: .... maybe? a slow process for sure since even fairly recently people expressed trying to use ?idrac6? in production | 14:55 |
| TheJulia | or maybe it was idrac7 | 14:56 |
| dtantsur | le sigh | 14:58 |
| opendevreview | Jay Faulkner proposed openstack/ironic-specs master: Add 2023.2 Workitems discussed at Ironic PTG https://review.opendev.org/c/openstack/ironic-specs/+/879381 | 15:16 |
| rpittau | good night! o/ | 15:59 |
| dtantsur | TheJulia, hjensas, a pretty raw thought dump on PXE/DHCP and x-conductor: https://etherpad.opendev.org/p/ironic-multiconductor-static-dhcp | 16:00 |
| dtantsur | wanted to get it out of my head before I leave for the holidays | 16:00 |
| TheJulia | my gut feeling is we've got a race condition which we're kind of skirting around with recovery on failure | 16:11 |
| TheJulia | but I think that would require a similar brain dump to handle | 16:11 |
| TheJulia | we could potentially do a dhcp service, but then we're re-inventing neutron :\ | 16:11 |
| TheJulia | and the idea I had was we would trigger the neutron update, cross-conductor wouldn't handle static dhcp updates... | 16:13 |
| TheJulia | hmm | 16:13 |
| dtantsur | Re-invent all your neutrons \o/ | 16:15 |
| dtantsur | I wish neutron was more (de-)composable | 16:16 |
| dtantsur | anyway, my brain is shrimp already, switching to the easter break mode. see you next Wednesday! | 16:16 |
| TheJulia | have a great break | 16:17 |
| TheJulia | I think I've identified what is going on with some of the grenade jobs | 16:17 |
| TheJulia | y'all wont like it | 16:17 |
| dtantsur | sounds promising... | 16:17 |
| TheJulia | tl;dr looks like the vm networking is coming online late | 16:17 |
| TheJulia | there could also be a dnsmasq crash red-herring involved | 16:18 |
| TheJulia | like splash screen not reached until 88 seconds after vm starts | 16:19 |
| TheJulia | network up 21 seconds in, network back up 83 seconds in | 16:19 |
| JayF | how does that present so weirdly w/r/t the flavor stuff? | 16:27 |
| TheJulia | so flavor errors are a result of our use and the case we were digging into yesterday remains a mystery | 16:28 |
| TheJulia | in this specific case, the vm fired up | 16:28 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 16:28 |
| TheJulia | everything worked until the fip couldn't be pinged | 16:28 |
| JayF | so this is the "vip won't ping" issue dtantsur was originally looking at | 16:28 |
| TheJulia | yup | 16:28 |
| TheJulia | I think we can extend the timeout for grenade and it should make that issue just go away | 16:29 |
| JayF | is the answer to sleep harder? serious question | 16:29 |
| JayF | +++++ | 16:29 |
| TheJulia | basically eys | 16:29 |
| TheJulia | yes | 16:29 |
| TheJulia | instead of sleep to cuddle cats for a moment, maybe take a quick cat nap | 16:29 |
| TheJulia | oh sigh | 16:33 |
| TheJulia | it is a static setting in grenade | 16:33 |
| TheJulia | https://github.com/openstack/grenade/blob/3f9fe2e8fc1fccf0324538274e3b07b3e90b96b9/projects/60_nova/resources.sh#L137 | 16:33 |
| JayF | only until we make it non-static at least | 16:34 |
| TheJulia | we upgdated our version of cirros last cycle didn't we | 16:34 |
| TheJulia | updated | 16:34 |
| JayF | IDR | 16:34 |
| TheJulia | neutron folks looked at some average jobs and indicated they see it take 35 seconds at present | 16:35 |
| TheJulia | so a bit longer is not... unreasonable | 16:35 |
| clarkb | I think the cirros update got reverted, but that may be worth double checking | 16:35 |
| TheJulia | ... this is also on rax right now | 16:36 |
| TheJulia | I wonder if we're seeing just rax hosts fail grenade | 16:36 |
| TheJulia | all emulated VM workloads | 16:37 |
| TheJulia | yeah, this is 5.2, so oldish | 16:38 |
| TheJulia | so inmotion and rax | 16:44 |
| TheJulia | and ovh | 16:45 |
| TheJulia | might just be luck of the draw as it were | 16:45 |
| clarkb | inmotion rax and ovh doesn't leave much else if that is where you see failures | 16:45 |
| TheJulia | yeah | 16:45 |
| TheJulia | time for a grenade change | 16:45 |
| clarkb | other than those three you've got vexxhost which only does special labels now and arm64 clouds | 16:45 |
| TheJulia | so interestingly enough, we likely need to check our vm networking settings again for spanning tree | 16:54 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Set ironic-grenade to wait 120 seconds https://review.opendev.org/c/openstack/ironic/+/879675 | 16:59 |
| TheJulia | okay, now we wait() | 16:59 |
| *** Peter is now known as Guest10078 | 17:33 | |
| opendevreview | Merged openstack/ironic-lib master: Exclude all files starting with . from flake8 tests https://review.opendev.org/c/openstack/ironic-lib/+/878402 | 17:37 |
| opendevreview | Merged openstack/ironic stable/2023.1: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879407 | 18:21 |
| opendevreview | Merged openstack/ironic bugfix/21.2: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879409 | 18:21 |
| opendevreview | Merged openstack/ironic bugfix/21.3: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879408 | 18:27 |
| opendevreview | Merged openstack/ironic master: Remove outdated API version information from the enrollment docs https://review.opendev.org/c/openstack/ironic/+/879428 | 18:27 |
| opendevreview | Merged openstack/ironic master: On rpc service stop, wait for node reservation release https://review.opendev.org/c/openstack/ironic/+/872834 | 18:48 |
| opendevreview | Verification of a change to openstack/ironic master failed: Always fall back from hard linking to copying files https://review.opendev.org/c/openstack/ironic/+/879161 | 18:48 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Add checksum generation support https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/878009 | 18:58 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Remove ussuri CI jobs https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/879357 | 19:12 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Always generate checksum files on image builds https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/878297 | 19:12 |
| opendevreview | Merged openstack/ironic master: Exclude all files starting with . from flake8 tests https://review.opendev.org/c/openstack/ironic/+/878788 | 19:20 |
| opendevreview | Merged openstack/ironic-python-agent master: Report system firmware information in the inventory https://review.opendev.org/c/openstack/ironic-python-agent/+/879049 | 19:37 |
| opendevreview | Verification of a change to openstack/ironic stable/wallaby failed: Suppress Bandit B509 in snmp.py https://review.opendev.org/c/openstack/ironic/+/860847 | 19:45 |
| opendevreview | Merged openstack/ironic master: Do not log into the template1 database during test-setup https://review.opendev.org/c/openstack/ironic/+/879148 | 20:17 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 21:26 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Add hold steps https://review.opendev.org/c/openstack/ironic/+/879060 | 21:59 |
| TheJulia | new state diagram on that change :) | 21:59 |
| * TheJulia calls it a day | 22:03 | |
| JayF | iurygregory: maybe we sync up earlier tomorrow? I'm having trouble getting even a basic testenv up for bifrost on stable/2023.1 | 22:20 |
| JayF | angry libvirt errors in `bifrost-cli testenv` | 22:20 |
| JayF | > fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "internal error: Network is already in use by interface enp1s0"} | 22:20 |
| iurygregory | JayF, sure, at what time are you thinking? | 22:24 |
| JayF | well, I just gotta get this actually working | 22:24 |
| iurygregory | humm networking already in use... | 22:24 |
| JayF | I looked at virsh net-list --all | 22:25 |
| JayF | only one there is default, autostart enabled | 22:25 |
| iurygregory | I seem to remember something like this | 22:25 |
| JayF | I disabled autostart, rebooted | 22:25 |
| JayF | same err | 22:25 |
| iurygregory | do you have any virbr0 available already? | 22:25 |
| JayF | (note: this is a VM I'm trying to install on) | 22:25 |
| JayF | only a single interface, it's the NAT network from the parent hypervisors' libvirt | 22:26 |
| iurygregory | gotcha, never tested this approach (but it's like our CI I would say) | 22:26 |
| JayF | yeah, my hunch is a package or some presetup is missing that's assumed | 22:26 |
| JayF | but missing in the testenv instructions | 22:27 |
| JayF | (or bitrotted over the life of ubu 22.04) | 22:27 |
| iurygregory | ./bifrost-cli testenv | 22:28 |
| iurygregory | ./bifrost-cli install --testenv | 22:28 |
| JayF | testenv will not succeed | 22:28 |
| JayF | that's what's erroring | 22:28 |
| iurygregory | oh wow | 22:28 |
| iurygregory | O.o | 22:28 |
| JayF | let me get you a bigger log with this reproduction | 22:28 |
| JayF | unless I fixed it :P | 22:28 |
| iurygregory | are you trying master? | 22:28 |
| iurygregory | never tested in ubuntu, but we do this in our CI... | 22:29 |
| JayF | https://gist.github.com/jayofdoom/98321f655f303a2127eb8587c7d2f4be | 22:29 |
| JayF | I'm on stable/2023.1 | 22:29 |
| JayF | I can try master if you think that's a better idea, but was hoping to demo with latest stable | 22:29 |
| iurygregory | I would use stable/x version | 22:30 |
| JayF | yeah, that's the bifrost I have checked out | 22:30 |
| JayF | stable/2023.1 | 22:30 |
| iurygregory | ensure libvirt network is running .-. | 22:30 |
| iurygregory | do you have qemu/kvm installed in the vm? | 22:30 |
| JayF | iurygregory: see the added comment on the gist | 22:31 |
| iurygregory | maybe you need to upgrade packages | 22:31 |
| * iurygregory press f5 | 22:31 | |
| JayF | running a dist-upgrade | 22:31 |
| JayF | I'll note it installed/uninstalled half the world the first testenv run | 22:32 |
| JayF | still a lot of updates | 22:32 |
| iurygregory | normally it's good to update packages before installing | 22:32 |
| iurygregory | I always do that | 22:32 |
| iurygregory | let me check the task in our code | 22:32 |
| JayF | I'm happy to change distro if you think it works better; but I believe we use ubu here so I'd prefer do that | 22:33 |
| JayF | dist-upgraded, restarted, trying again | 22:33 |
| JayF | same error :| | 22:35 |
| JayF | going to try master, then maybe zed | 22:35 |
| iurygregory | https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-create-vm-nodes/README.md | 22:38 |
| JayF | dtantsur: https://gist.github.com/jayofdoom/98321f655f303a2127eb8587c7d2f4be if you have any ideas on this, please lmk. I'm trying to get a bifrost running on ubuntu 22.04 | 22:38 |
| JayF | window bare | 22:38 |
| JayF | oops | 22:38 |
| JayF | looking at link | 22:38 |
| iurygregory | can you check if any the packages mentioned are installed? | 22:38 |
| JayF | so basically we need qemu-kvm but it's not installed by testenv | 22:38 |
| iurygregory | so we do have a bug | 22:38 |
| iurygregory | :D | 22:38 |
| iurygregory | because I think it should install | 22:38 |
| JayF | > Note, selecting 'qemu-system-x86' instead of 'qemu-kvm' | 22:39 |
| iurygregory | https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-create-vm-nodes/defaults/main.yml#L28 | 22:39 |
| JayF | which was already installed | 22:39 |
| JayF | bingo iurygregory | 22:39 |
| JayF | I'm on a default libvirt network | 22:39 |
| JayF | so it's conflicting with my local network | 22:39 |
| JayF | that's exactly the problem | 22:39 |
| JayF | I suspect it's easier to change it on the hyp and give me a clear demo lol | 22:40 |
| iurygregory | probably | 22:40 |
| iurygregory | https://docs.openstack.org/bifrost/latest/install/index.html#quick-start-with-bifrost-cli | 22:40 |
| iurygregory | maybe you can test with a specific network-interface and a dhcp-pool | 22:41 |
| JayF | I'm resetting my hyp to non-default settings | 22:42 |
| JayF | so I can have a less-messed-with VM to demo in | 22:42 |
| iurygregory | fingers crossed | 22:43 |
| JayF | I'm rebuilding the whole VM to fix the network | 22:44 |
| JayF | so you'll have to get the update in the morn I think :D | 22:44 |
| iurygregory | np o/ | 22:45 |
| JayF | whoever it was that told me that virt-manager can connect to an external hypervisor is a wonderful person :D | 22:48 |
| JayF | iurygregory: GREAT SUCCESS | 23:03 |
| JayF | dtantsur: ignore previous message | 23:03 |
| JayF | iurygregory: nice find, thank you | 23:03 |
| opendevreview | Jay Faulkner proposed openstack/bifrost master: Add warning about default libvirt networks https://review.opendev.org/c/openstack/bifrost/+/879711 | 23:06 |
| iurygregory | JayF, awesome! | 23:12 |
| iurygregory | +W since it was a doc change | 23:14 |
| iurygregory | just checked the build result | 23:14 |
| opendevreview | Merged openstack/bifrost master: Add warning about default libvirt networks https://review.opendev.org/c/openstack/bifrost/+/879711 | 23:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!