| opendevreview | Jacob Anders proposed openstack/sushy master: Skipping UsbCd workaround on Supermicro ARS-111GL-NHR https://review.opendev.org/c/openstack/sushy/+/949116 | 00:08 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 00:33 |
| opendevreview | Merged openstack/ironic master: ci: Remove code which has been long-dead https://review.opendev.org/c/openstack/ironic/+/950461 | 02:41 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 03:51 |
| rpittau | good morning ironic! o/ | 06:53 |
| rpittau | first run with the revert and metal3 job passed https://review.opendev.org/c/openstack/ironic/+/950540 | 07:30 |
| rpittau | I'll run it again a couple of times | 07:30 |
| rpittau | btw same issue happeninig in bifrost, CI is broken https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_076/openstack/07612d892843449d99c08ab636e1546a/logs/redfish-emulator.log | 07:50 |
| rpittau | loop loop loop https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_076/openstack/07612d892843449d99c08ab636e1546a/logs/ironic.log | 07:50 |
| rpittau | I'll try to test bifrost CI too with the revert | 07:51 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] test ci https://review.opendev.org/c/openstack/bifrost/+/950627 | 07:53 |
| dtantsur | nice catch | 08:40 |
| adam-metal3 | Hello Ironic | 08:40 |
| dtantsur | o/ | 08:40 |
| adam-metal3 | Could have been something merged between 7 and 2 days ago that causes the power state checks to go crazy? | 08:41 |
| adam-metal3 | like so: errorMessage: 'Failed to get power state for node d275e8be-ea6d-4ce5-bf8f-8736d99f778a. | 08:41 |
| adam-metal3 | Error: maximum recursion depth exceeded' | 08:41 |
| adam-metal3 | We have started using a new ironic image built with master Ironic source 2 days ago and all the test went mad with the same issue | 08:42 |
| adam_metal3 | I suspected this https://opendev.org/openstack/ironic/commit/4f22bade301d6957743e3913c221f3872e39fdfc but I did not investigate further | 08:45 |
| rpittau | adam_metal3: that's the issue I was talking before here :) | 08:51 |
| rpittau | adam_metal3: that's exactly my first suspect, we're tracking the issue here r https://bugs.launchpad.net/ironic/+bug/2070791 | 08:52 |
| rpittau | I also mentioned that in the ironic-image change https://github.com/metal3-io/ironic-image/pull/685 | 08:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [CI] metal3 job back to voting https://review.opendev.org/c/openstack/ironic/+/950333 | 08:55 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [CI] metal3 job back to voting https://review.opendev.org/c/openstack/ironic/+/950333 | 08:56 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Revert "Fix redfish driver URL parsing" https://review.opendev.org/c/openstack/ironic/+/950540 | 08:59 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [CI] metal3 job back to voting https://review.opendev.org/c/openstack/ironic/+/950333 | 09:01 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [CI] metal3 job back to voting https://review.opendev.org/c/openstack/ironic/+/950333 | 09:02 |
| adam__metal3 | rpittau. thanks then we are on the smae page | 10:00 |
| iurygregory | rpittau, I just approved 950540 | 11:37 |
| rpittau | thanks iurygregory | 11:54 |
| rpittau | and thanks cid :) | 11:57 |
| cid | \o/ | 11:57 |
| rpittau | bifrost CI is also passing with the revert https://review.opendev.org/c/openstack/bifrost/+/950627 | 11:57 |
| iurygregory | \o/ | 11:59 |
| rpittau | someone said a couple of weeks ago to not just put metal3 job in non-voting when there is an issue, I think I should listen to that more :P | 12:00 |
| iurygregory | XD | 12:00 |
| opendevreview | Merged openstack/ironic master: Revert "Fix redfish driver URL parsing" https://review.opendev.org/c/openstack/ironic/+/950540 | 13:07 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: ci: move n-g-s jobs to non-voting while we sort ngs in ironic gate https://review.opendev.org/c/openstack/networking-generic-switch/+/950655 | 13:13 |
| FreemanBoss[m] | rpittau: good morning. | 13:14 |
| FreemanBoss[m] | Please I've patches that need your attention | 13:14 |
| rpittau | FreemanBoss[m]: hey, I ahve that in my queue, but unfortunately I have other priorities at the moment, not sure when I'll get to them | 13:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 13:44 |
| opendevreview | Elod Illes proposed openstack/ironic-python-agent unmaintained/yoga: [CI] Remove centos-8 based jobs https://review.opendev.org/c/openstack/ironic-python-agent/+/938306 | 13:49 |
| Sandzwerg[m] | \o/ progress. The esp we build used the x86 shim instead of the proper x64 and now the secure boot error is gone and I just need to find out why it's then end up in the grub cli (and gives me an error when I manually boot from virtual media) instead of booting properly like before. But it's progress none the less :) | 14:11 |
| dtantsur | \o/ | 14:11 |
| TheJulia | yay! | 14:11 |
| TheJulia | Often, grub just going to a cli prompt is it failing to find grubenv and the other things its sort of expecting to be present to point it to base configuration | 14:12 |
| Sandzwerg[m] | I'll try to rebuild and in the meantime I'll try to get a centos esp build so I can use the ironic build ipa images. I'm only a bit surprised that kernel & ramdisk are provided but no matching esp, is there a reason for that? | 14:24 |
| shermanm[m] | JayF: catching up on IRC chat, for the snapshot RFE 100% on putting a spec together. context is that I'm mentoring a student this summer on fixing up our internal/hacky implementation of snapshot, and wanted to try and align it with something that would be useful upstream. | 14:49 |
| JayF | ++ | 14:50 |
| dtantsur | JayF, cid, so, the new oslo.service is finally out. Are you guys planning on experimenting with it or should I add it to my backlog? | 14:51 |
| TheJulia | Sandzwerg[m]: I guess nobody has set it up so we build them, but really they are super simple to make anyhow | 14:53 |
| JayF | dtantsur: we already have a patch up for nbm which would utilize it, I'm waiting for u-c to update before I polish it | 14:59 |
| JayF | dtantsur: that's a weird one in that we're mainly making it eventlet-agnostic, because we run under neutron we're too late to decide if it's monkey_patched or not | 14:59 |
| JayF | dtantsur: I think NGS will be same pattern | 14:59 |
| dtantsur | I see, I'm curious about Ironic | 14:59 |
| JayF | I think we've done nearly-zero in ironic proper so far | 15:00 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration job using UEFI boot (default) https://review.opendev.org/c/openstack/ironic/+/939694 | 15:25 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 15:43 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 15:44 |
| Sandzwerg[m] | <TheJulia> "Sandzwerg: I guess nobody has..." <- If you have a fitting machine sure, but now I have to install a centos in a VM just to build the ESP so it fits to the IPA image. That and as I experienced myself you can shoot yourself in the foot build that as well if you use the wrong shim.efi for example. So having something ftting to what is already provided would lower the entry barrier at least in my mind. | 15:45 |
| TheJulia | it totally would provide a lower barrier of entry | 15:53 |
| TheJulia | but could we name it "centos-esp-image-you-can-totally-build-yourself.img" ? ;) | 15:54 |
| Sandzwerg[m] | I'm flexible regarding naming :) | 15:58 |
| TheJulia | very wise ;) | 16:02 |
| TheJulia | (since naming is the hardest problem!) | 16:02 |
| opendevreview | Merged openstack/bifrost master: Add support for downloading CentOS Stream 10 image https://review.opendev.org/c/openstack/bifrost/+/950286 | 16:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Patch configdrive metadata https://review.opendev.org/c/openstack/ironic/+/946677 | 16:26 |
| TheJulia | le sigh: | 16:29 |
| TheJulia | +++ lib/tempest:configure_tempest:385 : oscwrap --os-cloud devstack-admin network show -f value -c id public | 16:29 |
| TheJulia | Error while executing command: No Network found for public | 16:29 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: ci: move n-g-s jobs to non-voting while we sort ngs in ironic gate https://review.opendev.org/c/openstack/networking-generic-switch/+/950655 | 16:36 |
| JayF | TheJulia: ^ I put a note on that change, it's not going to work as-is | 16:51 |
| JayF | say the word and I'll do the update for ya | 16:51 |
| TheJulia | oh, yeah, please if you wouldn't mind | 16:52 |
| JayF | Can we also generally ask cores to be ... judicious about approving changes :) | 16:53 |
| JayF | with our most complex jobs off :D | 16:53 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: ci: move n-g-s jobs to non-voting while we sort ngs in ironic gate https://review.opendev.org/c/openstack/networking-generic-switch/+/950655 | 16:53 |
| TheJulia | yeah, I don't know why it is failing about no public network right now | 16:54 |
| TheJulia | I think some stuff needs to just land to get further along | 16:54 |
| JayF | If you need someone to poke at this with you let me know. I'm obviously looking at things in the background just staying mum unless I have something actionable | 16:55 |
| JayF | I'm still stuck on stop_neutron appearing to do nothing to kill existing uwsgi processes | 16:58 |
| JayF | while start_neutron_service_and_check does | 16:58 |
| JayF | I feel like there's an interaction I don't understand happening | 16:58 |
| TheJulia | Yeah, same | 17:00 |
| TheJulia | its super weird | 17:00 |
| TheJulia | but the even weirder thing, it does seem to get restarted | 17:00 |
| TheJulia | so..... yeah. I'm just confused and also exhausted from jumping to item to item | 17:00 |
| TheJulia | *anyway* we're getting further | 17:00 |
| JayF | if you'd like, we can carve out some time tomorrow, I can have a few fresh VMs locally lined up for faster cycling | 17:01 |
| TheJulia | I'm off tomorrow | 17:01 |
| JayF | ack | 17:01 |
| TheJulia | RH is off tomororw, to be more precise | 17:01 |
| JayF | this afternoon is possible too, if you want | 17:01 |
| JayF | that's nice | 17:01 |
| JayF | I'm off Mon | 17:01 |
| JayF | Would usually take the long weekend too but skipping the one-off days off since I'm outta here for a while in like 3 weeks | 17:02 |
| JayF | the most hilarious side effect of these breakages? I'm recovering my devstack I had used benchmarking tools on, it takes a really long time to delete thousands of nodes lol | 17:07 |
| JayF | gnu parallel helps tho | 17:07 |
| TheJulia | I can this afternoon after my 1-on-1 | 17:07 |
| TheJulia | what is weird, and I haven't focused on yet, is afaik we're not carrying the flag to disable networks | 17:07 |
| TheJulia | and this seems new since yesterday | 17:08 |
| JayF | I suspect there's a subtle thing that's happening/not happening that was before | 17:08 |
| JayF | and has a major cascading effect | 17:08 |
| JayF | especially since the devstack diff itself wasn't that giant | 17:08 |
| JayF | and maybe i'm biased because of the first fix-it patch, but I bet we have an is_service_enabled somewhere else that's triggering/not triggering | 17:09 |
| * JayF just realized he hadn't grepped ngs for that | 17:09 | |
| TheJulia | yeah, likely | 17:10 |
| TheJulia | quite possibly | 17:10 |
| JayF | actually having my old devstack up might be helpful in this regard, it's a month old but could still be useful for a before/after | 17:11 |
| TheJulia | super true | 17:16 |
| TheJulia | I'm wondering if https://opendev.org/openstack/devstack/src/branch/master/lib/neutron_plugins/services/l3#L233-L237 is getting executed | 17:30 |
| TheJulia | i sort of see what is going on | 17:38 |
| TheJulia | https://opendev.org/openstack/devstack/src/branch/master/lib/neutron_plugins/services/l3#L216 doesn't pass | 17:39 |
| TheJulia | router is not found in the list | 17:39 |
| TheJulia | Yeah, running neutron just doesn't even list it | 17:45 |
| Sandzwerg[m] | <TheJulia> "Sandzwerg: I guess nobody has..." <- After I now rebuild it twice for centos (there is a shimx64 and a shimx64-centos and it's not at all clear to mow what the difference is) and it's still not booting with a secure boot error I'm not sure about the "super simple" :/ | 17:55 |
| TheJulia | if they are coming from the same rpm, they are likely the same | 17:56 |
| TheJulia | really, you should be starting with BOOTX64.EFI and GRUBX64.EFI and not using the shim files by name | 17:57 |
| TheJulia | the EFI loaders look for BOOT$SOMETHING.EFI | 17:57 |
| Sandzwerg[m] | well but centos doesn't call them BOOTX64.efi unless I looked wrong? Same for Fedora. Which fits to https://docs.openstack.org/ironic/latest/install/configure-esp.html | 17:58 |
| TheJulia | if I recall correctly, the rpm installers copy the files into place | 17:59 |
| opendevreview | Merged openstack/ironic master: CI: Reconfigure jobs to minimize tinyipa usage https://review.opendev.org/c/openstack/ironic/+/950192 | 17:59 |
| Sandzwerg[m] | I mean in the ESP the will be called that but in centos itself they are not, and on fedora my first error was to use the wrong shim at first, the x86 instead of the x64 | 17:59 |
| TheJulia | because suddenly changinging things unexpectedly could be bad^(TM) | 18:00 |
| TheJulia | so, for shim, what file are you copying? | 18:00 |
| Sandzwerg[m] | I stand corrected, there is a /boot/efi/EFI/BOOT/BOOTX64.efi which has the same sha256 as /boot/efi/EFI/centos/shimx64.efi | 18:02 |
| Sandzwerg[m] | I tried both, so far with the same error of "Unable to boot virtual optical drive because of secure boot policy" which is odd because the fedora IPA before was just showing me a secure boot error when it was broken | 18:03 |
| Sandzwerg[m] | But at least the grubx64.efi is at /boot/efi/EFI/centos/grubx64.efi | 18:07 |
| Sandzwerg[m] | Well one more try | 18:07 |
| opendevreview | Merged openstack/ironic master: CI: Remove the multinode grenade job https://review.opendev.org/c/openstack/ironic/+/950224 | 18:15 |
| Sandzwerg[m] | Ha now I'm back at a Security Violation. So probably my centos is not the same as the IPA. Probably time to build that as well 😅 | 18:15 |
| Sandzwerg[m] | Enough for today, have a nice one o/ | 18:19 |
| TheJulia | that seems weird, I guess worthwhile to try and verify the signing and detail, since there *are* two keys, one which was supposed to expire... at some point | 18:51 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/sushy master: Fix type of Targets in ShareParameters for Dell https://review.opendev.org/c/openstack/sushy/+/950694 | 19:26 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 20:32 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 21:07 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: ci: workaround neutron's move to uwsgi only https://review.opendev.org/c/openstack/networking-generic-switch/+/950559 | 21:18 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job https://review.opendev.org/c/openstack/ironic/+/950206 | 21:20 |
| TheJulia | minor bashate error it looks like, but easy to clean up later as stuff is running (and one has already passed) | 21:42 |
| JayF | https://zuul.opendev.org/t/openstack/build/19c046570986420c8796bee35e22247c start_ovn_services is going boom in the ironic job | 21:53 |
| JayF | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) <-- which seems really wrong | 21:53 |
| JayF | Do we even reconfigure OVN? | 22:07 |
| JayF | if not maybe the SKIP_STOP_OVN variable is the better move | 22:07 |
| JayF | instead of trying to stop/start it | 22:07 |
| * JayF will wait for jobs to check in before doing another swing at it | 22:07 | |
| opendevreview | Merged openstack/ironic master: remove explicit eventlet usage around time.sleep https://review.opendev.org/c/openstack/ironic/+/949664 | 22:08 |
| JayF | :-| we really shouldn't be approving impactful changes with most of our neutron jobs -nv | 22:11 |
| JayF | that one had a previous pass on the ovn-ipv6 job so should be fine | 22:11 |
| * JayF doesn't want to add any layers to the broken-ci-onion :D | 22:12 | |
| TheJulia | we don't change it | 22:32 |
| TheJulia | but stopping neutron tears it down afaik because one of the prior failures showed after stop, and then restart.... neutron failed to be able to talk to ovs | 22:32 |
| JayF | yeah makes sense | 22:43 |
| JayF | I'm going to revisit a little later and double check everything | 22:43 |
| JayF | but for now my wife just got home so going to step away, but I'll get a revision in before tonight | 22:43 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!