| opendevreview | waleed mousa proposed openstack/ironic-python-agent master: update NVIDIA NIC firmware images and settings by ironic-python-agent https://review.opendev.org/c/openstack/ironic-python-agent/+/566544 | 06:37 |
|---|---|---|
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/zed: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/867639 | 07:30 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/yoga: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/868777 | 07:43 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/xena: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/868778 | 07:44 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/wallaby: Retry on iDRAC SYS518 errors for all requests https://review.opendev.org/c/openstack/sushy/+/868779 | 07:48 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Fix selinux context of published image hardlink https://review.opendev.org/c/openstack/ironic/+/868521 | 08:26 |
| rpittau | good morning ironic! o/ | 08:26 |
| rpittau | TheJulia: I know, that's not ideal, I haven't figured out yet what changed but the final size seems in between 3 and 4gb which is roughly 1gb bigger than what we had before, way too much | 08:28 |
| rpittau | I guess some drivers/firmware and/or package name change, maybe some more libraries of osrt | 08:28 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Use tinycore 13 for base ramdisk image https://review.opendev.org/c/openstack/ironic/+/868093 | 08:44 |
| ajya | Hi Ironic, happy new year! | 08:49 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix selinux context of published image hardlink https://review.opendev.org/c/openstack/ironic/+/868521 | 08:49 |
| rpittau | hey ajya, happy new year :) | 08:49 |
| ajya | Hi rpittau, is this https://review.opendev.org/c/openstack/sushy/+/868759 fixing the "No extensions found for "system" under namespace "sushy.resources.system.oems""? | 08:49 |
| rpittau | ajya: yep, that should do it | 08:50 |
| rpittau | not sure why vmedia is failing though, haven't checked yet | 08:50 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix unbound variable in devstack plugin https://review.opendev.org/c/openstack/ironic/+/868990 | 09:12 |
| rpittau | the vmedia job works fine in ironic CI so if there's an issue looks like it's related to sushy | 09:13 |
| rpittau | there it is! https://review.opendev.org/c/openstack/sushy/+/868759 if anyone has a moment we should merge the rest of the fixes ASAP | 10:23 |
| iurygregory | good morning Ironic, happy new year! | 11:26 |
| opendevreview | Merged openstack/sushy master: Fix tox4 and setuptools errors https://review.opendev.org/c/openstack/sushy/+/868759 | 12:29 |
| kamleshChauvhan | Hi Ironic ! happy New Year . | 13:02 |
| kamleshChauvhan | Need help for second review on https://review.opendev.org/c/openstack/sushy/+/864845 & https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/865803. These are small changes. | 13:03 |
| opendevreview | Verification of a change to openstack/sushy-tools master failed: Fix tox4 error https://review.opendev.org/c/openstack/sushy-tools/+/868760 | 13:06 |
| ajya | rpittau: need to backport that sushy tox4 and setuptools fix too? | 13:15 |
| rpittau | ajya: not sure honestly, I guess it doesn't hurt, it's retro compatible anyway | 13:27 |
| rpittau | I found a "fun" issue with mock and oslo importutils that breaks tests in py3.10 and higher, hopefully we're not using it in too many places | 13:29 |
| ajya | rpittau: there are backports with the same error. kamleshChauvhan maybe you can check and backport? | 13:30 |
| rpittau | ajya: then we need it, unless tox version is being locked in upper-constraints, but I highly doubt it | 13:31 |
| rpittau | TheJulia, JayF plesae check https://review.opendev.org/c/x/ironic-staging-drivers/+/868762 when you have a moment, thanks | 13:33 |
| rpittau | the vmedia job keeps being very unstable | 13:36 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Fix for tox4 and setuptools https://review.opendev.org/c/openstack/ironic-python-agent/+/867962 | 13:41 |
| ajya | rpittau: me or kamleshChauvhan will backport it as we have some backports in sushy | 13:41 |
| rpittau | ajya: sounds good, thanks | 13:41 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/zed: Fix tox4 and setuptools errors https://review.opendev.org/c/openstack/sushy/+/868785 | 14:02 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/yoga: Fix tox4 and setuptools errors https://review.opendev.org/c/openstack/sushy/+/868786 | 14:02 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/xena: Fix tox4 and setuptools errors https://review.opendev.org/c/openstack/sushy/+/868787 | 14:03 |
| opendevreview | kamlesh chauvhan proposed openstack/sushy stable/wallaby: Fix tox4 and setuptools errors https://review.opendev.org/c/openstack/sushy/+/868788 | 14:03 |
| kamleshChauvhan | rpittau , aija : backported till wallaby release. | 14:06 |
| ajya | thanks | 14:07 |
| rpittau | tnx kamleshChauvhan | 14:13 |
| opendevreview | Riccardo Pittau proposed openstack/ironic stable/zed: Fix CI https://review.opendev.org/c/openstack/ironic/+/868789 | 15:51 |
| rpittau | good night! o/ | 16:04 |
| ajya | hm, yes, vmedia job in sushy is noticeably less stable | 16:34 |
| ajya | o/ | 16:34 |
| opendevreview | Merged openstack/virtualbmc master: Fixes for tox 4.0 https://review.opendev.org/c/openstack/virtualbmc/+/867843 | 17:01 |
| JayF | vmedia job everywhere is tbh | 17:09 |
| JayF | > ironic-tempest-uefi-redfish-vmedia | 18:26 |
| JayF | is the culprit of most of our CI failures | 18:26 |
| JayF | I've yet to determine why | 18:26 |
| kubajj | Hi everyone, Happy New Year! | 18:46 |
| kubajj | JayF: Is CI still failing? (I wanted to check before rechecking) | 18:47 |
| JayF | I think we have something intermittant with the ironic-tempest-uefi-redfish-vmedia on ironic-proper | 18:57 |
| JayF | and that is blocking some of the CI fix patches in other projects (like IPA) | 18:58 |
| JayF | Yeah; I've been looking at this off and on all day | 22:01 |
| JayF | and can't really figure out what's going on | 22:01 |
| JayF | things are failing to clean | 22:02 |
| JayF | no errors in conductor or sushy-tools logs | 22:02 |
| JayF | it all looks OK just maybe timed out ? | 22:02 |
| JayF | I wonder if there's some kind of knock-on side effect of raising ram in jobs | 22:15 |
| JayF | This is going to be my primary focus tomorrow; if non-US folks work on looking at this please document what you are doing so I can pick it up in the morning | 22:15 |
| TheJulia | Have the qemu logs been checked? Kernel log on hyper visor. Upping the ram to half the test VM basically leaves no room for the base is to breath | 22:32 |
| TheJulia | I’ll be around tomorrow so I’ll be able to look | 22:32 |
| iurygregory | we increased the VM_SPECS_RAM to 4096... | 22:35 |
| iurygregory | because of CS9 based IPA | 22:35 |
| iurygregory | https://opendev.org/openstack/ironic/commit/6b84fbf8f2a92fc9c591e26eeec4b69cc1aa2ffe | 22:35 |
| iurygregory | maybe we need to try to reduce the CS9 image, build a new one and publish so we can reduce to 2600 | 22:36 |
| TheJulia | I’d personally prefer it, I will be looking at the last published image tomorrow | 22:38 |
| JayF | looking at those TheJulia | 22:40 |
| JayF | nothing in syslog.txt indicating out of memory condition or errors from qemu | 22:44 |
| JayF | I can't find a dedicated kernel log; but given iptables block messages are in syslog I'm fairly confident I'm looking at the right spot | 22:45 |
| JayF | it's also just redfish vmedia | 22:45 |
| JayF | which makes me go hrm as to any indications that it'd be generally broken | 22:45 |
| JayF | (not saying that would be a wrong assumption; just that I can't connect the dots of why it'd only happen in redfish vmedia) | 22:46 |
| ashinclouds[m] | I would guess increased memory pressure by open files mapped into ram. We saw something like this the last time we bumped the memory allocation for the test vms to like 4GB | 22:56 |
| ashinclouds[m] | Err, wrong client | 22:56 |
| TheJulia | Anyway, I can look tomorrow. Going to hope the migraine meds do their thing as this next wave of storms roll through | 22:57 |
| JayF | yeah I was in bad shape w/r/t migraines all holiday b/c the weather was osscilating | 22:58 |
| JayF | hope you feel OK; I'll take a look tomorrow as well with fresher eyes | 22:58 |
| JayF | most of today spent catching up and looking into this at a surface level | 22:58 |
| TheJulia | I hope this weather chills out soon, but it looks like tomorrow might be a not fun day. But on a plus side I got some writing done \o/ | 23:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!