| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 00:54 |
|---|---|---|
| *** __ministry is now known as Guest381 | 07:10 | |
| opendevreview | Artem Goncharov proposed openstack/pbr master: add python 3.12 support https://review.opendev.org/c/openstack/pbr/+/924216 | 11:04 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 18:04 |
| clarkb | fungi: ^ so I thought things weren't determinstic with grub kernel updates on arm for whatever reasno then looking at it with fresh eyes today I realized I updated grub post reboot not pre reboot (the shame) | 18:04 |
| fungi | hah | 18:05 |
| clarkb | fungi: anyway I suspect that may actually be workable as a solution to our problem now | 18:05 |
| fungi | happens to all of us at some point | 18:05 |
| clarkb | this is the ozj side. Going to check system-config which is what was actually holding up noble packages next | 18:05 |
| clarkb | and then separately we should look into why git cache updates on nb04 are so flaky if that is still happening after I restarted nsd on ns04 | 18:05 |
| clarkb | ok the system-config chagne is hitting the issue whre upstream packages don't work for arm64 due to the invalid instruction in the kernel | 18:06 |
| clarkb | I think I may just mark those jobs non voting | 18:06 |
| fungi | is it still using older openafs then? | 18:07 |
| clarkb | fungi: yes it is the noble job that is failing | 18:07 |
| clarkb | so chicken and egg there | 18:07 |
| fungi | yeah, if only we could use our ppa build there | 18:08 |
| clarkb | bookworm also failed though which I thought was fixed | 18:08 |
| fungi | up to 350 uncaught bounce notifications to openstack-discuss-owner, i'm going to need to go through and manually disable these subscriptions (after i collate them all) | 18:09 |
| clarkb | I wonder if we unfixed the fix with the new updates | 18:09 |
| clarkb | that would be annoying if so | 18:09 |
| fungi | bookworm is not fixed in debian yet | 18:09 |
| fungi | https://packages.debian.org/search?keywords=openafs-client | 18:09 |
| clarkb | that rules out the security bugfixes regressing I guess. STill I thought upstream had fixed the cannot load the kernel module problem on bookworm | 18:10 |
| clarkb | and I was pretty sure I tested that so maybe some other update in the interim broke us? | 18:10 |
| fungi | bookworm is on 1.8.9, without any backported patches | 18:10 |
| clarkb | but it had the patch for the modprobe error right? that was the whole bug I filed with debian and it got updated. I'm rpetty sure I checked this after the bug was closed | 18:12 |
| fungi | it was also removed from testing (trixie) in march, so needs to get reintroduced before it can migrate, and probably won't see any backports until that happens unless the debian security team intervenes: https://tracker.debian.org/news/1512372/openafs-removed-from-testing/ | 18:13 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 18:13 |
| fungi | references in the autoremoval are to https://bugs.debian.org/1060896 and https://bugs.debian.org/1062829 | 18:14 |
| clarkb | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069781 | 18:15 |
| clarkb | maybe that bug is just saying the package updated in unstable? | 18:15 |
| fungi | yeah | 18:16 |
| fungi | the two bugs resulting in autoremoval are also fixed, i guess reintroduction to testing has just not been requested yet | 18:16 |
| fungi | related, i got a reply from sarnold in #ubuntu-security that was essentially "patches welcome" (if we have time to backport security patches and initiate the sru process to each release) | 18:17 |
| clarkb | oh wait those jobs are already marked non voting from when we first ran into this so the only failuer is still in getting the centos 9 arm64 job happy | 18:18 |
| clarkb | if we do that we should also backport the arm fix | 18:18 |
| clarkb | that patch is really straightforward iirc | 18:19 |
| clarkb | ok I think I have evidence now that we are install the 529 kernel and headers, doing a grub update, then rebooting and booting back onto the 527 kernel we were running before | 18:47 |
| clarkb | but sometimes this seems to work and we boot onto 529? This is very weird | 18:48 |
| clarkb | separately when you install the new kernel headers package for the kernel you will reboot onto dnf uninstalls the headers for the kernel you will boot onto | 18:48 |
| clarkb | I think this is the fundamental underlying issue and is why debuntu are fine they won't remove old kernels until you autoremove or similar to clean that stuff up and the dkms packages will build against all kernels on the system | 18:49 |
| fungi | which kernel we boot *could* be provider specific, depending on whether the boot routine reads the updated configuration | 18:54 |
| fungi | like grub config update vs efi config update | 18:55 |
| clarkb | oh ya the arm machines boot efi | 18:59 |
| clarkb | so maybe there is an additional step we need? however I think the log says it updates all that stuff | 18:59 |
| clarkb | https://zuul.opendev.org/t/openstack/build/2abe1eabbbfd4fa3b5f3caafcc7c0e88/console | 19:00 |
| fungi | at least on debian derivatives, it's up to how grub is configured whether update-grub also updates efi configs | 19:00 |
| fungi | but yeah, it's also possible it thinks it's updating the config but then the config it updated isn't the one the boot process is ultimately reading (e.g. more than one copy in different partitions/locations) | 19:01 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 19:02 |
| clarkb | this change is slowly becoming more debug info that actual effort :) | 19:02 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 19:12 |
| clarkb | I should probably hold a node at this point, but its a good exercise in debugging from afar (something we often ask others to do) | 19:14 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 19:19 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Don't explicitly install kernel-devel when building openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935984 | 21:46 |
| clarkb | fungi: ^ I took a long lunch break and it occurred to me we may be able to get away with something as simple as that. We'll see | 21:47 |
| clarkb | if that works then I'm going to say stepping away from the problem for a bit was the genius move. If it doesn't work oh well it was worth a shot anyway | 21:47 |
| clarkb | nope that doesn't work because rpmbuild can see there is a newer version and it wants that newer version ugh | 21:51 |
| clarkb | I think that must come from the dnf builddep -y $SRPM step | 21:53 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 21:59 |
| clarkb | the problem wtih log collection (or one of them anyway) is taht fetch-output and stage-output have different default expectations for where we stage the content on the test nodes | 22:00 |
| clarkb | BuildRequires: kernel-devel <- that is what the rpm spec file says so I guess its rpm/dnf that are seeing there is no other upstream version and demanding the latest and greatest | 22:04 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 22:57 |
| opendevreview | Clark Boylan proposed openstack/openstack-zuul-jobs master: Update system packages and reboot when building centos openafs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/935813 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!