| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix Cyborg endpoints again https://review.opendev.org/c/openstack/kolla-ansible/+/893019 | 05:25 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: bifrost: Populate bifrost host vars on deprovision https://review.opendev.org/c/openstack/kayobe/+/897751 | 06:07 |
| opendevreview | Merged openstack/kayobe stable/yoga: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896381 | 06:23 |
| opendevreview | Merged openstack/kayobe master: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897260 | 07:34 |
| opendevreview | Sven Kieske proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 07:42 |
| opendevreview | Alex Welsh proposed openstack/kayobe master: Add option to skip kolla docker registry login https://review.opendev.org/c/openstack/kayobe/+/896655 | 08:30 |
| opendevreview | Alex Welsh proposed openstack/kayobe master: Add option to skip kolla docker registry login https://review.opendev.org/c/openstack/kayobe/+/896655 | 08:32 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: Adding missing group_vars for gnocchi service. Using proper variables in haproxy config for vitrage and venus services. https://review.opendev.org/c/openstack/kolla-ansible/+/897782 | 09:39 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: Fix gnocchi/venus/vitrage when using single external frontend for haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/897782 | 09:48 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: Fix gnocchi/venus/vitrage when using single external frontend for haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/897782 | 09:49 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: Adds a kolla-variables parent job https://review.opendev.org/c/openstack/kolla-ansible/+/897548 | 10:00 |
| mnasiadka | hmm, seems we have some breakage - template error while templating string: Could not load \"namespace_haproxy_for_cell\": 'namespace_haproxy_for_cell'. | 10:04 |
| tcee_ | Hi all. I was wondering if any core reviewers would be willing to take a look at this: https://review.opendev.org/c/openstack/kolla-ansible/+/877776 | 10:35 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Add introspection rule to update deploy kernel location https://review.opendev.org/c/openstack/kayobe/+/808111 | 11:09 |
| opendevreview | Sven Kieske proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 11:25 |
| SvenKieske | mnasiadka: could you already take a look? I have reproducible errors building even kolla-toolbox | 12:11 |
| SvenKieske | ERROR:kolla.common.utils.kolla-toolbox:ca-certificates is already the newest version (20230311). Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. | 12:12 |
| SvenKieske | https://zuul.opendev.org/t/openstack/build/89f8022743704fdd9a1fb5cdb621a1ea/log/kolla/build/000_FAILED_kolla-toolbox.log | 12:12 |
| SvenKieske | not sure if this is the root cause, going through the logs now | 12:12 |
| hrw | ERROR:kolla.common.utils.kolla-toolbox:libssl-dev : Depends: libssl3 (= 3.0.9-1) but 3.0.11-1~deb12u1 is to be installed | 12:24 |
| hrw | mirror out of sync | 12:24 |
| SvenKieske | hrw: ah that seems to be about right, thanks! | 12:26 |
| SvenKieske | kevko: or zigo: can you take a look at http://osbpo.debian.net/debian ? the "bookworm-bobcat-backport-nochange/main" mirror seems to be out-of-sync (not sure if you can manually resync this stuff?) | 12:28 |
| SvenKieske | mhm no, that's not correct | 12:29 |
| SvenKieske | I'm terrible this week when it comes to reading logs.. | 12:29 |
| fungi | preparing for today's opendev sysadmins meeting, where did we get with the osbpo mirroring request? does kolla consume that in its ci jobs such that having mirrors of it near the job nodes would help with speed or stability of testing? | 12:30 |
| SvenKieske | fungi: from my pov it would help, but I don't have numbers how much speedup etc this would bring us :) | 12:31 |
| SvenKieske | ironically just debugging an out of sync mirror :D | 12:31 |
| fungi | no, that's fine. i just wanted to make sure there was some desire and potential benefit to it before i brought it up | 12:32 |
| SvenKieske | yeah sure, please send all complaints my way :) | 12:32 |
| fungi | looks like there are a bunch of dists in it. i suppose not all of them are needed (we'll have to list them individually in the reprepro config) | 12:34 |
| fungi | what's the difference between the backports and backports-nochange dists? do you happen to know? (do you need both?) | 12:35 |
| fungi | i guess you need both since they seem to have entirely different packages in their indices | 12:37 |
| SvenKieske | not sure, I assume the "nochange" is direct backport and without "nochange" there are some code changes? but just guessing. kevko or zigo now it. :) | 12:38 |
| zigo | SvenKieske: The diff is, -nochange contain packages we backport (but don't maintain in the OpenStack team) without changing them. | 12:39 |
| zigo | Both are needed. | 12:39 |
| fungi | got it, that's what it looked like to me. thanks for confirming | 12:39 |
| zigo | What do you mean by "out of sync" ? | 12:39 |
| SvenKieske | zigo: https://zuul.opendev.org/t/openstack/build/89f8022743704fdd9a1fb5cdb621a1ea/log/kolla/build/000_FAILED_kolla-toolbox.log hrw concluded the mirror is out of sync, see line 75 in that log. seems at least possible. currently building locally. | 12:40 |
| zigo | I don't think I've been doing funny stuff with OpenSSL ... | 12:41 |
| zigo | That's tipycally things I prefer not to touch. | 12:41 |
| SvenKieske | local build of kolla-toolbox at least works | 12:42 |
| SvenKieske | my local build says: INFO:kolla.common.utils.kolla-toolbox:Setting up libssl-dev:amd64 (1.1.1w-0+deb11u1) ... | 12:42 |
| SvenKieske | ah wrong lib..should search for libssl3 | 12:42 |
| fungi | kolla.common.utils.kolla-toolbox:libssl-dev : Depends: libssl3 (= 3.0.9-1) but 3.0.11-1~deb12u1 is to be installed | 12:42 |
| fungi | so libssl-dev wants outdated libssl3, this is common when mixing package repositories | 12:43 |
| SvenKieske | mhm weird, my local build does not even mention libssl3, like, in the complete build log? am I missing some option? | 12:43 |
| fungi | libssl3 3.0.11-1~deb12u1 is the current package in bookworm | 12:44 |
| SvenKieske | ah my local build is on deb11 and broken on deb12 | 12:45 |
| fungi | perhaps something is keeping apt from updating libssl-dev, but i don't see any bersion of it circa 3.0.9-1 | 12:45 |
| fungi | er, version | 12:45 |
| fungi | https://packages.debian.org/libssl-dev | 12:45 |
| kevko | Hi \o/ | 12:45 |
| SvenKieske | hey kevko :) | 12:46 |
| fungi | bullseye uses 1.1.1w and bookworm skips straight to 3 | 12:46 |
| fungi | "[2023-09-09] Accepted 3.0.10-1~deb12u1 in bookworm (medium) (Debian FTP Masters)" https://packages.qa.debian.org/o/openssl.html | 12:48 |
| zigo | SvenKieske: libssl3 is *NOT* in Bullseye (ie Debian 11). | 12:48 |
| kevko | reading | 12:48 |
| fungi | so 3.0.9 hasn't been current in bookworm since a month ago | 12:48 |
| zigo | So if you customize Bullseye to use libssl3, you're on your own ... | 12:48 |
| kevko | what are u trying to build ? | 12:48 |
| fungi | oh, that job is running on bullseye? | 12:49 |
| kevko | SvenKieske: can u paste the build command here ? | 12:49 |
| SvenKieske | just kolla-toolbox, originally fluentd-package, but fluentd works, just kolla-toolbox is now broken, two times in a row :) | 12:49 |
| zigo | libssl-dev : Depends: libssl3 (= 3.0.9-1) <--- Where does this come from? (it's not from Debian at least) | 12:49 |
| kevko | fungi: kolla uses bullseye in some branches ..i think bookworm is master currently / bobcat in some time | 12:49 |
| fungi | https://zuul.opendev.org/t/openstack/build/89f8022743704fdd9a1fb5cdb621a1ea/log/zuul-info/inventory.yaml#20 | 12:50 |
| kevko | SvenKieske: --base WHAT --base-tag WHAT ? | 12:50 |
| fungi | the job seems to have run on bookworm | 12:50 |
| SvenKieske | kevko: the build fungi referenced above | 12:50 |
| kevko | ah, ok | 12:51 |
| kevko | let me check | 12:51 |
| fungi | zigo: until 2023-09-09 the libssl-dev in bookworm would have depended on libssl3 3.0.9-1, so if something's got a month-old libssl-dev installed and apt isn't able to upgrade it for some reason, i guess that could happen | 12:51 |
| zigo | Oh, ok. | 12:52 |
| kevko | building | 12:52 |
| kevko | give me few minutes | 12:52 |
| fungi | i'm taking a look at our bookworm nodepool image builds to make sure they're current, at least | 12:52 |
| SvenKieske | now also building lookally | 12:52 |
| zigo | Do I understand well that we're getting a new mirror for osbpo ? | 12:52 |
| kevko | hmm - i've updated extrepo in kolla to use bobcat | 12:53 |
| fungi | zigo: SvenKieske was asking that we mirror it locally to ci job nodes, so i was going to bring it up in the opendev sysadmins meeting today | 12:53 |
| zigo | Should I get it added to the standard mirror list in extrepo, or is it better to keep it the way it is, as we don't want anyone execpt the CI do use the mirror? | 12:53 |
| zigo | fungi: Ok, thanks for it. FYI, it's kind of small, and (unless we have a new relaese) doesn't change often. | 12:54 |
| fungi | zigo: if we add them, they wouldn't be safe for outside use, just there to speed up the ci jobs | 12:54 |
| zigo | Ok, wont publish the URL then. | 12:54 |
| fungi | probably we'd start just mirroring the bookworm bobcat packages anyway | 12:54 |
| SvenKieske | seems reasonable | 12:54 |
| zigo | Though I'm happy there's another mirror (I know wikimedia foundation has a mirror of them too...). | 12:54 |
| zigo | fungi: Please do all, it's only 22GB in total ... | 12:54 |
| fungi | i don't foresee ci jobs wanting to run the stretch victoria era packages, for example | 12:54 |
| zigo | It's reassuring for me if there's a backup outside of my employer ... | 12:55 |
| fungi | zigo: well, we mirror apt package repositories with reprepro, so have to list every dist individually in the config anyway | 12:55 |
| zigo | Oh... ok. | 12:55 |
| zigo | Hint: apt-get install ftpsync ... :P | 12:56 |
| SvenKieske | kevko: noob question (never build debian images manually before): how do you build bookworm locally with zuul? :D my build errors fast with "only releases [..] Debian [..] (bullseye) are supported".. | 12:56 |
| fungi | i'm sure ftpsync is great until you want to have partial mirrors | 12:56 |
| SvenKieske | without* | 12:56 |
| kevko | i never build package in zuul locally :D | 12:57 |
| SvenKieske | rsync can do filters ;) just kidding, won't give unwanted advice on infrastructure topics. | 12:57 |
| kevko | without ? | 12:57 |
| kevko | sbuilder | 12:57 |
| fungi | SvenKieske: we already use rsync with filtering for rpm-based distro package mirroring | 12:57 |
| fungi | we've found reprepro convenient because 1. it guarantees the indices it generates are 100% in sync with the packages it retrieved, and 2. we can set it to delay removal of old packages for a while so that jobs don't break if they apt update and then apt install something and there was a mirror update in between those commands | 12:58 |
| kevko | SvenKieske: INFO:kolla.common.utils.kolla-toolbox:Removing intermediate container 69b54f656bb0 | 12:59 |
| kevko | INFO:kolla.common.utils.kolla-toolbox: ---> fd55e5d324ba | 12:59 |
| kevko | INFO:kolla.common.utils.kolla-toolbox:Successfully built fd55e5d324ba | 12:59 |
| kevko | INFO:kolla.common.utils.kolla-toolbox:Successfully tagged dockerhub.ultimum.io/kolla-dev/kolla-toolbox:16.1.0 | 12:59 |
| kevko | INFO:kolla.common.utils.kolla-toolbox:Built at 2023-10-10 14:58:23.894841 (took 0:03:39.426090) | 12:59 |
| kevko | kolla-build --nopush --base debian --base-tag bookworm --registry dockerhub.ultimum.io --namespace kolla-dev kolla-toolbox | 12:59 |
| kevko | kolla-toolbox builds fine | 12:59 |
| SvenKieske | mhm :/ | 12:59 |
| SvenKieske | interesting | 12:59 |
| opendevreview | Michal Arbet proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 13:01 |
| fungi | looks like that example build ran in rackspace's iad region starting at 11:31:51 utc. we uploaded a fresh debian-bookworm image to glance there over 18 hours ago, so it should be reasonably current | 13:02 |
| SvenKieske | kevko: when I run your exact same command locally in a freshly installed "kolla" venv on this fluentd branch, recently updated to master I get: | 13:05 |
| SvenKieske | INFO:kolla.common.utils.base:Step 3/33 : RUN . /etc/os-release; if [ "${PRETTY_NAME#Debian GNU/Linux 11 (bullseye)}" = "$PRETTY_NAME" ]; then echo "Only releases \"Debian GNU/Linux 11 (bullseye)\" are supported on debian"; false; fi | 13:05 |
| SvenKieske | ERROR:kolla.common.utils.base:The command '/bin/sh -c . /etc/os-release; if [ "${PRETTY_NAME#Debian GNU/Linux 11 (bullseye)}" = "$PRETTY_NAME" ]; then echo "Only releases \"Debian GNU/Linux 11 (bullseye)\" are supported on debian"; false; fi' returned a non-zero code: 1 | 13:05 |
| kevko | SvenKieske: installed from pypi ? | 13:05 |
| kevko | or installed from git | 13:05 |
| SvenKieske | ah yeah | 13:05 |
| SvenKieske | mb | 13:05 |
| SvenKieske | I fell into that old tarpit again | 13:06 |
| kevko | I am never using released packages in pypi :D | 13:06 |
| kevko | because it's not usable :D | 13:06 |
| kevko | for example kolla-ansible - I have my own docker image ..because it is unusable when you rely on pypi version | 13:07 |
| kevko | and zuul is testing actual git tree with applied change .. | 13:07 |
| SvenKieske | yeah I know that. there's almost every week a user I remind of that :D | 13:07 |
| kevko | I can send my scented dockerfile in light pink color for review :P | 13:09 |
| kevko | let's see if it will build .. | 13:10 |
| kevko | https://zuul.opendev.org/t/openstack/stream/630b13e4850c4b38a9ab352fa56a70a4?logfile=console.log | 13:10 |
| SvenKieske | now my local build also works | 13:16 |
| SvenKieske | fluentd works as well (locally) | 13:17 |
| kevko | does anyone know how can i acess log of some review if i know already done ? | 13:18 |
| kevko | for example - i know that build-debian finished ..but another jobs are still running ...but i want to access logs of build-debian job | 13:18 |
| SvenKieske | easy | 13:19 |
| kevko | so please :) | 13:19 |
| hrw | kevko: zuul has web interface | 13:19 |
| SvenKieske | 1. go to https://zuul.opendev.org/t/openstack/builds filter by changeid, done? | 13:20 |
| * kevko is trying not to look like a fool and currently searching on the web, wait | 13:20 | |
| SvenKieske | i.e. https://zuul.opendev.org/t/openstack/builds?change=894948&skip=0 | 13:20 |
| SvenKieske | for the current stuff | 13:20 |
| SvenKieske | I was so frustrated, I actually tried to use zuul UI, and it worked! ;) | 13:21 |
| SvenKieske | I still wonder what is the lag between a job being done and a bot writing the result to gerrit, it's a quite large delay. I guess there will be a good reason for it. | 13:21 |
| fungi | kevko: it's only very recently that the zuul dashboard started providing completed build results for incomplete (still enqueued) buildsets, but yes it should "just work" these days | 13:22 |
| SvenKieske | you can also filter by voting job etc, quite convenient actually :) | 13:22 |
| fungi | SvenKieske: zuul reports to gerrit when the buildset is done, so one comment with all job results from that pipeline rather than individual comments for each build that completes | 13:22 |
| SvenKieske | fungi: ah I guessed something in that direction, nice to know | 13:23 |
| SvenKieske | kevko: and it failed again: https://zuul.opendev.org/t/openstack/build/630b13e4850c4b38a9ab352fa56a70a4 | 13:23 |
| mnasiadka | I'm rather more interested what happened that importing a filter in kolla-ansible fails | 13:23 |
| fungi | related, you can filter the status view for a specific change, expand it to see the individual builds for that buildset, and click on the name of a completed job to get to its build results page | 13:24 |
| SvenKieske | mnasiadka: ah your previous question, also interesting, didn't look into it yet. | 13:24 |
| opendevreview | Michal Arbet proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 13:25 |
| SvenKieske | for ppl who don't want to scroll back that far, mnasiadka is talking about: https://review.opendev.org/c/openstack/kolla-ansible/+/897548 | 13:25 |
| kevko | SvenKieske: i know, but now I ordered zuul to show us his secrets , hahaha | 13:26 |
| SvenKieske | kevko: now you want to know :D good old "rdepends", good idea | 13:26 |
| SvenKieske | that is the single apt command I have imprinted on my brain | 13:27 |
| kevko | mnasiadka: this fail is not caused by that review ..it is something already merged | 13:27 |
| mnasiadka | kevko: but that code hasn't changed since months at least | 13:28 |
| kevko | mnasiadka: because same error has my trivial podman log fix - > https://review.opendev.org/c/openstack/kolla-ansible/+/893187 | 13:28 |
| mnasiadka | so that needs to be something else | 13:28 |
| mnasiadka | maybe some dependency version | 13:28 |
| kevko | which is actually changing only shell script :/ | 13:28 |
| mnasiadka | but of course Ansible is great at printing errors | 13:28 |
| SvenKieske | mhm | 13:30 |
| SvenKieske | time to test that ara stuff we merged? ;) | 13:30 |
| SvenKieske | don't know if the output would be detailed enough, but could give us a hint, no? | 13:31 |
| kevko | kolla build issue -> | 13:42 |
| kevko | http://mirror.iad3.inmotion.opendev.org/debian bookworm/main amd64 Packages | 13:42 |
| kevko | bad version | 13:42 |
| fungi | hah, in starting to work on a proposal for the osbpo mirroring, i see we've still got a vestigial debian-openstack mirror which was last updated in 2017. we could clean that up and repurpose it, no new afs volume required | 13:43 |
| opendevreview | Pedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Fix the docker container dimensions comparison for short notation https://review.opendev.org/c/openstack/kolla-ansible/+/886500 | 13:43 |
| kevko | fungi: http://mirror.iad3.inmotion.opendev.org/debian bookworm/main amd64 Packages -> 3.0.9-1 | 13:44 |
| kevko | mirror broken | 13:44 |
| fungi | kevko: i noticed a few minutes ago that our debian mirror was 3 days behind because of a leaked lockfile. i'm resyncing it now so should hopefully be updated shortly | 13:44 |
| kevko | okay | 13:45 |
| fungi | still, 3.0.9-1 shouldn't have been in there for the past month, so 3 days of not updating doesn't explain why that's there | 13:45 |
| fungi | unless we're pulling from an outdated mirror upstream of us | 13:45 |
| kevko | hmm, did you switch unstable to bookworm ? | 13:46 |
| kevko | in your mirror "something" | 13:46 |
| fungi | we don't mirror from unstable | 13:46 |
| kevko | because 3.0.9-1 is unstable | 13:47 |
| kevko | https://metadata.ftp-master.debian.org/changelogs//main/o/openssl/openssl_3.0.11-1~deb12u1_changelog | 13:47 |
| kevko | or ..better said ..deb12u1 means that this was released to stable debian | 13:48 |
| SvenKieske | what did I miss after "bad version"? sorry, had to reboot | 13:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: OpenSearch migration - prune old Kibana indices https://review.opendev.org/c/openstack/kolla-ansible/+/897667 | 13:49 |
| fungi | the mirror update which is in progress changes the Packages entry for libssl3 from 3.0.9-1 to 3.0.11-1~deb12u1 so this will be consistent with whatever third-party package you're pulling in that's trying to update libssl3 now | 13:50 |
| kevko | SvenKieske: they are mirroring something outdated :) | 13:50 |
| SvenKieske | yeah, just read the channel logs :) | 13:51 |
| SvenKieske | so I guess just wait and we are good? :) | 13:51 |
| fungi | SvenKieske: our debian mirror hadn't updated for the past 3 days due to a leaked lockfile, my best guess is that mixing latest debian-security but slightly outdated debian could cause it | 13:51 |
| kevko | i have my own irc server where i am logged in 24h/d :) | 13:51 |
| kevko | thelounge ..very nice thing :) | 13:51 |
| fungi | if something in debian-security started insisting on libssl 3.0.11 | 13:52 |
| SvenKieske | kevko: yeah tried to set that up as well and failed, well ran out of patience :D in the mean time I can go look at https://meetings.opendev.org/irclogs/%23openstack-kolla/%23openstack-kolla.2023-10-10.log.html | 13:52 |
| fungi | i just run weechat in tmux on a vm in an openstack cloud provider, and attach remotely to it via mosh | 13:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: OpenSearch migration - prune old Kibana indices https://review.opendev.org/c/openstack/kolla-ansible/+/897667 | 13:53 |
| SvenKieske | fungi: I just read about your setup over in #opendev :D might be worth a blogpost | 13:53 |
| kevko | SvenKieske: really ? i just ran docker run :D | 13:53 |
| kevko | 30 sec ? :D | 13:53 |
| kevko | i have it also on mobile ..so | 13:54 |
| SvenKieske | well I have cert based auth setup and it was a pain figuring out how to make that work, I gave up after 5 minutes :P | 13:54 |
| kevko | i like console apps ..but better to have android app on mobile :) | 13:54 |
| SvenKieske | I actually didn't know weechat is OSS, I thought it's a giant china data hoarding app :D | 13:55 |
| kevko | SvenKieske: https://irc.kevko.ultimum.cloud :) me too | 13:55 |
| fungi | i'm using a terminal on my phone, yep. granted my phone runs a debian derivative with a mainline linux kernel, not android | 13:55 |
| fungi | SvenKieske: weechat != wechat | 13:55 |
| SvenKieske | ah, that's where my confusion stems from | 13:56 |
| fungi | and yeah, i use cert-based auth to oftc with it, no problem | 13:56 |
| kevko | btw - kolla toolbox -> install ca_certificates -> depends openssl -> depends libssl3 -> fail | 13:56 |
| fungi | okay, so it's probably newer ca_certificates in debian-security wanting newer openssl than the mirror had | 13:56 |
| fungi | nope. ca-certificates in bookworm depends on openssl>=1.1.1 still | 13:58 |
| fungi | and there's no newer ca-certificates package for bookworm in debian-security | 13:58 |
| SvenKieske | I was just about to ask why do the certs themselves depend on a certain patch level openssl version | 13:59 |
| fungi | something must have changed around this, because 3.0.11 has been in bookworm for two weeks already according to https://packages.qa.debian.org/o/openssl.html | 14:00 |
| fungi | as far as the stale mirror, it looks like the problem coincides with the debian 12.2 point release, the influx of packages caused mirroring to take so long that it exceeded our 90-minute timeout on the script and reprepro got killed, leaving a stale lockfile in its wake | 14:07 |
| SvenKieske | sounds like a job for a (new?) cleanupscript in case a timeout is reached :) | 14:10 |
| fungi | well, we might want to increase the timeout | 14:18 |
| fungi | problem is, if reprepro is killed partway through, then it has to rescan all the local copies of packages, so it's preferable not to kill it before it's done | 14:19 |
| fungi | especially since those "local" copies aren't really local, they're in a network filesystem | 14:20 |
| SvenKieske | the thing is: every process managing tool (so: whatever starts reprepro here) needs to clean up after the process it managed dies. if it does not do that or can not do that, it's a bad process manager and should not be used for that or needs to be enhanced to do that. no? | 14:26 |
| SvenKieske | are these plain old cron jobs, or what is the process for the mirroring? | 14:27 |
| SvenKieske | so no matter the timeouts chosen, there will be a case where something needs to clean up stale data, like pid files or whatever. most of the time the system service manager does this stuff, at least somewhat reliable. | 14:30 |
| opendevreview | Pedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Fix the docker container dimensions comparison for short notation https://review.opendev.org/c/openstack/kolla-ansible/+/886500 | 14:32 |
| SvenKieske | mnasiadka: seems the problem is related to ansible-core 2.14.11? https://github.com/ansible/ansible/issues/81945 | 14:45 |
| mnasiadka | SvenKieske: well, I don't know the person that raised this issue :) | 14:46 |
| SvenKieske | but they tested it, and it seems to work with 2.4.10, so I just wanted to try a new patch and see if this is correct. | 14:47 |
| mnasiadka | sure, why not | 14:47 |
| fungi | SvenKieske: plain old cron jobs, but the timeouts are set inside a script so the script could do cleanup tasks | 14:50 |
| opendevreview | Sven Kieske proposed openstack/kolla master: fix broken ansible-core version https://review.opendev.org/c/openstack/kolla/+/897843 | 14:51 |
| fungi | SvenKieske: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/reprepro/files/reprepro-mirror-update | 14:53 |
| SvenKieske | there are even cleanupsteps already mentioned :) | 14:54 |
| fungi | basically each one of those $REPREPRO command lines gets the 90 minute timeout applied | 14:54 |
| fungi | well, that's not cleaning up from an abnormal termination, that's our delayed package removal i mentioned earlier, cleaning up packages which are no longer referenced in the indices is done on a delay so that we don't yank them out from under running jobs that may try to install them | 14:55 |
| fungi | mirror update is done. timestamp is current now: https://static.opendev.org/mirror/debian/timestamp.txt | 15:01 |
| SvenKieske | ty | 15:01 |
| SvenKieske | would it make sense to alert on very stale mirrors? (I don't know the alerting/monitoring infrastructure, so this question might not make any sense). | 15:02 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Fix condition for enabling disable-selinux element https://review.opendev.org/c/openstack/kayobe/+/897849 | 15:25 |
| SvenKieske | mnasiadka: I have a bad feeling this "fix" in 2.14.11 might be related, but really not sure yet: https://github.com/ansible/ansible/pull/81644 | 15:27 |
| mnasiadka | SvenKieske: maybe again they "fixed" something we used :D | 15:27 |
| SvenKieske | that is a performance fix to fix jinja plugin caching issues, so maybe something stopped being rendered late enough for us | 15:28 |
| SvenKieske | just a wild guess so far | 15:28 |
| SvenKieske | when can we start porting everything to https://www.jetporch.com/ ? :D | 15:29 |
| mnasiadka | SvenKieske: once things work there ;) | 15:31 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897809 | 15:33 |
| fungi | SvenKieske: mirroring can fall behind for a number of reasons, and we have a dashboard for mirror updates (and other afs details) here: https://grafana.opendev.org/d/9871b26303/afs | 15:38 |
| fungi | looks like opensuse mirroring is 4 days behind | 15:38 |
| SvenKieske | that's why I asked for alerting, having built monitoring solutions for datacenters myself for some years I know dashboards are not really that helpful because you tend to get too many dashboards to really look at them ;) | 15:39 |
| fungi | feel free to set up some alerting on that for yourself. the opendev sysadmins don't want pager duty | 15:40 |
| fungi | this is all best-effort, anyone is welcome to help out | 15:40 |
| SvenKieske | I can understand that :) | 15:40 |
| fungi | https://static.opendev.org/mirror/logs/rsync-mirrors/opensuse.log says the mirror we're rsyncing from isn't reachable or times out connecting, probably needs switching | 15:42 |
| SvenKieske | I just wasn't aware if any alerting is in place for anything, that's why I asked. It wasn't meant to demand anything from anyone. I just wanted to understand better how everything works together. | 15:43 |
| fungi | the end of https://static.opendev.org/mirror/logs/reprepro/debian.log.1 shows the errors we started getting after a timeout killed the debian 12.2 bulk update | 15:44 |
| fungi | the script could probably be improved to log a message when a timeout kills something | 15:44 |
| SvenKieske | the clarkson mirror has an issue tracker here: https://github.com/COSI-Lab/Mirror/issues at least the site still claims to mirror opensuse | 15:45 |
| fungi | possible something happened to their rsyncd | 15:45 |
| fungi | though i can reach it from home | 15:52 |
| fungi | looks like i can reach it from our mirror updater too, so maybe they've fixed it since the last try | 15:54 |
| fungi | next try will be at 18:06 utc (in a little over 2 hours), so i'll check it again after the opendev sysadmins meeting | 15:55 |
| *** darmach0 is now known as darmach | 18:23 | |
| fungi | and opensuse mirroring did catch back up on its own, so clarkson must have fixed their server in the meantime | 18:56 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Remove disable-selinux from default DIB elements https://review.opendev.org/c/openstack/kayobe/+/897866 | 19:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897809 | 19:08 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!