| stevebaker | hey all, there seems to be a problem with this fedora mirror https://zuul.opendev.org/t/openstack/build/a49e22e39bd94e098836a6dd46ad386c/log/logs/fedora_build-succeeds.FAIL.log#1021 | 02:43 |
|---|---|---|
| stevebaker | http://mirror-int.iad.rax.opendev.org | 02:43 |
| fungi | stevebaker: i've observed the same, we've pulled from the source mirror at least several times since i saw that error, so if it persists much longer we might have to switch to syncing from somewhere else | 02:46 |
| stevebaker | fungi: I get a "cannot resolve host" on that name | 02:48 |
| stevebaker | hmm, but a 404 in the logs | 02:48 |
| fungi | the mirror-int addresses are rfc 1918, drop the -int part to reach the public address on those | 02:49 |
| fungi | stevebaker: according to https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/fedora-mirror-update#L39 we're pulling that state from mirrors.mit.edu, if there's a different rsync mirror we should be pulling from instead of that one we can switch | 02:50 |
| fungi | i was hoping they'd have fixed it by now | 02:50 |
| stevebaker | fungi: I see, thanks. I'll keep an eye on it | 02:51 |
| fungi | keep in mind that for our distro package mirrors they're all backed by a shared network filesystem, so if you see an error like that on one of our mirrors you'll generally see it on all of them | 02:52 |
| stevebaker | ok | 02:52 |
| fungi | and we've been successfully pulling from the source mirror at mirrors.mit.edu every couple of hours, so the broken state seems to be getting copied from there | 02:53 |
| fungi | no idea whether wherever mit pulls from is similarly broken | 02:53 |
| fungi | if so, we might get the same errors even if we switched to a different rsync source | 02:54 |
| stevebaker | yeah the 404s are coming from this mostly empty dir http://mirrors.mit.edu/fedora/linux/updates/34/Everything/x86_64/repodata/ | 02:54 |
| stevebaker | fungi: curling this from your rsync host might give the best mirror potential hosts https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-34&arch=x86_64 | 02:57 |
| fungi | i'll take a look | 02:59 |
| stevebaker | also fedora 34 is old, we should probably update that dib job | 02:59 |
| fungi | mmm, the mirrorlist doesn't seem to indicate which ones have rsync support | 03:00 |
| fungi | just http and https | 03:00 |
| stevebaker | this may be some fedora-34 eol thing. This looks fine | 03:01 |
| stevebaker | http://mirrors.mit.edu/fedora/linux/updates/36/Everything/x86_64/repodata/ | 03:01 |
| stevebaker | ianw: my fix for the dib gate is blocked by another dib CI issue ^^ https://review.opendev.org/c/openstack/diskimage-builder/+/811048 | 03:02 |
| fungi | http://download-ib01.fedoraproject.org/pub/fedora/linux//releases/34/Everything/x86_64/os/repodata/ seems to still have content, so i don't expect it's been removed from the primaries | 03:03 |
| fungi | also the mit mirror seems to have at least some indices for older releases like f33 and f32 | 03:04 |
| fungi | looks like i switched to the mit mirror a year ago with https://review.opendev.org/746587 because pubmirror[12].math.uh.edu ceased updating their (at the time f31) copy | 03:06 |
| fungi | we could switch back to it, http://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/34/Everything/x86_64/repodata/ has indices | 03:07 |
| fungi | i've pushed up a revert as https://review.opendev.org/811832 but will try it manually first | 03:10 |
| fungi | stevebaker: ianw is on vacation for a few more days, i think | 03:11 |
| stevebaker | ok | 03:11 |
| fungi | anyway, i've got a refresh running with the revert manually applied, no idea how long it may take to complete though, switching source mirrors often results in having to retransmit the entire set | 03:15 |
| fungi | or at least large swaths of it | 03:16 |
| *** ykarel_ is now known as ykarel | 06:07 | |
| *** ysandeep|out is now known as ysandeep | 06:36 | |
| *** jpodivin is now known as jpodivin|ruck | 06:36 | |
| *** jpena|off is now known as jpena | 07:00 | |
| *** elodilles_pto is now known as elodilles | 07:12 | |
| *** ykarel is now known as ykarel|lunch | 09:13 | |
| *** ykarel|lunch is now known as ykarel | 10:05 | |
| *** jcapitao is now known as jcapitao_lunch | 10:39 | |
| *** rlandy is now known as rlandy|ruck | 10:39 | |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** tobberydberg_ is now known as tobberydberg | 11:51 | |
| *** jcapitao_lunch is now known as jcapitao | 12:09 | |
| *** jpena|lunch is now known as jpena | 12:16 | |
| *** bhagyashris is now known as bhagyashris|rover | 12:43 | |
| mnaser | im confused | 14:00 |
| mnaser | i dont think fedora 34 is eol? isn't it the latest release? lol | 14:01 |
| fungi | mnaser: yeah, not sure what the cause was for mit's mirror being incomplete so we're in the process of syncing from uh | 14:16 |
| fungi | it's settling now and getting ready to update the public volume replicas starting in roughly 10 minutes | 14:17 |
| fungi | but given it ran for most of the night transferring data, i expect the volume release to take hours as well | 14:18 |
| *** bhagyashris_ is now known as bhagyashris|rover | 14:19 | |
| clarkb | mnaser: fedora 35 is just releasing now aiui. So ya 34 is the current release unless you do betas | 14:26 |
| fungi | https://static.opendev.org/mirror/fedora/timestamp.txt | 14:28 |
| fungi | i think maybe it's updated now, i'll recheck one of my failing changes | 14:29 |
| fungi | i'm going to run a second manual sync once this one finishes though, just to make sure we're completely caught up | 14:29 |
| clarkb | ++ | 14:30 |
| fungi | looks like it's using slightly more of the volume after the source switch, but not significantly, and there's still plenty of headroom there: https://grafana.opendev.org/d/T5zTt6PGk/afs?viewPanel=29&orgId=1&from=now-24h&to=now | 14:34 |
| fungi | also the afs02.dfw replica is still releasing, and will be for a while, but looks like afs01.dfw is what the mirrors are serving and it released quickly since it's local | 14:35 |
| fungi | stevebaker: okay, as of moments ago i've got passing f34 jobs again | 14:53 |
| *** promethe- is now known as prometheanfire | 15:11 | |
| *** prometheanfire is now known as Guest1415 | 15:12 | |
| *** chandankumar is now known as raukadah | 15:21 | |
| *** Guest1415 is now known as prometheanfire | 15:35 | |
| *** jpena is now known as jpena|off | 15:52 | |
| gmann | fungi: mnaser this is to add stable/xena periodic jobs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/810999 | 16:09 |
| *** ysandeep is now known as ysandeep|dinner | 16:10 | |
| fungi | gmann: thanks, i'll take a look here shortly after the zuul restart | 16:17 |
| *** ysandeep|dinner is now known as ysandeep|out | 17:09 | |
| rlandy|ruck | hello | 17:42 |
| rlandy|ruck | we have failing tox jobs everywhere | 17:42 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1945682 | 17:42 |
| rlandy|ruck | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml#L38 | 17:42 |
| rlandy|ruck | from this code ^^ | 17:43 |
| rlandy|ruck | fungi: hi - ^^ any thoughts here? | 17:44 |
| clarkb | rlandy|ruck: it always always always :) helps if you can link directly to a failure | 17:44 |
| fungi | rlandy|ruck: checking, we did just merge some fixes for that | 17:44 |
| clarkb | is https://zuul.opendev.org/t/openstack/build/f8c23147316040beb63b29d8885fa7eb an example of it? | 17:44 |
| rlandy|ruck | yes - that is it | 17:45 |
| rlandy|ruck | clarkb: noted (about the linking) :) | 17:45 |
| clarkb | I'm going to guess this is some corner case triggred by the tripleo project tox.ini fiels because other projects are running tox successfully | 17:45 |
| fungi | specifically we fixed an instance where the tox_extra_args role var wasn't being applied to the tox invocations in the siblings tasks | 17:45 |
| rlandy|ruck | possibly - it is hitting tripleo-related repos | 17:46 |
| clarkb | [line 3]: 'using tox-3.24.4 from /home/zuul/src/opendev.org/openstack/tripleo-quickstart-extras/.tox/.tox/lib/python3.8/site-packages/tox/__init__.py (pid 2919)\n' is what it doesn't like | 17:46 |
| fungi | oh, yep, looks like its failing in the tox showconfig output parser | 17:46 |
| clarkb | I wonder if https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/tox.ini#L5-L6 causes that | 17:47 |
| fungi | i wonder what leads to the "using tox-..." line getting added to the output | 17:47 |
| clarkb | the .tox/.tox venv is the boostrapping venv that tox creates when you use requires | 17:47 |
| clarkb | fungi: its that requires block I think. It creates a meta venv that it actually runs tox and virtualenv and so on out of | 17:48 |
| clarkb | This allows you to bootstrap a newer tox than is present on the local system using tox (among other things) | 17:48 |
| fungi | ahh, yeah i don't think we have any testing of tox.requires in the jobs which exercise the tox role | 17:49 |
| fungi | according to zbr there's a *very* recent fix to tox which tries to guarantee the stdout from tox --showconfig will be parseable ini data, but we need to be able to parse output from older tox versions as well | 17:51 |
| fungi | clarkb: rlandy|ruck: i've got a revert on the way, when gerrit finally accepts my push | 17:56 |
| rlandy|ruck | fungi: thanks for the quick action on that | 17:57 |
| fungi | it's https://review.opendev.org/812001 | 17:58 |
| rlandy|ruck | saw https://review.opendev.org/c/zuul/zuul-jobs/+/812001/ merged - thank you! | 18:26 |
| fungi | rlandy|ruck: yw, i'll hit you up once i have a fixed revert of the revert passing a new test (once i come up with a good way to exercise it) and maybe you can depends-on from something to verify it's clean for a tripleo change | 18:50 |
| rlandy|ruck | sure | 18:51 |
| fungi | okay, i see the problem, and i agree it's the tox.requires triggering it | 19:03 |
| fungi | the way the siblings lib tries to parse tox --showconfig -vv is by discarding lines up until it sees one starting with [ since that would be a section heading in the ini output | 19:04 |
| fungi | except the verbose output in the nested pip install going on when there's tox.requires specified also prefixes some lines of output with what are (i think?) pids like "[176766] /some/path/to/a/command --with --options" | 19:05 |
| fungi | i bet if it were to discard until it reaches a line which begins with a [ AND ends with a ] we'll be set | 19:06 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Add stable/xena to periodic-stable templates https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/810999 | 19:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!