| opendevreview | Merged openstack/diskimage-builder master: Fix manifest element with non-root user https://review.opendev.org/c/openstack/diskimage-builder/+/922385 | 01:05 |
|---|---|---|
| opendevreview | Merged opendev/system-config master: Update test arm64 mirror node https://review.opendev.org/c/opendev/system-config/+/933155 | 06:45 |
| frickler | clarkb: JayF: bonus question for the gerrit filter discussion (which is very helpful already): can I somehow filter for all repos where I'd be allowed to do a CR+2? (except crafting an explicit list) | 08:02 |
| opendevreview | yatin proposed openstack/project-config master: Use 2024.2 constraints in master translation jobs https://review.opendev.org/c/openstack/project-config/+/934402 | 08:10 |
| noonedeadpunk | clarkb: returning to story with timeouts we see in CI. I've catched that also on my local laptop recently. Though there's another place now which fails. The difference there, is that there's no redirect and URI goes directly to gitea. And timeout in task is 30 sec already | 08:23 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/cae61a36665d4c8bbb10a2d0d8c7ec5a | 08:23 |
| opendevreview | Karolina Kula proposed openstack/diskimage-builder master: WIP: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 08:44 |
| SvenKieske | mhm, was yesterdays post_failure/logging issue in CI resolved? I wonder if I can just recheck a couple of failures. | 08:52 |
| SvenKieske | frickler, any insight if the post_failure log issue is resolved? | 09:01 |
| frickler | SvenKieske: apparently there was some short issue at rackspace, so yes, should be fine to recheck things | 09:02 |
| SvenKieske | thx, will do | 09:03 |
| opendevreview | Ian Wienand proposed opendev/system-config master: backups: add retirement and purge lists https://review.opendev.org/c/opendev/system-config/+/933700 | 09:18 |
| ianw | ok i took corvus's advice and instrumented ansible to dump to a file | 11:08 |
| ianw | in fact it's worse than this, https://github.com/python/cpython/blob/main/Lib/http/client.py uses print() so I monkey-patched print() in that file to write to a file, which works well enough | 11:09 |
| ianw | anyway ... so I have the ansible uri: call instrumented to dump the headers and ... it's _exactly_ the same between running in the zuul executor container and a fedora:latest | 11:10 |
| ianw | which is kinda what was expected ... but still | 11:10 |
| ianw | I have recorded the responses and put it into https://github.com/readthedocs/readthedocs.org/issues/11753 | 11:11 |
| ianw | possibly i think someone could look at the CF-RAY response and debug it on the cloudflare side. maybe that will happen, or maybe we'll just never know | 11:12 |
| ianw | don't tell me i don't know how to have a fun friday night! | 11:12 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: Use "curl" to trigger rtd.org webhook with http basic auth https://review.opendev.org/c/zuul/zuul-jobs/+/934243 | 11:19 |
| frickler | ianw: fun. just guessing, maybe the fingerprinting is happening on the tcp/tls level | 11:24 |
| ianw | yeah - i mean in a container same kernel - and i think basically the same ssl if using cryptography .whls | 11:25 |
| ianw | i dunno - if you have time try and replicate it :) i have that many containers going ... i don't think i made a mistake but never know | 11:25 |
| ianw | i thought i was onto something with the UA but that's not it | 11:26 |
| opendevreview | Ian Wienand proposed opendev/system-config master: backups: add retirement and purge lists https://review.opendev.org/c/opendev/system-config/+/933700 | 11:31 |
| Clark[m] | frickler: I'm not aware of a way to do that unfortunately and have just been listing projects. You can use globs for project names to list all opendev/.* and zuul/.* Repos for example though | 14:16 |
| Clark[m] | noonedeadpunk: can you link to where the failure occurs? I probably won't have time to debug today as I'm supposed to be out until Tuesday but having concrete direct data is helpful. Note there were gitea backend issues early this week that may correlated depending on timestamps. | 14:17 |
| noonedeadpunk | it fails here now: https://opendev.org/openstack/openstack-ansible-os_rally/src/branch/master/tasks/main.yml#L53-L59 | 14:18 |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Release team meeting does not follow DST changes https://review.opendev.org/c/opendev/irc-meetings/+/934432 | 14:18 |
| noonedeadpunk | and I've also proposed a patch to add retries there just in case | 14:19 |
| noonedeadpunk | but like - 30 sec timeout is quite a high bar I'd say... | 14:19 |
| Clark[m] | I meant can you link to the failure in the log rather than just to the failed job | 14:20 |
| Clark[m] | Also does that job record the Ansible logs twice? Maybe those are for different invocations? | 14:24 |
| fungi | i'm trying a keyword search through the log for "timeout" but it shows up in lots of non-error-related lines | 14:25 |
| Clark[m] | logs/ansible and logs/openstack/ansible-logging are the two locations fwiw | 14:26 |
| fungi | giving up. a link to the actual error would help yes | 14:26 |
| opendevreview | Merged opendev/irc-meetings master: Release team meeting does not follow DST changes https://review.opendev.org/c/opendev/irc-meetings/+/934432 | 14:30 |
| fungi | ianw: the other thing that *might* be useful to include in that rtd issue is that the behavior change started sometime between 2024-09-19T11:08:21Z (our last successful build) and 2024-09-23T21:32:29Z (the first observed failure for a previously working project) | 14:31 |
| fungi | i don't know whether that corresponds with a change in something client-side or server-side, but if it's the latter then it might narrow the window on possibilities | 14:32 |
| Clark[m] | fungi: do you know if the LE job successfully nooped the acme.sh install during the daily runs last night? I can check in a little over an hour after the school run | 14:37 |
| fungi | Clark[m]: it did in the deploy pipeline job when the change originally merged (we just didn't get a checkmark in the zuul builds list because of the rackspace outage) | 14:38 |
| fungi | i would assume the periodic run isn't any different, but i'll look shortly | 14:38 |
| fungi | the periodic run did report success | 14:41 |
| fungi | TASK [letsencrypt-acme-sh-install : Check status of acme.sh script] | 14:41 |
| fungi | ok: [codesearch01.opendev.org] => { | 14:41 |
| fungi | "changed": false, | 14:41 |
| fungi | [...] | 14:41 |
| fungi | "checksum": "5c298a2bd5f90635aef8d013b02b25f34027ad0cb2cef2bdca68f3d13b931216", | 14:41 |
| fungi | TASK [letsencrypt-acme-sh-install : Install acme.sh client] | 14:42 |
| fungi | skipping: [codesearch01.opendev.org] => { | 14:42 |
| fungi | "changed": false, | 14:43 |
| fungi | "false_condition": "not acme_sh_stat.stat.exists or acme_sh_stat.stat.checksum != \"5c298a2bd5f90635aef8d013b02b25f34027ad0cb2cef2bdca68f3d13b931216\"", | 14:43 |
| fungi | "skip_reason": "Conditional result was False" | 14:43 |
| fungi | looks like it did what we wanted gain, like in the deploy | 14:44 |
| fungi | s/gain/again/ | 14:44 |
| Clark[m] | Excellent now we can fill disks and have working ssl certs | 14:44 |
| corvus | just what we've always wanted | 14:45 |
| fungi | sounds like a great way to spend a friday | 14:45 |
| timburke | could i get a node hold on https://zuul.opendev.org/t/openstack/stream/cf4e77d1c21c4c65b4d522cf709ea481?logfile=console.log ? seems to be hung (no new messages for ~15min) and i want to figure out why | 16:05 |
| fungi | timburke: working on it | 16:25 |
| timburke | fungi, thanks! i'm guessing you'll want an ssh key: https://gist.githubusercontent.com/tipabu/d5f2319c19b2672143b9c153f6a67ebd/raw/0909646bc09ffa2b156d63163a5f4cc506899fd9/gistfile1.txt | 16:26 |
| fungi | timburke: yep, autohold is set so once that build fails i should be able to give you access. i'll see if i can track down the node ip address and get your key on sooner though, just a minute | 16:29 |
| fungi | timburke: ssh root@65.17.193.73 | 16:40 |
| timburke | fungi, thanks! | 16:49 |
| fungi | any time! | 16:55 |
| timburke | fungi, i think i've got all i need -- thanks again! | 17:47 |
| timburke | if you're curious, https://github.com/eventlet/eventlet/issues/989 came out of it | 17:48 |
| fungi | oh neat! | 17:55 |
| fungi | i hadn't seen your | 17:57 |
| fungi | python-stack-xray before (or i forgot about it at least) | 17:57 |
| fungi | anyway, the held node has been released for deletion | 17:58 |
| timburke | 👍 | 18:02 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Switch logo color in docs pages to dark blue https://review.opendev.org/c/zuul/zuul-jobs/+/934453 | 19:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!