| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/952006 | 02:21 |
|---|---|---|
| *** elodilles_pto is now known as elodilles | 07:55 | |
| opendevreview | Merged openstack/project-config master: Add release branch creation permission https://review.opendev.org/c/openstack/project-config/+/952643 | 12:45 |
| *** cloudnull19 is now known as cloudnull1 | 13:56 | |
| frickler | fungi: corvus: there's some pretty old looking autoholds in the zuul tenant, are these still needed? | 18:13 |
| frickler | also, does zl support autoholds by now? | 18:14 |
| fungi | frickler: thanks for the heads up! i deleted mine just now | 18:14 |
| frickler | oh, also two in opendev | 18:15 |
| fungi | those don't look like any of mine at least | 18:15 |
| corvus | frickler: i'll delete those autoholds | 18:21 |
| corvus | and that's a good point, zl does not support autoholds... i'll put that at the top of the list | 18:23 |
| corvus | (strictly speaking, it's the executors that need to have support added; the launcher does understand holds, and manual node holds work) | 18:23 |
| frickler | not to derail the meeting progress: on nl05 I see 3 mentions for xenial in the 10d logs and 2191 for bionic. so the former could likely be considered irrelevant the later rather not | 19:30 |
| fungi | make sure they're not min-server boots and for actual jobs. knowing the jobs and projects would be good too | 19:35 |
| frickler | will have to dig deeper into how to grep that from the logs, then | 19:52 |
| fungi | mainly just thinking if they *are* in use (ooh, also what pipeline?) for more than periodic jobs ~nobody looks at then it would be good to know who to reach out to in order to let them know that this is going away if they don't help | 19:53 |
| frickler | ya, good points | 20:02 |
| fungi | if it's just (probably completely broken) periodic jobs then i really don't care | 20:07 |
| corvus | heh, NODE_ERROR results for those would be a net improvement :) | 20:08 |
| fungi | if there are changes getting (voting, another thing worth maybe trying to identify?) results then i might care, particularly if the jobs succeed (maybe harder to correlate) | 20:08 |
| corvus | i approved https://review.opendev.org/952712 to switch more tenants to niz | 20:26 |
| fungi | thanks! | 20:53 |
| fungi | though openstack-zuul-jobs-linters hit a post_failure on it, i wonder if one of our log upload targets is offline/broken again | 20:53 |
| corvus | there are many post_failures... i'll check the logs | 20:56 |
| corvus | "Upload swift logs to ovh_bhs" | 21:04 |
| corvus | thats the first one i found | 21:04 |
| corvus | and gra | 21:05 |
| corvus | 19:47 start time for the cluster of events | 21:06 |
| corvus | 77 events | 21:06 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Remove ovh log upload targets https://review.opendev.org/c/opendev/base-jobs/+/952793 | 21:07 |
| corvus | infra-root: ^ | 21:08 |
| fungi | corvus: i approved it to resolve the current situation asap, but in prior changes like https://review.opendev.org/c/opendev/base-jobs/+/944152 we removed providers from playbooks/base/post-logs.yaml instead, does removing them from the secrets actually work? | 21:21 |
| corvus | oh, hrm, for some reason i thought that was input to the selection; we better do it the other way | 21:22 |
| fungi | can you also similarly invert it for base-test to make testing whether it's safe to revert easier? | 21:23 |
| fungi | sadly, we have done this... a lot (if you look at the git history) | 21:23 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Remove ovh log upload targets https://review.opendev.org/c/opendev/base-jobs/+/952793 | 21:24 |
| corvus | done | 21:24 |
| corvus | and maybe at some point, i'll look into updating the job to work the way i apparently think it works :) | 21:24 |
| fungi | https://public-cloud.status-ovhcloud.com/incidents/d6bbntp2x68q | 21:26 |
| fungi | that seems to likely be what we're experiencing | 21:26 |
| fungi | or maybe not | 21:27 |
| fungi | the dates on that are oldish | 21:27 |
| fungi | https://public-cloud.status-ovhcloud.com/ lists block storage maintenance in progress but nothing else for object storage | 21:28 |
| fungi | though further down it does show yellow warning triangles for object storage in bhs and gra | 21:29 |
| corvus | it got hit with the problem it's fixing; we'll need to recheck it | 22:01 |
| corvus | also, zl01 got stuck in a loop; i needed to restart it. i got a copy of the traceback, so i can troubleshoot it. | 22:02 |
| corvus | 'Connection to compute.bhs1.cloud.ovh.net timed out. (connect timeout=60.0)' | 22:03 |
| corvus | looks like the ovh issues are spreading | 22:03 |
| opendevreview | Merged opendev/base-jobs master: Drop gentoo-17-0-systemd nodeset https://review.opendev.org/c/opendev/base-jobs/+/946264 | 22:04 |
| fungi | strange that there's still no updates on their public cloud status page about any of this | 22:08 |
| corvus | i've removed my +w from 952712 because it's no longer a good time to merge that :) | 22:09 |
| fungi | then again it's like midnight in france now | 22:09 |
| opendevreview | Merged opendev/base-jobs master: Remove ovh log upload targets https://review.opendev.org/c/opendev/base-jobs/+/952793 | 22:30 |
| corvus | yay it ran the gauntlet | 22:32 |
| corvus | #status notice Zuul jobs reporting POST_FAILURE were due to an incident with one of our cloud providers; this provider has been temporarily disabled and changes can be rechecked | 22:35 |
| opendevstatus | corvus: sending notice | 22:35 |
| -opendevstatus- NOTICE: Zuul jobs reporting POST_FAILURE were due to an incident with one of our cloud providers; this provider has been temporarily disabled and changes can be rechecked | 22:36 | |
| opendevstatus | corvus: finished sending notice | 22:39 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!