| fungi | #status log Some changes and pipelines have been stuck for the past 8 hours due to an upgrade-related Zuul bug, fix is in flight at https://review.opendev.org/890026 | 15:52 |
|---|---|---|
| opendevstatus | fungi: finished logging | 15:53 |
| corvus | fungi: i think we can restart the schedulers now with that fix. you want to do that or shall i? | 16:31 |
| fungi | corvus: i can. how have you been doing it in the past? edited restart playbook that limits to just the scheduler servers? | 16:35 |
| corvus | fungi: for this -- i would probably just log into each of the 2 servers in turn and down/pull/up each of the schedulers -- there's only two :) | 16:36 |
| fungi | fair, i can do that now | 16:37 |
| fungi | i'm downing the containers on zuul01 | 16:38 |
| fungi | no need to graceful stop them first, right? | 16:38 |
| corvus | i think the down will send them a signal to stop | 16:38 |
| fungi | cool | 16:40 |
| fungi | downed and pulling in progress now | 16:40 |
| fungi | zuuld processes are still running | 16:41 |
| fungi | i guess i need to wait until they clean up? | 16:41 |
| corvus | yeah, should be relatively quick | 16:41 |
| fungi | quay.io/zuul-ci/zuul-scheduler latest 9f9a55134eb0 55 minutes ago 719MB | 16:42 |
| corvus | fungi: oh it is also probably worth doing the same for zuul-web; this change might affect that | 16:42 |
| fungi | i did all the containers on zuul01 | 16:42 |
| fungi | a full docker-compose down | 16:43 |
| corvus | fungi: zuul-web and zuul-fingergw have a separate docker-compose from zuul-scheduler | 16:43 |
| corvus | /etc/zuul-web/docker-compose.yaml | 16:44 |
| fungi | aha, right | 16:44 |
| fungi | forgot we had separate compose files | 16:44 |
| fungi | web and fingergw are down and have new images pulled too | 16:45 |
| fungi | and no zuuld processes running, so i'll up -d them | 16:45 |
| fungi | presumably i need to tail the debug logs to see when they're done starting up before i take down the containers on zuul02? | 16:46 |
| corvus | fungi: or check https://zuul.opendev.org/components till it's all green | 16:47 |
| fungi | oh, i didn't realize that was quite so real-time. awesome | 16:47 |
| corvus | yep; it doesn't auto-refresh, so you'll have to ctrl-r but it is real-time at time of render | 16:49 |
| fungi | and all green. starting on 02 now | 16:50 |
| fungi | openstack tenant's gate pipeline cleared out and stuff has started moving again in check as well | 16:54 |
| corvus | fungi: looks like things are unstuck now? | 16:54 |
| fungi | yep! | 16:54 |
| fungi | okay, both schedulers are back to running, now on 8.3.2.dev73 596da2d93 | 16:56 |
| fungi | #status log Zuul schedulers have been updated to fixed images and everything's moving normally again | 16:56 |
| opendevstatus | fungi: finished logging | 16:56 |
| fungi | thanks for all the help corvus! | 16:56 |
| corvus | yw :) | 17:02 |
| corvus | i'm going to restart all of zuul now to pick up 883318 | 18:34 |
| fungi | thanks! | 19:16 |
| fungi | that looks like an efficiency improvement we've been wanting for years | 19:16 |
| corvus | yeah i hope to do more with it later, but hopefully that's a good start | 19:41 |
| fungi | seems like it may result in a pretty good throughput improvement for dependent pipelines on rainy days | 19:42 |
| corvus | #status log restarted all of zuul on 6c0ffe565f1d0025ccee08a697cc73b4594942e5 | 19:54 |
| opendevstatus | corvus: finished logging | 19:54 |
| * fungi rubs his hands together | 19:54 | |
| corvus | now we just need some jobs to run :) | 19:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!