| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Tobias Henkel: [zuul/zuul] 816072: Increase load_multiplier in tests https://review.opendev.org/c/zuul/zuul/+/816072 | 00:20 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Tobias Henkel: [zuul/zuul] 816072: Increase load_multiplier in tests https://review.opendev.org/c/zuul/zuul/+/816072 | 01:55 | |
| @jim:acmegating.com | we seem to mostly be looking at dockerhub failures at this point | 02:18 |
|---|---|---|
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 816073: Cancel stats election on shutdown https://review.opendev.org/c/zuul/zuul/+/816073 | 06:28 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 815450: Create bundle items during queue deserialization https://review.opendev.org/c/zuul/zuul/+/815450 | 06:33 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 815495: Fix Gerrit change (de-)serialization https://review.opendev.org/c/zuul/zuul/+/815495 | 06:45 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815565: Remove unecessary assignment in re-enqueue https://review.opendev.org/c/zuul/zuul/+/815565 | 06:45 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: | 06:52 | |
| - [zuul/zuul] 815616: Only reset the pipeline state if needed https://review.opendev.org/c/zuul/zuul/+/815616 | ||
| - [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 06:52 | |
| - [zuul/zuul] 815744: Use a metaclass to deserialize event objects https://review.opendev.org/c/zuul/zuul/+/815744 | ||
| - [zuul/zuul] 815764: Add a pipeline change list object to ZK https://review.opendev.org/c/zuul/zuul/+/815764 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815916: Reduce use of OrderedDict in PipelineState https://review.opendev.org/c/zuul/zuul/+/815916 | 08:26 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Matthieu Huin https://matrix.to/#/@mhuin:matrix.org: [zuul/zuul] 735586: Zuul-web: Add authentication-realm attribute to tenants https://review.opendev.org/c/zuul/zuul/+/735586 | 09:25 | |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] 816088: Pin CacheControl to <0.12.7 https://review.opendev.org/c/zuul/zuul/+/816088 | 09:33 | |
| @tobias.henkel:matrix.org | corvus, Clark : this just broke our update today ^ | 09:33 |
| -@gerrit:opendev.org- Zuul merged on behalf of Matthieu Huin https://matrix.to/#/@mhuin:matrix.org: [zuul/zuul] 736968: zuul-web: add pipelines' manager, triggers data in status https://review.opendev.org/c/zuul/zuul/+/736968 | 09:34 | |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] 816088: Pin CacheControl to <0.12.7 https://review.opendev.org/c/zuul/zuul/+/816088 | 09:34 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Felix Edel: | 09:50 | |
| - [zuul/zuul] 760806: UI: Add actions and reducers to retrieve components https://review.opendev.org/c/zuul/zuul/+/760806 | ||
| - [zuul/zuul] 760807: UI: Add components page https://review.opendev.org/c/zuul/zuul/+/760807 | ||
| @tobias.henkel:matrix.org | corvus, Clark : fyi, I've created an upstream issue in cachecontrol: https://github.com/ionrock/cachecontrol/issues/263 | 10:06 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815917: Update Pipeline for symmetry https://review.opendev.org/c/zuul/zuul/+/815917 | 11:14 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 813255: Perform actual HTTP requests in gitlab tests https://review.opendev.org/c/zuul/zuul/+/813255 | 12:42 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 12:51 | |
| - [zuul/zuul] 813256: Add gitlab disable_connection_pool option https://review.opendev.org/c/zuul/zuul/+/813256 | ||
| - [zuul/zuul] 813289: Stop GerritWebServer in tests https://review.opendev.org/c/zuul/zuul/+/813289 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815979: Use activeContext instead of explicit _save calls https://review.opendev.org/c/zuul/zuul/+/815979 | 14:07 | |
| @jim:acmegating.com | tobiash: i went ahead and approved the cachecontrol change | 14:18 |
| @jim:acmegating.com | and hey, it looks like all the sos pipeline stuff merged, along with the components page; that will be handy :) | 14:19 |
| @jim:acmegating.com | as soon as i can focus a little better, i'll restart opendev | 14:19 |
| -@gerrit:opendev.org- Zuul merged on behalf of Tobias Henkel: [zuul/zuul] 816088: Pin CacheControl to <0.12.7 https://review.opendev.org/c/zuul/zuul/+/816088 | 15:33 | |
| @jim:acmegating.com | i'm restarting opendev's zuul, and the first thing i note is that we may want to remove some of the extra change cache debug log lines i added. there's a bunch of logs at startup as it adds 70k watches | 15:43 |
| @jim:acmegating.com | the second thing is i'd like to know why there are 70k watches | 15:43 |
| @jim:acmegating.com | erm, did the fonts on the status page change? | 15:54 |
| @jim:acmegating.com | there's something weird with the layout in the boxes too | 15:55 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 816094: Adjust spacing on status page https://review.opendev.org/c/zuul/zuul/+/816094 | 16:04 | |
| @jim:acmegating.com | that's a quick fix for the status page spacing | 16:05 |
| @jim:acmegating.com | i'm starting a second scheduler now | 16:06 |
| @jim:acmegating.com | the first thing that happens is 70k lines about adding elements to the change cache (sorry) | 16:06 |
| @jim:acmegating.com | it's loading the config right now | 16:09 |
| @jim:acmegating.com | (i wonder if we should add a new run state for priming config?) | 16:09 |
| @jim:acmegating.com | it is currently handling some gearman jobs (like status requests) which it shouldn't | 16:10 |
| @jim:acmegating.com | but that's not a long term problem; we'll be removing that functionality | 16:10 |
| @jim:acmegating.com | it's processing pipelines now | 16:16 |
| @jim:acmegating.com | apparently successfully | 16:16 |
| @tobias.henkel:matrix.org | yay :) | 16:17 |
| @jim:acmegating.com | it processed a completed build | 16:17 |
| @jim:acmegating.com | 2021-10-30 16:16:26,280 DEBUG zuul.Pipeline.openstack.check: Build <Build 6a8f03bb9ff0432bbee0698215fec9ff of nova-tox-functional-py39 voting:False> of <Change 0x7fa33e71f880 openstack/nova 806683,17> completed | 16:17 |
| @jim:acmegating.com | https://zuul.opendev.org/t/openstack/build/6a8f03bb9ff0432bbee0698215fec9ff | 16:18 |
| @jim:acmegating.com | the first build to be started by one scheduler and finished by another? :) | 16:18 |
| @jim:acmegating.com | just confirmed in the logs that 02 did not process the completion event, only 01 did | 16:20 |
| @jim:acmegating.com | not much happening now... saturday... :) | 16:22 |
| @jim:acmegating.com | 2021-10-30 16:24:05,903 DEBUG zuul.Scheduler: Skipping locked pipeline check in tenant openstack | 16:24 |
| @jim:acmegating.com | that's a good sign | 16:24 |
| @jim:acmegating.com | and on the other scheduler immediately afterwords: | 16:26 |
| @jim:acmegating.com | 2021-10-30 16:24:06,125 DEBUG zuul.Scheduler: Skipping locked pipeline gate in tenant openstack | 16:26 |
| @jim:acmegating.com | one thing we may want to think about: the schedulers can get into a pattern where one follows the other, so we might see one process a pipeline and then the other immediately process the same one. of course, the second one is fast since there are no events left to process. This may be no big deal and we can just leave it alone; or we could consider skipping the actual pipeline processing if there are no events. | 16:31 |
| @jim:acmegating.com | i think we should leave it, unless we find that it uses too much cpu time, then consider optimizing. | 16:32 |
| @tobias.henkel:matrix.org | Otherwise when having many pipelines this would reduce locking overhead consuderably | 16:37 |
| @tobias.henkel:matrix.org | * Otherwise when having many pipelines this would reduce locking overhead considerably | 16:37 |
| @jim:acmegating.com | this error happened: https://paste.opendev.org/show/810298/ | 16:41 |
| @jim:acmegating.com | it's not critical; i wonder if that's a race on 2 schedulers | 16:41 |
| @jim:acmegating.com | only one of them logged that error; the other just noted it was deleting an entry from the cache | 16:42 |
| @tobias.henkel:matrix.org | Does the cleanup do locking? | 16:42 |
| @jim:acmegating.com | the general cleanup/connection cache maintenance does, so there was only one copy of that running at that time. | 16:48 |
| @jim:acmegating.com | i don't know why it had a cache entry with no data though. i don't see any cache removals near then. so i don't understand why that happened | 16:49 |
| @jim:acmegating.com | that was the only general cleanup that's run since we started, so it may have been cleaning up an old error in zk. | 16:58 |
| @jim:acmegating.com | (and that cache key was in zk before startup; it was not added after the restart) | 16:58 |
| @jim:acmegating.com | i think i'm going to leave this running as long as possible today and see if anything major breaks. we know that the branch cache is going to get out of date, but that shouldn't be a big deal for a relatively short period of time on a not-too-busy system. | 17:09 |
| @clarkb:matrix.org | corvus I guess it went well. Exciting. Can you expand a bit on the branch cache problem? I'm curious to know what we are missing there. | 18:01 |
| @jim:acmegating.com | Clark: yes, is here: https://review.opendev.org/815788 | 18:13 |
| @jim:acmegating.com | i still haven't dived into it yet, but that's next i think. | 18:13 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!