Thursday, 2021-10-28

@iwienand:matrix.orgwhen i just run a curl, I get "curl: (6) getaddrinfo() thread failed to start"00:06
@iwienand:matrix.orgit appears to be this -> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=99577700:12
@iwienand:matrix.orgfurther confirmed by https://review.opendev.org/c/openstack/diskimage-builder/+/815763 working to build f3400:13
@iwienand:matrix.orgfedora is f*ing annoying, but i guess fundamentally these are real problems that it's better to sort out now one-by-one, than in several years at major LTS updates00:13
@clarkb:matrix.orgI think the issue is that we're running into them and have nothing to do with the OS development :)00:14
@jim:acmegating.comianw: what are you doing in 815763 to make it work?00:15
@jim:acmegating.comi feel like i'm missing something00:15
@iwienand:matrix.orgcorvus: nothing -- that's using bullseye native podman to build f34, so it shows that it's not a generic problem00:16
@iwienand:matrix.orgusing the custom .debs from that bug, the f35 containers work 00:16
@jim:acmegating.comoh, i think i was missing the f34/f35 bit.  sorry and thanks00:17
@jim:acmegating.com(my understanding now: f34 works with bullseye; that means that the latest problem is f35 specific and matches that bug)00:18
@jim:acmegating.com(and the nodepool bullseye change should be okay to merge?)00:18
@iwienand:matrix.orgthat's correct.  the only problem is that f34 ships a initramfs without drivers to boot on Xen (at least, same issue might also be causing boot issues on some of our other non-xen providers)00:19
@iwienand:matrix.orgso it builds, but it's not a particularly useful output00:19
@iwienand:matrix.orgso, it is safe i think to merge the nodepool "use bullsye podman" change.  but practically we either need to keep debugging that issue, or move to f3500:22
@iwienand:matrix.orgmoving to f35 seems the most practical solution.  however that now means we need this fixed podman for the clone3() issue00:24
@iwienand:matrix.org(or figure out the custom seccomp flags)00:25
@iwienand:matrix.org... reading more00:27
@iwienand:matrix.org> I could get convinced if the issue was fixable by just upading the seccomp.json policy file.00:27
> Sadly, that's not the case.
@iwienand:matrix.orgdifferent, but related bug -> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=99445100:27
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 815766: Dockerfile: install podman from unstable https://review.opendev.org/c/zuul/nodepool/+/81576601:02
-@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 815753: Use bullseye podman in container https://review.opendev.org/c/zuul/nodepool/+/81575302:08
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com:03:58
- [zuul/zuul] 815752: Rely on bullseye image for skopeo in container image https://review.opendev.org/c/zuul/zuul/+/815752
- [zuul/zuul] 813552: Remove Worker class https://review.opendev.org/c/zuul/zuul/+/813552
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 813805: Remove project pipeline config from queue item https://review.opendev.org/c/zuul/zuul/+/81380504:04
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 815766: Dockerfile: install podman from unstable https://review.opendev.org/c/zuul/nodepool/+/81576604:24
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815764: Add a pipeline change list object to ZK https://review.opendev.org/c/zuul/zuul/+/81576405:34
@iwienand:matrix.orgdib-nodepool-functional-openstack-fedora-35-containerfile-src: **success** 🎉05:40
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 815766: Dockerfile: install podman from unstable https://review.opendev.org/c/zuul/nodepool/+/81576605:42
-@gerrit:opendev.org- Simon Westphahl proposed:08:31
- [zuul/zuul] 815278: DNM: execute tests with two schedulers https://review.opendev.org/c/zuul/zuul/+/815278
- [zuul/zuul] 815787: Refresh pipelines in tests when settled https://review.opendev.org/c/zuul/zuul/+/815787
- [zuul/zuul] 815788: wip: Allow refreshing project branches https://review.opendev.org/c/zuul/zuul/+/815788
-@gerrit:opendev.org- Simon Westphahl proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815764: Add a pipeline change list object to ZK https://review.opendev.org/c/zuul/zuul/+/81576408:31
-@gerrit:opendev.org- Simon Westphahl proposed:08:32
- [zuul/zuul] 815787: Refresh pipelines in tests when settled https://review.opendev.org/c/zuul/zuul/+/815787
- [zuul/zuul] 815788: wip: Allow refreshing project branches https://review.opendev.org/c/zuul/zuul/+/815788
- [zuul/zuul] 815278: DNM: execute tests with two schedulers https://review.opendev.org/c/zuul/zuul/+/815278
-@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 815813: Increase log level of messages https://review.opendev.org/c/zuul/nodepool/+/81581309:40
-@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 815813: Increase log level of messages https://review.opendev.org/c/zuul/nodepool/+/81581309:46
-@gerrit:opendev.org- Simon Westphahl proposed:10:14
- [zuul/zuul] 815787: Refresh pipelines in tests when settled https://review.opendev.org/c/zuul/zuul/+/815787
- [zuul/zuul] 815788: wip: Allow refreshing project branches https://review.opendev.org/c/zuul/zuul/+/815788
- [zuul/zuul] 815278: DNM: execute tests with two schedulers https://review.opendev.org/c/zuul/zuul/+/815278
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 815844: Provide zstat version when updating Node Request in ZooKeeper https://review.opendev.org/c/zuul/zuul/+/81584413:21
@westphahl:matrix.orgcorvus: I'm interested in your thoughts on 81578813:24
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 813809: Lookup event class names from global symbol table https://review.opendev.org/c/zuul/zuul/+/81380914:23
@spamaps:spamaps.ems.hostAre we all aware that GitLab came up with a cooler name for what Zuul does in the dependent pipeline? https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html14:26
@clarkb:matrix.orgBut with artificial rev limiter in place14:27
@spamaps:spamaps.ems.hostWhat's that?14:28
@clarkb:matrix.orgThey limit running jobs to 20 requests at a time. Zuul makes that configurable (and it does a slow start expansion)14:29
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 813826: Store and resolve queue item's ahead/behind refs https://review.opendev.org/c/zuul/zuul/+/81382614:29
@spamaps:spamaps.ems.hostThat's probably just a compromise because they want it to be simple.14:29
@clarkb:matrix.orgAlso you have to be able to push to the branch to use them. Thats a weird choice for having computers do this stuff14:30
@clarkb:matrix.orgI wonder if that is just a limitation of their ACL system14:30
@clarkb:matrix.orgeg no way to express "can merge things via the computer" vs "can merge things"14:31
@spamaps:spamaps.ems.hostStill I've now run into two different groups who listened to me explain Zuul and went "oh like GitLab merge trains"14:31
@spamaps:spamaps.ems.hostSo it might make sense to make some reference to them in docs to explain the difference. But really, the outcome is likely very similar, even if Zuul has some advantages.14:32
@spamaps:spamaps.ems.hostHard to beat billion dollar marketing budgets. Might as well join them.14:33
@clarkb:matrix.orgYes, I think the outcomes are very similar. The biggest differences are going to be ability to scale up more with zuul and ability to use finer controls since you can delegate to zuul then you just have to control who can trigger what in zuul.14:33
@clarkb:matrix.orgFwiw github just announced their feature for this and they use a third term iirc14:34
@jim:acmegating.comalso maybe cross-repo speculative dependencies?14:34
@clarkb:matrix.orgMerge Queue: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/using-a-merge-queue14:34
@jim:acmegating.comnot to mention cross-source speculative dependencies... which i can't imagine either of them will be implementing any time soon14:34
@clarkb:matrix.orgcorvus: oh ya + all of zuul's other awesome features14:35
@mhuin:matrix.orgCan we come out with our new own hype term? I propose "the merginator"14:35
@jim:acmegating.commhu: ++  or merge rockets?  those are cooler than trains right?14:35
@jim:acmegating.comspamaps: but yes; frequently people ask for "how does zuul compare to X?" marketing material, and i think including those in such material would be good.  unfortunately, no one has written that yet.  :(14:36
@jim:acmegating.comwe also still haven't fleshed out gating.dev.  i believe we wanted to include things about dependencies on it.14:38
@clarkb:matrix.orghttps://review.opendev.org/c/zuul/zuul/+/815747 should be an easy test race fix if anyone has time for review14:38
@clarkb:matrix.orgAnd now school run time. Hoping to dive right into hashtag:sos after wards though (maybe after some email catchup)14:38
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 813895: Move job_graph attribute to BuildSet https://review.opendev.org/c/zuul/zuul/+/81389514:43
@mhuin:matrix.orgcorvus: regarding 736968 would you like me to edit the commit message with the rationale behind it? This patch came to be when I split the changes between zuul-web and the GUI, so the commit message was a bit of an afterthought14:59
@jim:acmegating.commhu: i'm not too worried about it... it's somewhat self evident (the driver bit was what i couldn't figure out on my own), and there's a comment record in gerrit now.   if you happen to update for something else, it wouldn't hurt.  :)15:01
@mhuin:matrix.orgnp15:01
@clarkb:matrix.orgfungi: maybe you want to review https://review.opendev.org/c/zuul/nodepool/+/815766 since it adds debian unstable stuff to nodepool dib images (for podman specifically)15:29
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/nodepool] 815545: Add py39 testing to nodepool https://review.opendev.org/c/zuul/nodepool/+/81554515:57
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815864: Fix race in change cache removal https://review.opendev.org/c/zuul/zuul/+/81586416:17
@jim:acmegating.comClark: i think that's the change_cache error i added the debugs to catch (they helped)16:18
@jim:acmegating.comtobiash, swest: ^16:18
@clarkb:matrix.orgthanks +216:23
@spamaps:spamaps.ems.host> <@jim:acmegating.com> we also still haven't fleshed out gating.dev.  i believe we wanted to include things about dependencies on it.16:27
I've been trying to find some time to work on this specifically.
@jim:acmegating.comtobiash: https://review.opendev.org/815747 is another race fix if you have a sec16:33
@clarkb:matrix.orgcorvus: comments on https://review.opendev.org/c/zuul/zuul/+/814070 if you have a second16:41
@jim:acmegating.comClark: looking16:41
@jim:acmegating.comClark: replied16:47
@clarkb:matrix.orgoh heh I shoudl've expanded diffs more to be sure of the _deepUpdate return type. The name and class method imply you're getting an updated Job type  but ya this should be fine as dict is serializable16:51
@jim:acmegating.comyeah, was just trying to reuse that with minimal churn :)16:52
@goneri:matrix.orghttps://review.opendev.org/c/zuul/zuul-jobs/+/815866 prevents use from using the role on Fedora 34 (py39). The line does not make a lot of sense and it looks like a mistake was introduce during the import of the module.17:04
@clarkb:matrix.orgcorvus: super minor thing on https://review.opendev.org/c/zuul/zuul/+/814242 but I feel like I'm overthinking it so went ahead and approved it17:11
@jim:acmegating.comClark: ++17:14
@goneri:matrix.orgThank for the super quick review ❤️.17:21
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 813913: Serialize JobGraph objects to ZK https://review.opendev.org/c/zuul/zuul/+/81391317:29
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 814065: Serialize ProjectMetadata on JobGraph https://review.opendev.org/c/zuul/zuul/+/81406517:32
@clarkb:matrix.orgcorvus: couple more minor things on https://review.opendev.org/c/zuul/zuul/+/814243 can you take a look before I approve?17:52
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815894: Use ZK transactions when deleting job request locks https://review.opendev.org/c/zuul/zuul/+/81589417:55
@jim:acmegating.comClark: will do; there's another race fix ^17:55
@jim:acmegating.comClark: rpld18:00
@clarkb:matrix.orgcorvus: question on the race fix18:05
@jim:acmegating.comClark: replied there18:14
@jim:acmegating.comi did miss a NoNodeError check there though; will update18:16
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815894: Use ZK transactions when deleting job request locks https://review.opendev.org/c/zuul/zuul/+/81589418:17
@jim:acmegating.comactually that's a silly fix hang on18:17
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815894: Use ZK transactions when deleting job request locks https://review.opendev.org/c/zuul/zuul/+/81589418:18
@jim:acmegating.combetter, i think.18:18
@clarkb:matrix.orgah ok clean check in the tests is the problem but in production another pass of the cleanup will prune appropriately18:35
@jim:acmegating.comyeah, and i figured the tighter we can make that the better in general18:37
@clarkb:matrix.orgGonéri: comment on your zuul-jobs fips change19:08
@clarkb:matrix.orgthough now that I'm rethinking it maybe its fine. To the truth table19:11
@clarkb:matrix.orgya ist fine in my head I thought we wanted Fedora to produce a true output for some reason. Its True to fail so we want it to produce False19:17
@goneri:matrix.orgClark: I did some tests before using a Fedora 34 VM and I was working as expeected, but I was also confused by the way the test is done. I've simplify the when condition.19:23
@jpew:matrix.orghuh, apparently `kubectl` has a hard (non configurable) 4 hour time limit on a single connection... this is disappointing19:57
@jim:acmegating.comi guess they figured 14.4ksec should be enough for anybody?19:58
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 814071: Add test_freeze_noop_job https://review.opendev.org/c/zuul/zuul/+/81407120:04
@jpew:matrix.orgcorvus: Ya, it's just really annoying b/c it limits my ability to use k8s for my builds :(20:05
@jpew:matrix.orgUnless maybe I can ssh in instead of using kubectl20:05
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 814069: Remove setBase from job freeze API https://review.opendev.org/c/zuul/zuul/+/81406920:12
@jim:acmegating.comClark: what do you think about concurrency n-2?20:22
@jim:acmegating.comif all the opendev nodes are 8cpus now, that's 6, vs 4 prior20:23
@clarkb:matrix.orgcorvus: probablyworht a try if we're still seeing the load issues20:27
@jim:acmegating.comyeah.  i'm not fully there yet, but mulling it over.20:28
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815864: Fix race in change cache removal https://review.opendev.org/c/zuul/zuul/+/81586420:50
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815894: Use ZK transactions when deleting job request locks https://review.opendev.org/c/zuul/zuul/+/81589420:52
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815747: Fix race is test_zookeeper_disconnect https://review.opendev.org/c/zuul/zuul/+/81574721:02
@jim:acmegating.comClark: now we're cooking with gas :)21:03
@clarkb:matrix.orgNice I was just going to start reviewing the next change in the sos stack21:03
@jim:acmegating.com\o/21:03
@clarkb:matrix.orgAlso I just had a crazy idea. We could do two node test jobs for zuul and run mysql postgres and zk on the second node21:04
@jim:acmegating.comooh21:04
@clarkb:matrix.orgto try and offload that load from where the unitetsts run21:04
@jim:acmegating.comwe could share that node between all the tox- jobs; i don't know if that's a good or bad idea though.21:05
@clarkb:matrix.orgya I haven't really thought it through yet, but realized that at least some of the load overhead is in the databases and we can in theory run them elsewhere to limit what the executor governors are seeing21:06
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815910: Clean up leaked lock path in request queue https://review.opendev.org/c/zuul/zuul/+/81591021:17
@jim:acmegating.comunfortunately, the atomic lock delete wasn't sufficient, so we'll need that to fix that race ^21:17
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 814070: Create Abstract and FrozenJob classes https://review.opendev.org/c/zuul/zuul/+/81407021:18
@jim:acmegating.comi think i may go ahead and approve that one with just votes from Clark and I; it's a followup race fix to well-tested code21:25
@jim:acmegating.comthis failure really has me stumped: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_32c/814242/9/gate/zuul-tox-py38/32cacbd/testr_results.html21:27
@jim:acmegating.comi'm wondering if it's something about how we check for watches.  i'm actually able to reproduce it locally occasionally, but i have to run the full test suite for it, and it only happens like 20% of the time, so it's still not a quick debug cycle21:27
@jim:acmegating.commaybe the output from the server is limited21:29
@clarkb:matrix.orgwhen reviewing this sos stacks I really wish gerrit let me align the two sides of the diff independently21:29
@jim:acmegating.comClark: ah like when functions move... that would be cool21:31
@clarkb:matrix.orgya21:31
@jim:acmegating.comaha! the response from the server to admin commands is capped at 8k21:33
@clarkb:matrix.orgis that true for the http 4 letter words too?21:34
@clarkb:matrix.orgmight be better off using that if not21:34
@clarkb:matrix.orgcorvus: couple of notes on https://review.opendev.org/c/zuul/zuul/+/81432921:41
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815911: Remove getZKWatches https://review.opendev.org/c/zuul/zuul/+/81591121:43
@jim:acmegating.comClark: it's a client-side limitation.  i'm sure we could either fix the tcp impl, or switch to http.  but maybe it's not worth it.  i just removed the check ^21:44
@clarkb:matrix.orgcorvus: looking at that change it seems we really only care if watches is not empty or if it is empty. Maybe we should keep the check but boil it down to that binary rather than caring about a full deserialization?21:46
@jim:acmegating.comClark: the problem is that the wchs command returns, say 10k of data, and our watches might be in the last 2k of it, so we don't get it.  the failure mode is returning empty when there really are watches.21:47
@jim:acmegating.comif we get a value, we can trust it.  but we can't trust non-values.21:48
@clarkb:matrix.orgI see, and we're always going to get some value back21:48
@clarkb:matrix.orgor rather some output back which may or may not have the values we want due to the limit21:48
@jim:acmegating.comright21:48
@jim:acmegating.comi guess we could keep it and fail if it's non-empty at the end.  that would correctly fail when run in isolation, but might incorrectly pass when run under contention.21:49
@jim:acmegating.commaybe that's better than nothing?21:49
@jim:acmegating.comcould do that and add a TODO to re-implement the command method21:49
@clarkb:matrix.orgthat might help future us debugging a test in isolation21:49
@jim:acmegating.comk, will rework.21:50
@jim:acmegating.comClark: replied on 81432921:53
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815911: Remove false-negative in zk test https://review.opendev.org/c/zuul/zuul/+/81591121:57
@jim:acmegating.comClark: reworked ^21:57
@clarkb:matrix.orgcorvus: https://review.opendev.org/c/zuul/zuul/+/814679 has a suggestion for a simplification and possible performance improvement22:07
@clarkb:matrix.orgAlso an other general thing is maybe we should use a bunch of constants for the various zk paths we're interacting with? Would be easier to find where we interact with specific nodes that way22:10
@clarkb:matrix.orgcorvus: comment on https://review.opendev.org/c/zuul/zuul/+/814570 about simplification too22:18
@jim:acmegating.comClark: i wish we could make that optimization on 814679, but i don't see a way to do that (except maybe storing the serialized json and passing it in as an extra argument to bypass that case); see comment reply22:32
@clarkb:matrix.orgis the issue that we haev to save the path and some minimal content to zk before writing the JobData object?22:36
@jim:acmegating.comClark: yes because the JobData is hierarchicaly under the job22:37
@jim:acmegating.comClark: agree on 814570; i'll make a followup22:38
@clarkb:matrix.orgcorvus: for 814679 can you check my response. I think it may still be possible? Though I may be missing something important there and it may not actually be more efficient now22:40
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815910: Clean up leaked lock path in request queue https://review.opendev.org/c/zuul/zuul/+/81591022:42
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815916: Reduce use of OrderedDict in PipelineState https://review.opendev.org/c/zuul/zuul/+/81591622:44
@clarkb:matrix.orgcorvus:  https://review.opendev.org/c/zuul/zuul/+/815916 has a bit that needs updating that was missed (comment left inline)22:49
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 814242: Make FrozenJob.updateParentData a static method https://review.opendev.org/c/zuul/zuul/+/81424222:50
@jim:acmegating.comClark: thx, replied on 67922:54
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815916: Reduce use of OrderedDict in PipelineState https://review.opendev.org/c/zuul/zuul/+/81591622:55
@clarkb:matrix.orgmakes sense approved22:55
@clarkb:matrix.orgcorvus: left a note on https://review.opendev.org/c/zuul/zuul/+/814571 about some code symmetry23:03
@clarkb:matrix.orgI might need to take a break here and exercise the brain on something different for a bit :)23:03
@jim:acmegating.comClark: replied23:06
@clarkb:matrix.orgcorvus: thanks I suggested some edits for symmetry then23:08
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815917: Update Pipeline for symmetry https://review.opendev.org/c/zuul/zuul/+/81591723:14
@jim:acmegating.comClark: ^ too much typing so i went with your original suggestion :)23:14
@clarkb:matrix.orgheh ok. Mostly for me reading that I imemdiately have alarm bells when the add and get don't operate on the same objects (they do but it isn't clear the way it was written). I'm happy as long as we avoid those alarm bells23:15
@clarkb:matrix.orgunrelated is https://review.opendev.org/c/zuul/zuul/+/815343 worht single core approving? It doesn't fix anything super important but while we are at the game of improve CI seems worthwhile23:15
@jim:acmegating.comClark: yeah i agree the symmetry helps, at least locally.  i didn't see that all the others were that way already :)23:16
@jim:acmegating.comClark: yeah, i went ahead and approved it.23:17

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!