Tuesday, 2022-03-01

-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 831323: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/83132300:04
@iwienand:matrix.orgClark: ahh, good thought.  i bet also it's not the distro as such update, but that the repos have different podman versions for focal v bionic00:05
@iwienand:matrix.orgsorry, distracted by ^^ which i hope passes but we want to get in, because it looks like we've got borked apt configs 00:06
@jim:acmegating.com> <@clarkb:matrix.org> ianw: corvus is it possible one of the earlier test playbooks is leaking the localhost:9000 pushed file to the registry into when that playbook runs? Then when you check the listing it still shows as localhost:9000 because the sha matches what was in docker so the pull is a noop?00:07
No, you can have multiple image tags pointing to the same image. If that test is failing it means podman isn't pulling the image we requested.
@iwienand:matrix.orgthat is the case, bionic is install 3.0.1-2 and focal 3.4.something00:08
@jim:acmegating.comMaybe something about podman not assuming docker.io.00:09
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 831323: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/83132300:10
@y2kenny:matrix.orgare there any way to help debug zuul-web authorization issue?  I tried adding -d to launching zuul-web but that don't seem to add any more info.  I can see the http 401 but I have no idea which admin rule failed.00:16
@y2kenny:matrix.orgor how the rule failed... I actually only set the iss condition00:18
@y2kenny:matrix.orgare there other ways to do enqueue/dequeue beside web auth?  I guess with 5.0.0 web auth is the only way left?00:28
@clarkb:matrix.orgKenny Ho: yes, with 5.0 there is no gearman anymroe so need to rely on webauth00:29
@clarkb:matrix.orgI'm not sure if the logging can be set up to be more verbose on the web side00:29
@y2kenny:matrix.orgsigh... ok...  I am out of idea00:29
@y2kenny:matrix.orgI have setup dex to bridge oidc with my corp ldap and able to authenticate but not able to authorize00:30
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 831326: ensure-podman: add containernetworking-plugins https://review.opendev.org/c/zuul/zuul-jobs/+/83132600:30
@clarkb:matrix.orgKenny Ho: maybe set up a local token and ensure that works then build up from there00:30
@clarkb:matrix.orgthat removes a lot of moving parts00:30
@iwienand:matrix.org^^ that should remove all the extra fluff warnings we're getting with the focal ugprade anyway00:31
-@gerrit:opendev.org- Ian Wienand proposed:00:31
- [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/831135
- [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/831131
@y2kenny:matrix.orgClark: How do I setup local token?  Is it via allow_authz_override?00:32
@clarkb:matrix.orgKenny Ho: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1-L4 that is how we do it. We set the issuer to zuul itself and then set it up in our zuul config thatway00:33
@y2kenny:matrix.orgooo... I was wondering about that... because my iss is my dex server and I wasn't sure how the opendev example works.00:35
@clarkb:matrix.orgKenny Ho: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul/templates/zuul.conf.j2#L57-L70 that is what the zuul config looks like00:36
@y2kenny:matrix.orgah yes... I was just about to ask that.00:37
@y2kenny:matrix.orgok... I will give that a try.  Thanks!00:37
@y2kenny:matrix.organd the zuul_local_auth_secret is the auth-token used by zuul-client?00:38
@clarkb:matrix.orgKenny Ho: you'll need to manually provision the token once you've done this. It isn't automated via oidc00:39
@clarkb:matrix.orgKenny Ho: no thats the secret used to generate tokens. There is a zuul-client (or is it zuul?) command to generate an HS256 token00:39
@y2kenny:matrix.orgah ok.00:39
@y2kenny:matrix.orginstead of generating the token via web login00:39
@clarkb:matrix.orgKenny Ho: yup https://zuul-ci.org/docs/zuul/latest/client.html#create-auth-token it is a zuul command00:40
@y2kenny:matrix.orgClark: yay it worked!  Thanks!  One thing I noticed though... for zuul-client enqueue --change is required (which needs change#,patch# for gerrit) but I think --ref should be allowed also?01:08
@clarkb:matrix.orgIs there an enqueue-ref command with zuul-client? It was split with the gearman tools01:25
@y2kenny:matrix.orgOh01:25
@y2kenny:matrix.orgthere may be01:25
@y2kenny:matrix.orgI think I came across that before but it escape my mind... sorry about that01:26
-@gerrit:opendev.org- Ian Wienand proposed:01:26
- [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/831135
- [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/831131
@jpew:matrix.orgWhen I have a single gerrit gating in Zuul, it works just fine. When Zuul starts stacking gerrits, it will merge the top one, but then thinks it fails and requeues all the subsequent ones which causes a lot of rebuilds. Does anyone know why this is?01:38
@jpew:matrix.orge.g. the top change in the gate queue cherry-picks successfully, but Zuul thinks it fails for some reason01:38
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 831326: ensure-podman: add containernetworking-plugins https://review.opendev.org/c/zuul/zuul-jobs/+/83132601:45
@clarkb:matrix.orgCherry picking changes the sha. Maybe it is re enquing to reflect that the upstream state has changed01:54
@iwienand:matrix.orgwell pushing to docker.io/test/image doesn't seem to work either01:54
@iwienand:matrix.orghttps://zuul.opendev.org/t/zuul/build/68842987c3704d4f9464d07426698a25/console#3/6/10/ubuntu-focal01:54
@iwienand:matrix.orgthen get unauthorized on the pull, suggesting it's actually going to dockerhub01:58
-@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/zuul-jobs] 831323: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/83132303:56
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/83113505:12
@iwienand:matrix.orgClark: corvus the "leaked image" theory seems correct; although upon investigation it's not clear if it's a feature or a bug.  I have filed https://github.com/containers/podman/issues/1338306:29
-@gerrit:opendev.org- Ian Wienand proposed:06:36
- [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/831135
- [zuul/zuul-registry] 831339: podman: make sure we remove the pulled image https://review.opendev.org/c/zuul/zuul-registry/+/831339
-@gerrit:opendev.org- wangxiyuan proposed: [zuul/zuul-jobs] 830706: Add openEuler to iptalbe firewall persist https://review.opendev.org/c/zuul/zuul-jobs/+/83070606:47
-@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/zuul-jobs] 831354: Don't enable pip mirror for Debian Buster https://review.opendev.org/c/zuul/zuul-jobs/+/83135410:29
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 831222: GUI: Do not show sign-in button if no IdP is available https://review.opendev.org/c/zuul/zuul/+/83122210:41
-@gerrit:opendev.org- Zuul merged on behalf of Matthieu Huin https://matrix.to/#/@mhuin:matrix.org: [zuul/zuul] 831231: Doc: stress the role of HS256 authenticator https://review.opendev.org/c/zuul/zuul/+/83123111:22
@jpew:matrix.orgPer the question I asked later yesterday: If gerrit is set to the "cherry-pick" submit type, does that mean that there is no point in speculatively gating? I just seems to rebuild everything again anytime something changes.... or I'm missing some configuration thing?14:26
@avass:vassast.orgjpew: reading the gerrit docs I'm guessing it's possible to enqueue changes in the "wrong order" since gerrit ignores dependencies between changes unless `change.submitWholeTopic` is configured and maybe zuul doesn't handle that well14:45
@avass:vassast.orgbut that's just a wild guess14:46
@fungicide:matrix.orgjpew: i wouldn't rule out the possibility that there are few users relying on gerrit's cherry-pick submit type with zuul. the way the dependent pipeline manager works, i have a feeling the merge submit types are the only ones which will gate reliably as those closely match how zuul constructs its own refs and they don't alter the commits on submit. i know people have brought up similar problems with the rebase and fast-forward-only submit types (the former because they revise the commit which was tested similar to cherry-pick, and the latter because it doesn't allow reordering of changes)15:02
@avass:vassast.orgfungi: i think I used `rebase if necessary` a lot previously and i don't think we experienced any issues with that. Though I'm currently trying to figure out some issues with `cherry-pick`15:05
@avass:vassast.orgbut cherry-pick could maybe be causing zuul to rebuild the gate queue since gerrit creates a new patch-set to add commit message footers and merges that instead of what was actually tested15:08
@clarkb:matrix.orgAlbin Vass: note that a cherry pick always implies a new sha because it will update the timestamp on the commit. This means even if Gerrit didn't update the message footers you would likely have the same issue.15:10
@avass:vassast.orgClark: true15:10
@clarkb:matrix.orgThis is a common source of confusion for us with people using git-review -x to reconstruct stacks of changes. They are surprised when every change updates in Gerrit and not just the child most they edited by hand too.15:11
@clarkb:matrix.orgThat said I wonder if Gerrit could represent it not as a new patchset which is likely the part that confuses Zuul since only the most recent patchset is mergeable15:14
@avass:vassast.orgI also saw an issue where zuul had set up the wrong speculative state with cherry-pick but still ran the gate pipeline and succeeded, then failed to merged (luckily) because gerrit was reporting a merge conflict while zuul didn't detect it.15:14
@avass:vassast.orghaven't been able to reproduce that though, but I think it was specifically when there was a commit dependency and the dependency had already been merged (so the second change was dependent on an old patch-set).15:16
@jpew:matrix.orgHmm, that's unfortunate..... I guess I either need to change Gerrit, or limit my speculative gating window to 1 then?15:18
@jpew:matrix.org(Changing Gerrit is going to be a hard sell)15:19
@dpawlik:matrix.orgAlbin Vass: hey, could you check https://review.opendev.org/c/zuul/zuul-jobs/+/829533/ ?15:23
@fungicide:matrix.orgjpew: since Albin Vass has experience with rebase submits working, that may mean there's hope for getting cherry-pick submits working too. since the gerrit docs indicate that both types create a new patchset/revision on the change, so this might just be a bug in the gerrit driver which nobody's worked out a fix for15:30
@fungicide:matrix.org * jpew: since Albin Vass has experience with rebase submits working, that may mean there's hope for getting cherry-pick submits working too. the gerrit docs indicate that both types create a new patchset/revision on the change, so this might just be a bug in the gerrit driver which nobody's worked out a fix for15:30
@jpew:matrix.orgIt's weird that it only fails when "stacked"15:30
@avass:vassast.orgfungi: jpew I think we used `rebase if necessary` and not `rebase always`though15:31
@avass:vassast.org`rebase always` creates a new patch set while `rebase if necessary` doesn't15:31
@fungicide:matrix.orgjpew:  "stacked" meaning you have multiple changes in the gate pipeline at the same time, or that you have multiple changes depending on one another?15:32
@jpew:matrix.orgmultiple in the pipeline at the same time15:32
@fungicide:matrix.orgAlbin Vass: "rebase if necessary" creates a new patch set when necessary, just not when it's not necessary15:32
@avass:vassast.orghuh15:32
@avass:vassast.orgdpawlik: looking at it but I'm trying to figure out where `python_version` comes from in the Centos 9 test :)15:34
@avass:vassast.orgfungi: how come `Always merge` or `Merge if necessary` doesn't cause zuul to rebuild the gate queue then since the merge commit in zuul and what gerrit creates should differ?15:36
@avass:vassast.orgI guess zuul handles that somehow?15:36
@avass:vassast.orgjpew:  I don't think we're seeing the gate queue being rebuilt in zuul 2 using cherry-pick in gerrit right now15:37
@jpew:matrix.orgHmm, OK. Maybe it's something else then.... Why is the build being marked as ERROR, but all the jobs passed, and it merged15:38
@jpew:matrix.orgThe buildset is being marked as ERROR15:39
@avass:vassast.orgmaybe it succeeds reporting to gerrit and submits the change but fails just after that?15:40
@jpew:matrix.orgIt must15:40
@avass:vassast.orgin a way so zuul thinks reporting failed15:40
@clarkb:matrix.org> <@avass:vassast.org> fungi: how come `Always merge` or `Merge if necessary` doesn't cause zuul to rebuild the gate queue then since the merge commit in zuul and what gerrit creates should differ?15:41
There is no new patchset with merges. It is the new patchset that causes this problem I think.
@jpew:matrix.orgAh, it must race with the new patchset being created by the gerrit cherry-pick15:42
@jpew:matrix.orgWhich make sense, when there is only one thing in the gate queue, the load is much lower so it beats gerrit. When it is merging speculatively, its doing other stuff and thus is more likely to see the new PS15:45
@jpew:matrix.org(all my zuul components are on one machine)15:45
@jpew:matrix.orgI bet if I set `pipeline.require.<gerrit source>.current-patchset` to false it would fix it....15:47
@jpew:matrix.orgOr move it to the trigger `require-approval` instead of the `pipeline.require`.... although that could be bad if someone pushes a new patch while its gating?15:49
@fungicide:matrix.orgyeah, for the driver, i think the trick will be finding a way to tell a user pushing a new revision before submit apart from gerrit pushing a new revision on submit15:53
@fungicide:matrix.orgif that's really what's causing the problem15:54
@fungicide:matrix.orgseems like it should be possible to create a test case for it in zuul's integration testing15:54
@fungicide:matrix.orgor even a unit test if we have a good model for how that submit type behaves15:55
-@gerrit:opendev.org- Zuul merged on behalf of Szymon Datko: [zuul/zuul-jobs] 829533: [ensure-python] Fix for CentOS/RHEL 9 packages https://review.opendev.org/c/zuul/zuul-jobs/+/82953316:05
@jpew:matrix.orgIt has the `SUBM` approval in gerrit16:12
@jpew:matrix.orgMaybe that should just be ignored?16:12
@clarkb:matrix.orgZuulians if you have time for https://review.opendev.org/c/zuul/zuul-registry/+/831235 and its child that aims to address some problems with zuul-registry that opendev has found. I believe worst case we can fall back to zuul-registry 1.1.0 in zuul-jobs and anywhere else that needs it should this go sideways.16:50
@clarkb:matrix.orgjpew: Gerrit should still prevent older patchsets from merging. That is a requirement on the Gerrit side that only the most recent patchset can merge. I suspect setting the current-patchset flag may fix this. The downside will be you have to wait for submit to fail after a new patchset is pushed rather than resetting sooner16:51
@clarkb:matrix.orgthat tradeoff may be worthwhile?16:51
@jpew:matrix.orgClark: Ya. I'd rather have it build ahead16:51
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831430: Dequeue items that can no longer merge https://review.opendev.org/c/zuul/zuul/+/83143017:26
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831316: Check if job request is still queued before yielding https://review.opendev.org/c/zuul/zuul/+/83131617:34
@jim:acmegating.comtristanC: i have a report that the ansi escape codes [m and [K don't work with re-ansi -- it looks like it isn't handling the default values (eg [m == [0m).  does that sound plausible?18:22
@tristanc_:matrix.orgcorvus: that is correct, let me fix that18:27
@avass:vassast.org> <@avass:vassast.org> Regarding: https://review.opendev.org/c/zuul/zuul/+/83084018:29
> I'm wondering if it's better to piggy back on `zuul_return` so it's possible do something similar do:
>
> ```
> - zuul_return:
> data:
> retry: false
>
> - fail:
> msg: ...
> ```
>
> which could also make it possible to set `retry: true` to retry more dynamically in a `run` or `post` playbook.
> Or if it's better to implement the "fail early" functionality with a dedicated `zuul_fail` module.
I'm most likely going to start implementing this in a day or two so if i could steal the attention from someone who has any opinion about this for a second that would be great. Just so it doesn't get shot down during review :)
@tristanc_:matrix.orgcorvus: not sure what to do with [K. The spec says `clear from cursor to the end of the line`, but gnome-terminal seems to do a noop instead18:49
@tristanc_:matrix.orgI guess we can interpret it as the current erase line code, here is the fix I can publish if you are ok with my interpretation: https://softwarefactory-project.io/r/c/software-factory/re-ansi/+/2417518:56
@clarkb:matrix.orgtristanC: 1K is erase from beginning of the line to cursor and 0K is from cursor to end of line. Is differentiating that behavior not possible for some reason?19:00
@tristanc_:matrix.orgwell it is possible, but i can't reproduce the behavior in my terminal, so i'm not sure it is relevant19:01
@clarkb:matrix.orgI was reading a blog post about this recently and gnome-terminal is extremely buggy when it comes to this sort of thing. xterm is apparently the standard to compare against19:08
@jim:acmegating.com`foo bar^[[3D^[[Khi` is an example, right?  that should show as `foo hi`19:11
@jim:acmegating.comwhereas `foo bar ^[[3D^[[1Khi` renders as `    hir`19:12
@jim:acmegating.com * whereas `foo bar ^[[3D^[[1Khi` renders as `____hir`  (edit: spaces to underscores for clarity in chat)19:13
@jim:acmegating.com * whereas `foo bar^[[3D^[[1Khi` renders as `____hir`  (edit: spaces to underscores for clarity in chat)19:13
@tristanc_:matrix.orgoh well that kind of cursor movement is currently not supported and it's unclear how to do it efficiently...19:15
@jim:acmegating.commaybe it's better to noop then?19:16
@jim:acmegating.com(or even leave the escape sequences unhandled, i guess like they are now?)19:17
@tristanc_:matrix.orgfrom what I understand, [_K is used to update the current line, and erasing what was before seems to be good19:17
@jim:acmegating.comwhat is `_K` ?19:18
@tristanc_:matrix.orge.g. `[K` or `[1K`, or `[2K`19:19
@tristanc_:matrix.orgi meant, unless it is an interactive program, then I can't imagine why a CI tool would erase forward19:20
@jim:acmegating.comah.  well, you could do "clear this line and start over" with either "[1K[G" or "[G[0K"19:21
@tristanc_:matrix.orgright, and I think `[K` or `[0K` is often used after a `[G` to clear the previous output, thus I think it's fine to interpret this code as, drop the current line and starts over19:23
@tristanc_:matrix.orgor even after a `\r`19:23
@jim:acmegating.comtristanC: i went and looked up the example that was reported to me -- it's `[1A[KStatus output` -- that's repeated after a \n.  so basically the \n is returning to column 1, then the 1A moves up to the line, then [K clears it.19:28
@jim:acmegating.comso that particular case, since it's showing up on multiple log lines anyway, it doesn't matter.  a noop or clear-to-start would work equally well.19:28
@tristanc_:matrix.orgif necessary i can look into adding support for `\[1A`, but that would require a bit of overhead to be able to reach the previous line, thus if that's ok, i prefer to go forward with the simplest solution.19:30
@jim:acmegating.comi think i'd lean toward doing a noop for anything you don't want to implement, but that's because i have a bias towards maximum information.19:31
@jim:acmegating.comtristanC: yeah, if i were in your position, i probably wouldn't implement the movement characters at all, and maybe even stop handling [1K19:31
@jim:acmegating.com(i definitely agree with wanting to keep it simple.  and to be clear, i don't think the current [K handling is bothering anyone -- just reporting that observation for completeness)19:36
@jim:acmegating.com([m can be a problem though as it can produce weird/illegible color output)19:37
-@gerrit:opendev.org- Ian Wienand proposed:19:46
- [zuul/zuul-registry] 831319: podman buildset testing: dump image list https://review.opendev.org/c/zuul/zuul-registry/+/831319
- [zuul/zuul-registry] 831339: podman: make sure we remove the pulled image https://review.opendev.org/c/zuul/zuul-registry/+/831339
- [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/831135
- [zuul/zuul-registry] 831440: Fix and/or matching for image pre-conditions https://review.opendev.org/c/zuul/zuul-registry/+/831440
-@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/zuul-jobs] 831443: Fix ensure-pip test on Debian Buster https://review.opendev.org/c/zuul/zuul-jobs/+/83144320:13
-@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 831108: Update to DIB 3.19.1 https://review.opendev.org/c/zuul/nodepool/+/83110820:34
-@gerrit:opendev.org- Ian Wienand proposed:20:39
- [zuul/zuul-registry] 831440: Fix and/or matching for image pre-conditions https://review.opendev.org/c/zuul/zuul-registry/+/831440
- [zuul/zuul-registry] 831135: Update testing to Ubuntu Focal https://review.opendev.org/c/zuul/zuul-registry/+/831135
-@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul] 831450: web: bump re-ansi version https://review.opendev.org/c/zuul/zuul/+/83145020:53
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/83113120:56
@jim:acmegating.comtristanC: thanks, +2!20:59
@clarkb:matrix.orgtristanC: corvus do we want to check that on the preview site before approving or should I +A?21:00
@jim:acmegating.commaybe we could sanity check it against a docker build or something?  i think that gets color and movement...21:02
@tristanc_:matrix.orggood idea, let me see, i think there is a change for that21:02
@clarkb:matrix.orgsounds good I +2'd so we can +A it once we've checked one of those21:02
-@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul-jobs] 763054: DNM: generate ansi log for benchmark purpose https://review.opendev.org/c/zuul/zuul-jobs/+/76305421:07
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831430: Dequeue items that can no longer merge https://review.opendev.org/c/zuul/zuul/+/83143021:09
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/83113121:26
-@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul] 831453: web: enable re-ansi in the logfile view https://review.opendev.org/c/zuul/zuul/+/83145321:54
@iwienand:matrix.orgwell this is a silly and annoying line : https://cs.opensource.google/go/x/net/+/master:http/httpproxy/proxy.go;l=18522:25
@iwienand:matrix.orgi'm not sure why go is the only environment they does not obey http_proxy to a local address22:26
@iwienand:matrix.orghttps://github.com/golang/go/issues/51416 we'll see if anyone cares/agrees/disagrees I guess.  22:51
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/83113123:20
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-registry] 831131: [WIP] Enable mitmproxy between docker/podman to dump test https://review.opendev.org/c/zuul/zuul-registry/+/83113123:36
@clarkb:matrix.orgcorvus: couple of thoughts on the stack at https://review.opendev.org/c/zuul/zuul/+/83092523:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!