Wednesday, 2023-03-22

@iwienand:matrix.org```00:02
- zuul-jobs-test-build-container-image-docker-release
- zuul-jobs-test-build-container-image-podman-release
- zuul-jobs-test-build-container-image-docker-promote
- zuul-jobs-test-build-container-image-podman-promote
- zuul-jobs-test-build-docker-image-release
- zuul-jobs-test-build-docker-image-release-multiarch
```
@iwienand:matrix.orgdo we not need  ```zuul-jobs-test-build-docker-image-promote``` for symmetry?00:03
@clarkb:matrix.orgHrm yes I think so?00:05
@iwienand:matrix.organyway it wasn't there before00:06
@jim:acmegating.comClark: not possible to test00:11
@iwienand:matrix.orgi think we're ok to merge the stack now, so we minimise conflicts etc. moving forward?    00:11
@jim:acmegating.com * ianw:  Clark: not possible to test docker promote00:11
@jim:acmegating.com(because the apis for dockerhub aren't available on other registries; it only works against the actual single instance public service dockerhub)00:12
-@gerrit:opendev.org- Zuul merged on behalf of Oleksandr Kozachenko: [zuul/zuul-jobs] 838919: Add promote-container-image role https://review.opendev.org/c/zuul/zuul-jobs/+/83891900:32
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com:00:32
- [zuul/zuul-jobs] 878048: Revert "Use --password-stdin for upload-container-image" https://review.opendev.org/c/zuul/zuul-jobs/+/878048
- [zuul/zuul-jobs] 878049: Add container repository cred permission checks https://review.opendev.org/c/zuul/zuul-jobs/+/878049
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com:00:32
- [zuul/zuul-jobs] 878137: Refactor docker/container image variables https://review.opendev.org/c/zuul/zuul-jobs/+/878137
- [zuul/zuul-jobs] 878172: Refactor docker/container image jobs https://review.opendev.org/c/zuul/zuul-jobs/+/878172
@jim:acmegating.comI sent an email to zuul-discuss with the background/current state of the docker situation/work00:43
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:00:49
- [zuul/nodepool] 878178: Vendor persistent recursive watch Kazoo support https://review.opendev.org/c/zuul/nodepool/+/878178
- [zuul/nodepool] 878179: Use a persistent recursive watch for caches https://review.opendev.org/c/zuul/nodepool/+/878179
-@gerrit:opendev.org- Ian Wienand proposed:00:59
- [zuul/zuul-jobs] 878175: containers : update test variable https://review.opendev.org/c/zuul/zuul-jobs/+/878175
- [zuul/zuul-jobs] 878176: container role docs : clarify requirements https://review.opendev.org/c/zuul/zuul-jobs/+/878176
@jjbeckman:matrix.orgHi folks!06:20
Would really appreciate some clarification regarding setting up the `executor` to sync with a github.com repo, using the GutHub App method, via `zuul-operator`.
The following documentation states:
https://zuul-ci.org/docs/zuul/latest/drivers/github.html#attr-%3Cgithub%20connection%3E.app_key
> <github connection>.app_key
> Path to a file containing the secret key Zuul will use to create tokens for the API interactions. In Github this is known as Private key and must be collected when generated.
Should this key be the GitHub App private key? If so, how is one expected to place this key in the executor pod instance, zuul-operator-wise?
Or, should this be a general private key for which the executor uses to `git clone` repos? Again, if so, how is one expected to place this key in the executor pod instance?
The example `zuul.conf` found in the official zuul-config documentation isn't clear to me on how one should interpret this.
https://opendev.org/zuul/zuul-operator/src/branch/master/doc/source/index.rst#specification-reference
By the way, with my current zuul.conf, I am zuul.ExecutorServer is unable to `git clone`.
```
2023-03-17 14:06:21,967 ERROR zuul.ExecutorServer: Got exception while updating repo github/xxx/yyy
...
2023-03-17 14:06:21,967 ERROR zuul.ExecutorServer: git@github.com: Permission denied (publickey).
```
```
[connection "github"]
app_id={redacted}
app_key=/etc/zuul/connections/github/sshkey # the GitHub App private key
driver=github
rate_limit_logging=False
secretName=github-secrets # Kubernetes secret that holds the GitHub App private key
webhook_token={redacted} # GitHub App Webhook token
sshkey=/etc/zuul/connections/github/sshkey
```
-@gerrit:opendev.org- Dong Zhang proposed: [zuul/nodepool] 878093: Filter out shutting-down instances in listInstances() https://review.opendev.org/c/zuul/nodepool/+/87809306:25
@jjbeckman:matrix.org> <@jjbeckman:matrix.org> Hi folks!06:29
>
> Would really appreciate some clarification regarding setting up the `executor` to sync with a github.com repo, using the GutHub App method, via `zuul-operator`.
>
> The following documentation states:
>
> https://zuul-ci.org/docs/zuul/latest/drivers/github.html#attr-%3Cgithub%20connection%3E.app_key
>
> > <github connection>.app_key
> > Path to a file containing the secret key Zuul will use to create tokens for the API interactions. In Github this is known as Private key and must be collected when generated.
>
> Should this key be the GitHub App private key? If so, how is one expected to place this key in the executor pod instance, zuul-operator-wise?
> Or, should this be a general private key for which the executor uses to `git clone` repos? Again, if so, how is one expected to place this key in the executor pod instance?
>
> The example `zuul.conf` found in the official zuul-config documentation isn't clear to me on how one should interpret this.
> https://opendev.org/zuul/zuul-operator/src/branch/master/doc/source/index.rst#specification-reference
>
> By the way, with my current zuul.conf, I am zuul.ExecutorServer is unable to `git clone`.
> ```
> 2023-03-17 14:06:21,967 ERROR zuul.ExecutorServer: Got exception while updating repo github/xxx/yyy
> ...
> 2023-03-17 14:06:21,967 ERROR zuul.ExecutorServer: git@github.com: Permission denied (publickey).
> ```
>
> ```
> [connection "github"]
> app_id={redacted}
> app_key=/etc/zuul/connections/github/sshkey # the GitHub App private key
> driver=github
> rate_limit_logging=False
> secretName=github-secrets # Kubernetes secret that holds the GitHub App private key
> webhook_token={redacted} # GitHub App Webhook token
> sshkey=/etc/zuul/connections/github/sshkey
> ```
Also, having `sshkey` be the key of the app_key seems to be the only way to get `zuul-operator` to place a file in the `zuul-*` pods. Is my understanding correct?
https://opendev.org/zuul/zuul-operator/src/branch/master/zuul_operator/zuul.py#L218-L219
-@gerrit:opendev.org- Dong Zhang proposed: [zuul/nodepool] 878093: Filter out shutting-down instances in listInstances() https://review.opendev.org/c/zuul/nodepool/+/87809307:50
-@gerrit:opendev.org- Dong Zhang proposed: [zuul/nodepool] 878094: Handle NoNodeError in _assignHandlers https://review.opendev.org/c/zuul/nodepool/+/87809410:59
-@gerrit:opendev.org- Dong Zhang proposed: [zuul/nodepool] 878094: Handle NoNodeError in _assignHandlers https://review.opendev.org/c/zuul/nodepool/+/87809411:00
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:15:34
- [zuul/nodepool] 878178: Vendor persistent recursive watch Kazoo support https://review.opendev.org/c/zuul/nodepool/+/878178
- [zuul/nodepool] 878179: Use a persistent recursive watch for caches https://review.opendev.org/c/zuul/nodepool/+/878179
-@gerrit:opendev.org- Tobias Urdin proposed:16:12
- [zuul/zuul] 877587: web: add dark mode and theme selection https://review.opendev.org/c/zuul/zuul/+/877587
- [zuul/zuul] 878236: web: migrate pages to react-table https://review.opendev.org/c/zuul/zuul/+/878236
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878239: Add support for passing env vars to the container build env https://review.opendev.org/c/zuul/zuul-jobs/+/87823917:36
@clarkb:matrix.orgcorvus: ianw ^ ok thats the first thing I noticed when actually digging int othe code of adding multiarch for the container roleset. It has no way of simply enabling buildkit which the docker roleset did/does17:38
@clarkb:matrix.orgI still need to digest the whole buildx multiarch runtime and put together a change for that.17:38
@clarkb:matrix.orgI'm planning to dig into that this afternoon after a long lunch break. I got up early to help keep an eye on the openstack release and haven't managed breakfast yet17:39
@jim:acmegating.comClark: we don't just want to do what the docker role does?17:41
@clarkb:matrix.orgcorvus: the docker role has a very specific "docker_use_buildkit" flag which then sets the env var properly if you set the ansible var17:42
@clarkb:matrix.orgI think for the generic role being able to pass a set of env vars seems more useful? users of podman or docker or etc could set arbitrar env vars for various tool options17:42
@jim:acmegating.comClark: gotcha17:44
@clarkb:matrix.orgcorvus: should I copy roles/build-docker-image/tasks/setup-buildx.yaml into build-container-image or just include_tasks from that file and let it live in docker (or maybe swap the locations around?)17:47
@jim:acmegating.comClark: comment on that env change17:50
@jim:acmegating.comClark: reference sounds tempting but we might not trigger all the test jobs on changes.  i'd probably copy just to keep it simple, even though that's blah.17:51
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:18:16
- [zuul/nodepool] 878178: Vendor persistent recursive watch Kazoo support https://review.opendev.org/c/zuul/nodepool/+/878178
- [zuul/nodepool] 878179: Use a persistent recursive watch for caches https://review.opendev.org/c/zuul/nodepool/+/878179
- [zuul/nodepool] 877431: Use image cache when launching nodes https://review.opendev.org/c/zuul/nodepool/+/877431
- [zuul/nodepool] 877432: Use node cache in node deleter https://review.opendev.org/c/zuul/nodepool/+/877432
- [zuul/nodepool] 877565: Log the reason we decline a request https://review.opendev.org/c/zuul/nodepool/+/877565
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878239: Add support for passing env vars to the container build env https://review.opendev.org/c/zuul/zuul-jobs/+/87823918:22
@clarkb:matrix.orgI think we also need to add sibling support?18:26
@clarkb:matrix.orgso many things18:26
-@gerrit:opendev.org- Tobias Urdin proposed: [zuul/zuul] 878244: client: add autohold_delete to rest client https://review.opendev.org/c/zuul/zuul/+/87824418:31
@clarkb:matrix.orgI'm getting a bit lost in all the tagging we do18:42
@clarkb:matrix.orgspecifically at https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-docker-image/tasks/buildx.yaml#L69 and https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-docker-image/tasks/buildx.yaml#L7918:42
@clarkb:matrix.orglooking at build-container-image/tasks/build.yaml I think I can mimic that but I'm not sure if that is correct18:43
@clarkb:matrix.orgI guess I'll push that as a start18:44
@clarkb:matrix.orgreading things I think we may not need those two blocks at all? It seems like we're trying to tag things for the final repository location. I'm going with that but unsure if it is needed since we already seem to push to the intermediate and buildset registries then a separte set of tasks will acutally publish to the final location?18:49
@clarkb:matrix.orgI do half wonder if we would have a better time updating build-docker-image and friends to talk to a different registry18:50
@clarkb:matrix.orgthen sort through making all of this generic later in the container rolesets18:50
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/87824618:52
@jim:acmegating.comistr having a conversation with ianw about extra tags recently18:54
@clarkb:matrix.orgI think that patchset is close but probably not complete18:54
@clarkb:matrix.orgI need to go eat lunch now. I'll be back later to see if it explodes spectacularly :)18:54
@jim:acmegating.comClark: ah it was https://review.opendev.org/872806 that was the extra tags18:55
@clarkb:matrix.orgoh interesting that makes me feel like the code I linked above is dead code18:56
@clarkb:matrix.orgbecause it basically just retags things for something we don't need?18:56
@clarkb:matrix.orghrm no the registries differ18:56
@clarkb:matrix.orgI find this very confusing. If we sort it out we should add comments18:57
@clarkb:matrix.orgIn particular the only place we seem to push in the old buildx path is before that retagging18:58
@clarkb:matrix.orgwhich implies to me nothing ends up seeing the retagging later18:58
@jim:acmegating.comClark: it looks like that's the final tag18:59
@jim:acmegating.comi think that's required, otherwise we won't have an image with the tag we expect19:00
@jim:acmegating.com(ie, we're trying to build "quay.io/foo/bar" -- that's what causes us to have "quay.io/foo/bar" in our local image cache at the end of the process)19:00
@jim:acmegating.com(so then, whatever comes next, be it "podman run quay.io/foo/bar" or "podman push quay.io/foo/bar" will work)19:01
@jim:acmegating.comi believe ianw was removing the extra tags in 872806 because the "temp registry" is an extra layer of indirection here, and so those tags won't end up in our local image cache19:02
@jim:acmegating.comlong story short; i think the file is currently correct and everything it does is necessary19:02
@clarkb:matrix.orgGotcha in that case I think what I attempted in my change is correct though it may have bugs :)19:04
@clarkb:matrix.orgThe intent was right19:04
@jim:acmegating.comClark: left a couple of comments on that.  accidentally resolved them sorry19:07
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/87824621:02
@clarkb:matrix.orgcorvus: thank you for the review. I addressed that and mixed siblings support in21:02
@clarkb:matrix.orgianw: ^ fyi21:02
@clarkb:matrix.orgLooks like that change isn't triggeringthe jobs that test the image builds for some reason. I need to do a school run but if anyone understands why feel free to push an update that actually triggers those jobs21:05
@clarkb:matrix.orgis it because zuul files: are completely overridden in child jobs?21:10
@jim:acmegating.comi'll figure it out and fix21:10
@clarkb:matrix.orgcorvus: fwiw I suspect it is ^21:11
@iwienand:matrix.orghttps://review.opendev.org/c/zuul/zuul/+/747614 has been on my todo list around making the inheritance more flexible for ... 2 years :/21:17
@jim:acmegating.comi think it's worth thinking about yaml tags for it21:21
@jim:acmegating.comi'm working on the zuul-jobs change now -- but i think i'm going to restructure this slightly again to try to balance config simplicity vs maybe running a few extra jobs21:22
@jim:acmegating.comi'll have a proposal in a few mins21:22
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878273: Fix file matchers in docker/container jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87827321:30
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Clark Boylan:21:30
- [zuul/zuul-jobs] 878239: Add support for passing env vars to the container build env https://review.opendev.org/c/zuul/zuul-jobs/+/878239
- [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/878246
@jim:acmegating.comClark: ianw ^ i think/hope that strikes a good balance21:31
@jim:acmegating.com(the cost is we'll be running the podman jobs even if we just touch ensure-docker -- the benefit is that the lists are much easier to maintain)21:32
@jim:acmegating.combut i think the build-container/docker-image roles are much more likely to change than ensure-docker/podman so it probably works out okay21:34
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878282: Add multi-arch build-container-image test jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87828221:41
@jim:acmegating.comClark: ^ then that adds jobs to exercise the multi-arch stuff you're writing (may want to squash that into your change)21:41
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl:21:51
- [zuul/zuul] 877341: Fix variable name in job request queue log message https://review.opendev.org/c/zuul/zuul/+/877341
- [zuul/zuul] 877246: Don't connect to MQTT broker in zuul-web https://review.opendev.org/c/zuul/zuul/+/877246
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl:21:52
- [zuul/zuul] 876255: Truncate Github file annotation message to 64 KB https://review.opendev.org/c/zuul/zuul/+/876255
- [zuul/zuul] 876159: Don't discard all cat job results in case of error https://review.opendev.org/c/zuul/zuul/+/876159
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl:21:52
- [zuul/zuul] 875899: Don't add PR title in commit message on squash https://review.opendev.org/c/zuul/zuul/+/875899
- [zuul/zuul] 875039: Retry jobs on transient IO errors on repo update https://review.opendev.org/c/zuul/zuul/+/875039
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878282: Add multi-arch build-container-image test jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87828221:55
@clarkb:matrix.orgI see failures on those multiarch jobs now. I'll look at fixing them and squashing the changes22:24
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-client] 878289: Publish images to quay.io https://review.opendev.org/c/zuul/zuul-client/+/87828922:26
@jim:acmegating.comthere's our canary for when everything is ready ^22:26
@clarkb:matrix.orgheh my ssh keys are already expired because I started so early today /me refreshes keys22:28
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/87824622:30
@clarkb:matrix.orgThats a fairly early failure so there may be more lurking in the updates.22:31
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878291: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87829122:35
@clarkb:matrix.org`ERROR: invalid tag "127.0.0.1:5100/127.0.0.1:5200/testrepo:latest": invalid reference format` is the latest error that that occurs because zj_image.repository is fully qualified in this case. Is the solution there to parse out only the path suffix from the url and use that instead of the name and port prefix ?22:41
@jim:acmegating.comugh i wondered about that.22:41
@jim:acmegating.comClark: i think so -- i think it's either that, or redefine the meaning of the variables (so repository doesn't include the registry).22:43
@clarkb:matrix.orgI think on the docker side it was an implicit root so that was/is similar to dropping the hostname:port prefix22:43
@clarkb:matrix.orgI think we end up using it so that repository means the full name elsewhere22:44
@clarkb:matrix.orgI think we can probably parse and get what we want for these special cases22:44
@jim:acmegating.comyep22:44
@clarkb:matrix.orghttps://docs.ansible.com/ansible/latest/playbook_guide/playbooks_filters.html#splitting-urls22:45
@clarkb:matrix.orgof course docker image locations don't actually specify a protocol so not sure if that will work. I'll test it22:46
@jim:acmegating.comif not, can probably do a regex with the /22:46
@clarkb:matrix.orgconfirmed urlsplit does not work. It treats the entire string as a path22:48
@clarkb:matrix.orgcorvus: ianw do we need to handle the implicit docker hub push location in the container roleset or can we assume dockerhub will always be explicit there?22:56
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878291: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87829122:56
@clarkb:matrix.orgif we need to handle the implicit location then parsing htis string becomes more fun as we may not have a host prefix22:56
@jim:acmegating.comi think we should expect it to always be explicit22:56
@clarkb:matrix.orgok that makes me life easier so I'm on board with that :)22:56
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/87824623:02
@clarkb:matrix.orgsomething like that maybe23:02
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:23:13
- [zuul/nodepool] 877431: Use image cache when launching nodes https://review.opendev.org/c/zuul/nodepool/+/877431
- [zuul/nodepool] 877432: Use node cache in node deleter https://review.opendev.org/c/zuul/nodepool/+/877432
- [zuul/nodepool] 877565: Log the reason we decline a request https://review.opendev.org/c/zuul/nodepool/+/877565
@clarkb:matrix.orgI pushed an Ansible parse error. I thought I had tested it locally sufficiently but maybe I got quoting wrong or something 23:20
@iwienand:matrix.orgif you want to take a break i can pull it up; if i'm going to review it need to understand it anyway :)23:20
@clarkb:matrix.orgGo for it. It's going to be in the most recent ps where the Ansible parse fail is23:21
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 878291: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87829123:27
-@gerrit:opendev.org- Ian Wienand proposed on behalf of Clark Boylan: [zuul/zuul-jobs] 878246: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/87824623:30
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:23:43
- [zuul/nodepool] 878179: Use a persistent recursive watch for caches https://review.opendev.org/c/zuul/nodepool/+/878179
- [zuul/nodepool] 877431: Use image cache when launching nodes https://review.opendev.org/c/zuul/nodepool/+/877431
- [zuul/nodepool] 877432: Use node cache in node deleter https://review.opendev.org/c/zuul/nodepool/+/877432
- [zuul/nodepool] 877565: Log the reason we decline a request https://review.opendev.org/c/zuul/nodepool/+/877565
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 878273: Fix file matchers in docker/container jobs https://review.opendev.org/c/zuul/zuul-jobs/+/87827323:56
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 878293: buildx: remove experimental flags https://review.opendev.org/c/zuul/zuul-jobs/+/87829323:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!