| @michael_kelly_anet:matrix.org | corvus: Was poking at the zuul-operator functional test and trying to get it to work with quay.io images. Afaict (based on my limited understanding of how these things work), it appears as though for the *operator* image it's skipping the buildset registry. The only entries I see in the buildset registry log (https://ea85f51ebeafa0e9773b-6f37c7c427c6c0e6779a1a850e4d910b.ssl.cf1.rackcdn.com/910556/5/check/zuul-operator-build-image/c53cc60/docker/buildset_registry.txt) appear to be related to the image build job. Plus the pod created is the `latest` image from quay.io (https://06f2530b99265d05581b-7be76c0c9611893d6bd92b07026b74c2.ssl.cf2.rackcdn.com/910556/5/check/zuul-operator-functional-k8s/113ae0d/describe-pods.txt). It also appears as though for the other bits and pieces (zuul, nodepool, etc) it's first attempting to hit the buildset registry. | 00:34 |
|---|---|---|
| @michael_kelly_anet:matrix.org | Any suggestions as to what might be going on here? | 00:34 |
| @michael_kelly_anet:matrix.org | (or anyone else that's familiar with this stuff, for that matter) | 00:40 |
| @jim:acmegating.com | Michael Kelly: On https://review.opendev.org/881245 patchset 3, i wrote the following: `This is not using the build zuul-operator image. It's likely that something related to minikube + buildset-registry is broken. The next step is probably to switch to using microk8s for testing this since that has seen recent maintenance in use-buildset-registry.` i believe that is one path to getting it working, and worth doing since ianw blazed that trail for other k8s tests. dpawlik took a different approach with having minikube use podman with minikube. i have not evaluated that, though i have not seen 881245 pass even with its dependency. | 01:46 |
| @jim:acmegating.com | put another way: there are working jobs using microk8s, so i expect that switching to that would work and i still consider that the most promising path; but i'm open to something else if someone gets it working. | 01:49 |
| @michael_kelly_anet:matrix.org | Gotcha. I missed that note. I was trying to narrow it down so minimize the moving pieces in the change, not avoid moving it forward in the end. Part of my confusion is that the use of docker.io *does* appear to be captured. | 02:23 |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 06:06 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-operator] 910556: Publish container images to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/910556 | 06:07 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 06:16 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 06:25 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 07:11 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 07:24 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 07:55 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 910746: Only return the latest config for project-branch https://review.opendev.org/c/zuul/zuul/+/910746 | 08:01 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 910746: Only return the latest config for project-branch https://review.opendev.org/c/zuul/zuul/+/910746 | 08:03 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-operator] 910556: Publish container images to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/910556 | 08:24 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 911042: debugging registry config https://review.opendev.org/c/zuul/zuul-jobs/+/911042 | 08:24 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-operator] 910556: Publish container images to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/910556 | 08:24 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 911052: Store pool name in component registry https://review.opendev.org/c/zuul/nodepool/+/911052 | 09:51 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Samuel Jan Surovka: [zuul/nodepool] 908579: Add a new metric, for handleable requests per provider https://review.opendev.org/c/zuul/nodepool/+/908579 | 10:13 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Samuel Jan Surovka: [zuul/nodepool] 908579: Add a new metric, for handleable requests per provider https://review.opendev.org/c/zuul/nodepool/+/908579 | 11:13 | |
| @fungicide:matrix.org | Michael Kelly: also, not sure if it might be related, but if you're specifically using the docker client, from what i understand it can't use an arbitrary registry as a mirror for another arbitrary registry, only as a mirror for actual dockerhub. in opendev, that's caused us to roll back our earlier plan to move our own custom images off of dockerhub, because when we tried it left us unable to use the docker client to do speculative container builds/testing: https://github.com/moby/moby/pull/34319 | 13:22 |
| -@gerrit:opendev.org- Zuul merged on behalf of Benjamin Schanzel: [zuul/nodepool] 901456: Use seprate log package for NodescanRequest messages https://review.opendev.org/c/zuul/nodepool/+/901456 | 15:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Benjamin Schanzel: [zuul/nodepool] 909993: Metastatic: Copy cloud attribute from backing node https://review.opendev.org/c/zuul/nodepool/+/909993 | 16:51 | |
| @michael_kelly_anet:matrix.org | fungi: Bleh. That's annoying. Thanks for the pointer though - it might explain what's going on. It's very strange because in the buildset registry I can definitely see the images being pushed, but when I was looking yesterday it's obvious that nothing is trying to pull the image that I pushed :P | 17:01 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 911163: Add rackspaceauth as a nodepool dependency https://review.opendev.org/c/zuul/nodepool/+/911163 | 17:53 | |
| @clarkb:matrix.org | I don't like ^ but I think it offers the most direct path forward in the immediate future for existing rax users (like opendev) | 17:55 |
| @fungicide:matrix.org | i suppose the alternative would be for opendev to build custom nodepool container images with the additional library injected? | 18:09 |
| @clarkb:matrix.org | fungi: yes we could start building our own images and quicky add the dep to them | 18:12 |
| @clarkb:matrix.org | but we may not be the only rackspace nodepool user so having it in nodepool makes sense | 18:12 |
| @jim:acmegating.com | agreed; if it requires any actual code changes, we may want to subclass the openstack driver and make a new rax driver. hopefully the config is all that's necessary though. | 18:14 |
| @clarkb:matrix.org | the bit I don't like is that we have to use a specail cloud specific auth lib when clouds enable MFA. This hsould be supported by openstack properly | 18:17 |
| @fungicide:matrix.org | seems (based on my rudimentary testing) to be a keystoneauth1 plugin which gets automatically picked up | 18:17 |
| @clarkb:matrix.org | (and clouds that want to add MFA should ensure it does before doing so) | 18:17 |
| @clarkb:matrix.org | fungi: yup I opened the sdist and it installs some keystoneauth1 plugin entrypoints | 18:17 |
| @jim:acmegating.com | hopefully "automatically picked up" doesn't mean "automatically interferes with other openstack clouds" | 18:18 |
| @fungicide:matrix.org | Clark: i think it is supported by openstack, just in different ways than rackspace (who implemented a different solution on their own years ago) | 18:18 |
| @clarkb:matrix.org | fungi: my understanding is that frickler's concern still holds in openstack proper. YOu can MFA with keytone if configured properly but then you're getting a bearer token with a (relatively) short timeout | 18:18 |
| @clarkb:matrix.org | which makes it not useful for nodepool unless you configure nodepool with the 2fa secret then its no different than your situation before | 18:19 |
| @jim:acmegating.com | side note: you could use the intermediate registry image to test that with opendev creds before landing the change. | 18:19 |
| @clarkb:matrix.org | corvus: oh good idea | 18:19 |
| @fungicide:matrix.org | well, i meant openstack/keystone has its own mfa support and it doesn't force people to stop using passwords for api interaction | 18:19 |
| @clarkb:matrix.org | except that docker can't fetch the multiarch images from there anymore and nodepool builder is one of them | 18:20 |
| @clarkb:matrix.org | but I think it will work for launcher | 18:20 |
| @clarkb:matrix.org | > <@fungicide:matrix.org> well, i meant openstack/keystone has its own mfa support and it doesn't force people to stop using passwords for api interaction | 18:20 |
| But it does break applications like nodepool unless you intentionally reduce the value of 2fa to basically zero | ||
| @clarkb:matrix.org | Github seems to haev a reasonableish approach which is scoped long term tokens are fine | 18:22 |
| @clarkb:matrix.org | we also need to determine if the swift log uploads are affected but I think corvus set those up with special internal account that are swift specific and in theory they won't be affected. | 18:47 |
| @clarkb:matrix.org | and if anyone else using swift at rax for logs is affected they can switch to that setup | 18:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!