| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 871409: Update to DIB 3.26.0 https://review.opendev.org/c/zuul/nodepool/+/871409 | 03:21 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 08:16 | |
| - [zuul/zuul] 871106: Require latest layout for processing mgmt events https://review.opendev.org/c/zuul/zuul/+/871106 | ||
| - [zuul/zuul] 871107: Periodically cleanup leaked pipeline state https://review.opendev.org/c/zuul/zuul/+/871107 | ||
| - [zuul/zuul] 871108: Cleanup deleted pipelines and and event queues https://review.opendev.org/c/zuul/zuul/+/871108 | ||
| @iwienand:matrix.org | ```time="2023-01-23T07:20:00Z" level=fatal msg="Error creating an updated image manifest: Error preparing updated manifest, layer \"sha256:a03e79f3ec8e6259083b6d659789979d8a13a30af294845fb5992bb30aa51cdd\": unsupported MIME type for compression: application/vnd.in-toto+json"``` | 08:40 |
|---|---|---|
| @iwienand:matrix.org | https://zuul.opendev.org/t/zuul/build/9b81ac96201f4f709e9190da4f24bc9f - nodepool-build-image | 08:40 |
| @iwienand:matrix.org | i don't have time to look right now, but i haven't seen that one before | 08:40 |
| @iwienand:matrix.org | https://github.com/in-toto/attestation/tree/main/spec#statement | 08:43 |
| @iwienand:matrix.org | maybe it is something new? | 08:43 |
| -@gerrit:opendev.org- Per Wiklund marked as active: [zuul/nodepool] 871102: Introduce driver for openshift virtualmachines https://review.opendev.org/c/zuul/nodepool/+/871102 | 08:55 | |
| @jpew:matrix.org | fungi: Ya, I've got a shell script + ansible script that selects the machine with the longest uptime, makes sure everything on the cluster is settled (mostly our longhorn persistent volumes), does all the update stuff for a node (updates packages, etc.) then restarts it. I'd automate it with Zuul, but then zuul would potentially restart the executor the job was running on and cause havoc :) | 14:29 |
| @fungicide:matrix.org | right, we have a management bastion with a cronjob on it that runs ansible-playbook on https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul_reboot.yaml under a lock with output redirected to a log | 14:36 |
| @fungicide:matrix.org | however, we do deploy the configuration which adds that cronjob with this ansible playbook via a zuul job: https://opendev.org/opendev/system-config/src/branch/master/playbooks/service-bridge.yaml#L44-L55 | 14:39 |
| @fungicide:matrix.org | so in that sense, it's automated (indirectly) by zuul | 14:40 |
| @jpew:matrix.org | Ah, nice | 14:40 |
| @fungicide:matrix.org | we keep essentially all of our server configuration in ansible plays/roles and use nested ansible run from a persistent bastion with zuul's checkouts of our configuration repositories pushed to it, as a way to implement cd via zuul jobs | 14:43 |
| @fungicide:matrix.org | which also makes it really easy to test our server configs, since zuul can run multi-node check pipeline jobs with temporary deployments of all the relevant server configuration, just with dummy values substituted for any sensitive data/credentials | 14:45 |
| @fungicide:matrix.org | and we can even perform complex integration testing between those temporary substitute servers, in order to make sure that a proposed configuration change for one service doesn't break another service that communicates with it | 14:48 |
| @jpew:matrix.org | OK, I'm trying to write a role to promote docker images in artifactory (like the promote-docker-image role), but there is something I don't understand: How does the post pipeline get the `zuul.change` value to know which tag to promote? In my post pipeline, `zuul.change` is not set do it defaults to "post" (the pipeline name) | 16:01 |
| @jpew:matrix.org | I feel like I'm missing something in this workflow, but I can't quite figure out what | 16:02 |
| @clarkb:matrix.org | jpew: its done via the zuul artifact iirc | 16:16 |
| @jpew:matrix.org | Clark: Sorry, I don't follow what that means | 16:17 |
| @jpew:matrix.org | Can an artifact be transfered between pipelines or something? | 16:18 |
| @clarkb:matrix.org | oh wait no, these jobs are expected to run in a merge pipeline where the change value is set | 16:21 |
| @clarkb:matrix.org | ya if you look at opendev's pipeline definitions post is a ref-updated pipeline and we have a separate promote pipeline that triggers on change-merged. The change info comes from the change-merged events | 16:22 |
| @jpew:matrix.org | Ah! I got it | 16:22 |
| @jpew:matrix.org | promote != post | 16:22 |
| @jim:acmegating.com | the promote pipeline operates on the change, so its triggered by change-merged; post operates on the resulting repo state (where we want builds to have the exact content and sha of the repo) so it's triggered by ref-updated. promoted artifacts should be functionally equivalent, but they won't have been built from a sha actually contained in the repo, so that's noteworthy distinction. | 16:28 |
| @jpew:matrix.org | Right | 16:28 |
| @jpew:matrix.org | Also, I assume that if I forced push outside of gerrit, `post` would run, but `promote` wouldn't | 16:29 |
| @jpew:matrix.org | Or just regular push outside of gerrit | 16:29 |
| @jim:acmegating.com | jpew: if gerrit recognizes that the push closes an open change, it may mark that change and closed; i don't know if it would emit an event. i would suggest experimentally validating that if it's important. | 16:31 |
| @jim:acmegating.com | * jpew: if gerrit recognizes that the push closes an open change, it may mark that change as closed; i don't know if it would emit an event. i would suggest experimentally validating that if it's important. | 16:31 |
| @jim:acmegating.com | * jpew: if gerrit recognizes that the push closes an open change, it may mark that change as closed; i don't know if it would emit an event for the closure. i would suggest experimentally validating that if it's important. | 16:32 |
| @jim:acmegating.com | but post would definitely run yes | 16:32 |
| @jpew:matrix.org | It's not thankfully :) | 16:32 |
| @fungicide:matrix.org | probably also worth reminding that our example image uploading and promotion workflow is that we upload the images pre-merge with gate pipeline jobs, and then simply tag them post-merge to indicate that they're intended for use | 16:35 |
| @jpew:matrix.org | fungi: yep. I'm trying to get that to work with artifactory instead of docker hub. I'm close I think | 16:36 |
| @fungicide:matrix.org | the main down-side to that model is that at the time of upload we don't know what the resulting git merge commit will be in the final repository state since it doesn't exist yet | 16:36 |
| @jpew:matrix.org | Ya, so the SHA-1 is useless | 16:37 |
| @jpew:matrix.org | * Ya, so the SHA-1 is useless? | 16:37 |
| @fungicide:matrix.org | if we ever get to the point of zuul pushing its constructed merges instead of telling the review system to merge, that could be solved i think | 16:37 |
| @clarkb:matrix.org | It would also be fine in a fast forward only repo but that has lots of other impacts (many of which negate many of the benefits of zuul) | 16:48 |
| @jpew:matrix.org | Hey it worked! | 17:07 |
| @jpew:matrix.org | Awesome | 17:07 |
| @namrata03:matrix.org | Hi, I am trying to upload a file to S3 bucket and using secret data for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. | 17:08 |
| ``` | ||
| - secret: | ||
| name: environment_secret | ||
| data: | ||
| AWS_ACCESS_KEY_ID: <redacted> | ||
| AWS_SECRET_ACCESS_KEY: <redacted> | ||
| ``` | ||
| And using aws cli in tox.ini as the following: | ||
| ``` | ||
| deps = awscli | ||
| passenv = AWS_* | ||
| setenv = | ||
| file|build.env | ||
| commands = | ||
| aws --endpoint-url={env:S3_ENDPOINT} \ | ||
| s3 sync \ | ||
| --content-type application/pdf \ | ||
| {toxinidir}/build/pdf/ \ | ||
| s3://{env:S3_BUCKET}/ | ||
| ``` | ||
| But I am unable to passenv, I am getting error "unable to locate credenials". Can someone please help me how to use the secret as environment variable? | ||
| @namrata03:matrix.org | * Hi, I am trying to upload a file to S3 bucket and using secret data for AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY. | 17:09 |
| ``` | ||
| - secret: | ||
| name: environment_secret | ||
| data: | ||
| AWS_ACCESS_KEY_ID: <redacted> | ||
| AWS_SECRET_ACCESS_KEY: <redacted> | ||
| ``` | ||
| And using aws cli in tox.ini as the following: | ||
| ```` | ||
| deps = awscli | ||
| passenv = AWS_* | ||
| setenv = | ||
| file|build.env | ||
| commands = | ||
| aws --endpoint-url={env:S3_ENDPOINT} \ | ||
| s3 sync \ | ||
| --content-type application/pdf \ | ||
| {toxinidir}/build/pdf/ \ | ||
| s3://{env:S3_BUCKET}/ | ||
| ``` | ||
| But I am unable to passenv, I am getting error "unable to locate credenials". Can someone please help me how to use the secret as environment variable? | ||
| @clarkb:matrix.org | Namrata Sitlani: you need something in your job that takes the secret info and injects it into the environment of your tox process | 17:12 |
| @namrata03:matrix.org | Clark: Can you point me to an example or any documentation where I can get the reference for the same. I am not able to figure out how to do that | 17:14 |
| @clarkb:matrix.org | I don't know if any examples off the top of my head. But the secret is part of the job and you need to make it part of the Ansible the job runs. You'll need to look at setting the Ansible task/role environment | 17:20 |
| @clarkb:matrix.org | https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_environment.html | 17:20 |
| @namrata03:matrix.org | I am new to zuul , so pardon me if I am asking basic questions . My .zuul.yaml looks like following: | 17:24 |
| @namrata03:matrix.org | ``` | 17:24 |
| @namrata03:matrix.org | * ``` | 17:24 |
| --- | ||
| - secret: | ||
| name: environment_secret | ||
| data: | ||
| AWS_ACCESS_KEY_ID: <redacted> | ||
| AWS_SECRET_ACCESS_KEY: <redacted> | ||
| - job: | ||
| name: tox-upload | ||
| parent: tox-pandoc | ||
| secrets: | ||
| - name: tox_environment | ||
| secret: environment_secret | ||
| vars: | ||
| tox_envlist: "build,upload" | ||
| - project: | ||
| check: | ||
| jobs: | ||
| - tox-pandoc | ||
| gate: | ||
| jobs: | ||
| - tox-pandoc | ||
| promote: | ||
| jobs: | ||
| - tox-upload | ||
| ``` | ||
| @namrata03:matrix.org | so if I understand correctly I should set the environment variable in .zuul.yaml and tox will be able to pick that env | 17:25 |
| @clarkb:matrix.org | If you are doing this to upload the results of pandoc builds you might be better off using https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-s3. But yes you'd either edit your .zuul.yaml or your ansible playbooks (or maybe both) | 17:28 |
| @namrata03:matrix.org | Clark: Thanks! I will look into thi further | 18:33 |
| @namrata03:matrix.org | * Clark: Thanks! I will look into this further | 18:43 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 871012: Use importlib for versioning https://review.opendev.org/c/zuul/zuul/+/871012 | 18:51 | |
| @clarkb:matrix.org | corvus: not the end of the world but did the openstack driver statemachine rewrite change how hostnames are set? Looks like they are `'np' + nodeid` now? | 19:25 |
| @clarkb:matrix.org | I'm guessing thats something done because other systems don't allow hostnames as long as openstack does | 19:25 |
| @jim:acmegating.com | Clark: yes, that's now normalized with the other drivers and windows is the limiting factor there | 19:31 |
| @clarkb:matrix.org | Ack | 19:31 |
| @iwienand:matrix.org | > <@iwienand:matrix.org> ```time="2023-01-23T07:20:00Z" level=fatal msg="Error creating an updated image manifest: Error preparing updated manifest, layer \"sha256:a03e79f3ec8e6259083b6d659789979d8a13a30af294845fb5992bb30aa51cdd\": unsupported MIME type for compression: application/vnd.in-toto+json"``` | 21:19 |
| ok; the last time https://zuul.opendev.org/t/zuul/builds?job_name=nodepool-build-image&project=zuul/nodepool worked (https://zuul.opendev.org/t/zuul/build/0b8d716c74d44e639529e1ffe300d8c9/console) it was using docker 20.10.22 ... the failing builds (e.g. https://zuul.opendev.org/t/zuul/build/9b81ac96201f4f709e9190da4f24bc9f/console) have 20.10.23 | ||
| @clarkb:matrix.org | ianw: which bit of software is complaining about the mime type? Is it the zuul registry? | 21:20 |
| @iwienand:matrix.org | i'm not 100% sure ... yet :) | 21:20 |
| @iwienand:matrix.org | https://docs.docker.com/engine/release-notes/ does not have a 20.10.23 ... | 21:21 |
| @iwienand:matrix.org | i guess it's https://github.com/moby/moby/releases/tag/v20.10.23 | 21:25 |
| @iwienand:matrix.org | (which doesn't show anything super obvious about this) | 21:27 |
| @iwienand:matrix.org | > <@clarkb:matrix.org> ianw: which bit of software is complaining about the mime type? Is it the zuul registry? | 21:28 |
| i guess it is skopeo copying it here | ||
| @iwienand:matrix.org | ... and yeah ... | 21:29 |
| @iwienand:matrix.org | https://github.com/containers/skopeo/issues/1874 | 21:29 |
| @clarkb:matrix.org | https://github.com/in-toto/docs/blob/master/in-toto-spec.md is the underlying protocol | 21:29 |
| @clarkb:matrix.org | that buildx release is a couple weeks old. I wonder why we didn't see this sooner? I mean we just did nodepool and zuul releases which udpated images iirc | 21:30 |
| @iwienand:matrix.org | skopeo comes from the kubic repos ... | 21:31 |
| @iwienand:matrix.org | actually not on jammy. it's included there | 21:32 |
| @clarkb:matrix.org | phew | 21:32 |
| @clarkb:matrix.org | its also in debian bullseye (where we install it for the zuul executors | 21:33 |
| @iwienand:matrix.org | huh, so we must be using the executor version, it's not installed anywhere | 21:36 |
| @iwienand:matrix.org | Package: skopeo (1.2.2+dfsg1-1 and others) | 21:37 |
| @iwienand:matrix.org | ... which is a long way behind ```That’s not expected with ≥ 1.9.0 AFAICS.``` | 21:37 |
| @iwienand:matrix.org | this is why we can't have nice things | 21:38 |
| @clarkb:matrix.org | Ya I'm guessing no one really consdiered compatibility with clients when making that update | 21:39 |
| @clarkb:matrix.org | would've been nice if they only sent the attestation info if the client indicates it is an acceptable type | 21:39 |
| @iwienand:matrix.org | there's no backport, but testing/unstable at 1.9.3 https://tracker.debian.org/pkg/skopeo | 21:40 |
| @clarkb:matrix.org | Is that the sort of thing that should be suggested for the backports repo considering it seems necessary now? | 21:41 |
| @clarkb:matrix.org | (I don't know how that works) | 21:41 |
| @iwienand:matrix.org | we could suggest it, but it's probably the type of thing that takes weeks more than hours | 21:41 |
| @iwienand:matrix.org | it's not in kubic afaics -- i was thinking maybe that still builds a version | 21:43 |
| @iwienand:matrix.org | Skopeo container images are available at quay.io/skopeo/stable:latest ... | 21:46 |
| @clarkb:matrix.org | we're already running in a container in that context though so that might get tricky | 21:47 |
| @clarkb:matrix.org | would be a third level of nesting | 21:47 |
| @iwienand:matrix.org | yep :/ dib's container-in-container efforts has not been smooth | 21:47 |
| @iwienand:matrix.org | sigh ... https://salsa.debian.org/go-team/packages/skopeo/-/blob/master/debian/changelog | 21:51 |
| @iwienand:matrix.org | it doesn't seem the 1.9.3 changes went through salsa | 21:51 |
| @iwienand:matrix.org | i'm trying to build it in bullseye -- i don't think this actually needs later deps | 22:16 |
| @iwienand:matrix.org | ... that's wrong. a whole bunch of things have moved around | 22:46 |
| @iwienand:matrix.org | it needs a more recent version of golang-github-containers-common-dev, which then explodes into a whole bunch of packages | 22:47 |
| @iwienand:matrix.org | i *can* build it "standalone" ... which might be good enough | 22:54 |
| @iwienand:matrix.org | it seems like the binary only relies on packaged libraries | 22:55 |
| @iwienand:matrix.org | ``` | 22:55 |
| @iwienand:matrix.org | ```# ldd ./bin/skopeo | 22:55 |
| linux-vdso.so.1 (0x00007ffff9d38000) | ||
| libgpgme.so.11 => /usr/lib/x86_64-linux-gnu/libgpgme.so.11 (0x00007f72b3442000) | ||
| libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f72b3420000) | ||
| libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f72b341a000) | ||
| libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007f72b33ad000) | ||
| libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f72b31d8000) | ||
| libassuan.so.0 => /usr/lib/x86_64-linux-gnu/libassuan.so.0 (0x00007f72b31c2000) | ||
| libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f72b319a000) | ||
| /lib64/ld-linux-x86-64.so.2 (0x00007f72b4b19000) | ||
| libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f72b316e000) | ||
| libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x00007f72b3146000) | ||
| libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f72b3002000) | ||
| libpcre2-8.so.0 => /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f72b2f6a000)``` | ||
| @iwienand:matrix.org | so, even though it would be "binary blob" -- that blob was built on bullseye against bullseye libraries. so it's consistent | 22:56 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: [wip] build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 23:04 | |
| @jim:acmegating.com | ianw: want to put that in the dockerfile maybe? | 23:09 |
| @iwienand:matrix.org | corvus: yeah, i was considering passing it in as an artifact. will it blow out the docker layer sizes? | 23:15 |
| @iwienand:matrix.org | i think that "make bin/skopeo" actually ends up pulling in half of github ... but i guess that's no different to any sort of nodejs build etc. | 23:16 |
| @clarkb:matrix.org | That is why we use the throwaway build image | 23:19 |
| @clarkb:matrix.org | Then you copy only the end result binary off that and don't inflate the distributed image too much | 23:20 |
| @iwienand:matrix.org | Clark: yeah -- i wasn't sure about another run layer @ https://opendev.org/zuul/zuul/src/branch/master/Dockerfile#L50 | 23:24 |
| @iwienand:matrix.org | i guess just keep adding & \ lines ... :/ | 23:25 |
| @clarkb:matrix.org | A new layer is fine too since it is discarded | 23:25 |
| @jim:acmegating.com | i'd just build the binary in a builder image then copy it over | 23:26 |
| @iwienand:matrix.org | it is a real comedown from writing separated ansible tasks to string-concatenated shell blobs. but yeah, ok, i'll try that | 23:26 |
| @jim:acmegating.com | no need to worry too much about the layer size in a throwaway builder | 23:27 |
| @iwienand:matrix.org | it might be possible to use the debian go to build it, which would be one less thing | 23:27 |
| @jim:acmegating.com | ianw: ++ | 23:27 |
| @jim:acmegating.com | ianw: here's a very similar construct (that we didn't end up using: https://etherpad.opendev.org/p/A1uVsOScponsLK8jUHUG | 23:29 |
| @jim:acmegating.com | but that builder image can be anything -- like we use the node builder for js stuff | 23:31 |
| @iwienand:matrix.org | ``` | 23:32 |
| @iwienand:matrix.org | ```vendor/github.com/containers/image/v5/pkg/sysregistriesv2/system_registries_v2.go:5:2: cannot find package "." in: | 23:32 |
| /tmp/skopeo-1.9.3/vendor/io/fs | ||
| ``` | ||
| @iwienand:matrix.org | ```vendor/github.com/containers/image/v5/pkg/sysregistriesv2/system_registries_v2.go:5:2: cannot find package "." in: | 23:32 |
| /tmp/skopeo-1.9.3/vendor/io/fs | ||
| ``` | ||
| @iwienand:matrix.org | ``` | 23:32 |
| vendor/github.com/containers/image/v5/pkg/sysregistriesv2/system_registries_v2.go:5:2: cannot find package "." in: | ||
| /tmp/skopeo-1.9.3/vendor/io/fs | ||
| ``` | ||
| @iwienand:matrix.org | that was actually a similar error i got when trying to backport the package | 23:33 |
| @iwienand:matrix.org | that's using debian's packaged go. for whatever reason, it doesn't happen with the upstream blob go | 23:33 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: [wip] build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 23:46 | |
| @iwienand:matrix.org | ``` | 23:49 |
| $ react-app-rewired --max_old_space_size=4096 build | ||
| Failed to compile. | ||
| EEXIST: file already exists, mkdir '/tmp/src/build' | ||
| ``` | ||
| @iwienand:matrix.org | for some reason, locally the dockerfile doesn't build ... | 23:49 |
| @clarkb:matrix.org | are you using podman and buildah? I wonder ifthey treat that as a failure | 23:52 |
| @iwienand:matrix.org | this is docker on fedora, always an interesting combo | 23:55 |
| @jim:acmegating.com | ianw: that can happen if you've built the js stuff locally due to some symlinks it sets up. try iit in a clean git repo. | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!