| @iwienand:matrix.org | ```1364355.930631] Out of memory: Killed process 1321507 (zuul-manage-ans) total-vm:35431628kB,``` | 00:04 |
|---|---|---|
| @iwienand:matrix.org | somehow that managed to get itself killed for oom :/ | 00:04 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: [wip] build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 00:07 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 871539: roles: Add git-submodule-init role https://review.opendev.org/c/zuul/zuul-jobs/+/871539 | 00:07 | |
| -@gerrit:opendev.org- Michael Kelly proposed: | 00:13 | |
| - [zuul/zuul] 869785: manager: Remove unncessary job_graph check from executeJobs() https://review.opendev.org/c/zuul/zuul/+/869785 | ||
| - [zuul/zuul] 869784: manager: Fix TODO in executeJobs() https://review.opendev.org/c/zuul/zuul/+/869784 | ||
| - [zuul/zuul] 869786: model: Eliminate semaphore_handler parameter on findJobsTo*() https://review.opendev.org/c/zuul/zuul/+/869786 | ||
| -@gerrit:opendev.org- Michael Kelly proposed: | 00:14 | |
| - [zuul/zuul] 869785: manager: Remove unnecessary job_graph check from executeJobs() https://review.opendev.org/c/zuul/zuul/+/869785 | ||
| - [zuul/zuul] 869784: manager: Fix TODO in executeJobs() https://review.opendev.org/c/zuul/zuul/+/869784 | ||
| - [zuul/zuul] 869786: model: Eliminate semaphore_handler parameter on findJobsTo*() https://review.opendev.org/c/zuul/zuul/+/869786 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: [wip] build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 00:19 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: Dockerfile: build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 01:03 | |
| @iwienand:matrix.org | corvus: Clark ^ i think that is the final state of it | 01:04 |
| @clarkb:matrix.org | ianw left a couple of thoughts | 01:07 |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 871539: roles: Add git-submodule-init role https://review.opendev.org/c/zuul/zuul-jobs/+/871539 | 01:12 | |
| @iwienand:matrix.org | Clark: would that be a second FROM with the go builder image? | 01:37 |
| @clarkb:matrix.org | Yup and then copy from that image | 01:44 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 871538: Dockerfile: build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 02:27 | |
| @iwienand:matrix.org | > <@clarkb:matrix.org> Yup and then copy from that image | 03:21 |
| ok, the new revision does that and seems to pass ci | ||
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 871539: roles: Add git-submodule-init role https://review.opendev.org/c/zuul/zuul-jobs/+/871539 | 04:53 | |
| -@gerrit:opendev.org- Michael Kelly proposed: [zuul/zuul-jobs] 871539: roles: Add git-submodule-init role https://review.opendev.org/c/zuul/zuul-jobs/+/871539 | 06:02 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 06:39 | |
| - [zuul/zuul] 871106: Require latest layout for processing mgmt events https://review.opendev.org/c/zuul/zuul/+/871106 | ||
| - [zuul/zuul] 871107: Periodically cleanup leaked pipeline state https://review.opendev.org/c/zuul/zuul/+/871107 | ||
| - [zuul/zuul] 871108: Cleanup deleted pipelines and and event queues https://review.opendev.org/c/zuul/zuul/+/871108 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: | 09:36 | |
| - [zuul/zuul] 871106: Require latest layout for processing mgmt events https://review.opendev.org/c/zuul/zuul/+/871106 | ||
| - [zuul/zuul] 871107: Periodically cleanup leaked pipeline state https://review.opendev.org/c/zuul/zuul/+/871107 | ||
| - [zuul/zuul] 871108: Cleanup deleted pipelines and and event queues https://review.opendev.org/c/zuul/zuul/+/871108 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 871012: Use importlib for versioning https://review.opendev.org/c/zuul/zuul/+/871012 | 11:12 | |
| @dpawlik:matrix.org | hello folks. May I ask if someone start nodepool builder in the container without using root user inside the container? | 15:54 |
| @clarkb:matrix.org | I don't think that is currently possible due to the loopback and mounting operations that are performed | 15:58 |
| @dpawlik:matrix.org | thank you Clark ++ | 15:59 |
| @clarkb:matrix.org | corvus: ianw for https://review.opendev.org/c/zuul/zuul/+/871538 I'm not sure how much we exercise the use of skopeo on the docker image in CI. Are we good with landing that and then reverting OpenDev back to 8.1.0 images if necessary? Also how is it that the image build job there succeeded if skopeo is having problems? Is it while talking to a specific registry? | 16:29 |
| @clarkb:matrix.org | I guess we don't upload to dockerhub in check so that may explain it and approving this might fail? | 16:30 |
| @jim:acmegating.com | Clark: we just gated and promoted a change, so i think everything is still working for zuul.... | 16:52 |
| @jim:acmegating.com | Clark: is the issue that the skopeo on the executor doesn't work in the nodepool image build job (because its image build is more complex -- using bulidx) ? | 16:52 |
| @clarkb:matrix.org | yes that could be it. | 16:53 |
| @clarkb:matrix.org | yup the role automatically uses buildx if arch is set and nodepool does that | 16:54 |
| @clarkb:matrix.org | I think that may be the underlying cause | 16:54 |
| @clarkb:matrix.org | * I think that may be the underlying trigger | 16:54 |
| @jim:acmegating.com | Clark: and yeah, i don't think we exercise it at all in ci. so i think probably the best way forward is to merge it (and expect it to merge since things seem to be working); restart opendev; and then check the nodepool changes that failed (let's identify one). if anything goes wrong, revert to 8.1.0 and then revert the change. | 16:55 |
| @jim:acmegating.com | * Clark: and yeah, i don't think we exercise it at all in ci. so i think probably the best way forward is to merge it (and expect it to merge since things seem to be working); restart opendev; and then check the nodepool changes that failed (let's identify one). if anything goes wrong, revert opendev to 8.1.0 and then revert the change. | 16:55 |
| @clarkb:matrix.org | ack I'll go ahead and approve now | 16:55 |
| @jim:acmegating.com | Clark: https://review.opendev.org/871409 looks to be the thing to recheck | 16:56 |
| @jim:acmegating.com | (and it shows the failure in check which makes testing easier) | 16:57 |
| -@gerrit:opendev.org- Alex Hornung proposed: [zuul/zuul] 871100: gerrit driver: fix bug around unicode branch names https://review.opendev.org/c/zuul/zuul/+/871100 | 17:09 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [zuul/zuul-website] 869756: Remove the Summit CFP banner now that it is closed https://review.opendev.org/c/zuul/zuul-website/+/869756 | 18:43 | |
| @iwienand:matrix.org | > <@jim:acmegating.com> Clark: is the issue that the skopeo on the executor doesn't work in the nodepool image build job (because its image build is more complex -- using bulidx) ? | 19:00 |
| yep it is buildx 0.10.0 that causes the extra signing tag to be added | ||
| @iwienand:matrix.org | i have only just pulled it up and it is probably a config error, but seems linaro launches are failing with ```TypeError: create_server() requires either 'image' or 'boot_volume'``` -> https://paste.opendev.org/show/bRhxa0ix8C982EI5jypb/ ; this ring any bells? | 19:10 |
| @jim:acmegating.com | ianw: can we switch to #_oftc_#opendev:matrix.org | 19:15 |
| @jim:acmegating.com | ftr, tracked this down to a mismatched config, but also we can probably catch the error and log it explicitly for better aids to operators | 19:38 |
| -@gerrit:opendev.org- Joshua Watt proposed: [zuul/zuul-jobs] 871657: doc: docker-image: Add recommended dependency https://review.opendev.org/c/zuul/zuul-jobs/+/871657 | 19:58 | |
| @jpew:matrix.org | is job.match-on-config-update supposed to match changes to local variant? For example: | 20:40 |
| ``` | ||
| - project: | ||
| check: | ||
| - foo: | ||
| files: ["bar"] | ||
| ``` | ||
| doesn't seem to rerun the `foo` job if I change anything about it in the pipeline local definition | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/zuul] 871538: Dockerfile: build later skopeo https://review.opendev.org/c/zuul/zuul/+/871538 | 21:10 | |
| @iwienand:matrix.org | > <@jim:acmegating.com> ftr, tracked this down to a mismatched config, but also we can probably catch the error and log it explicitly for better aids to operators | 21:26 |
| so now linaro doesn't seem to be accepting anything with ```2023-01-24 21:25:54,829 DEBUG nodepool.PoolWorker.osuosl-regionone-main: Handler is paused, deferring request``` | ||
| @iwienand:matrix.org | i think what happened here is another combination of things. i had the ram limit set at 51200 | 21:27 |
| @iwienand:matrix.org | it seems like it accepted a few nodes to start until it thought that smallish quota had run out | 21:27 |
| @iwienand:matrix.org | those nodes all failed to launch with that error | 21:28 |
| @iwienand:matrix.org | afaics, i don't think nodepool has now realised that those resources were never used, so it is just sitting paused | 21:28 |
| @iwienand:matrix.org | hang on, i'm wrong, that's osuosl | 21:38 |
| @iwienand:matrix.org | not linaro | 21:38 |
| @iwienand:matrix.org | ... but ... linaro doesn't seem to want to pick up node requests | 21:38 |
| -@gerrit:opendev.org- Joshua Watt proposed: [zuul/zuul-jobs] 869760: use-buildset-registry: Prepend buildset registry to mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/869760 | 21:40 | |
| -@gerrit:opendev.org- Joshua Watt proposed: | 21:54 | |
| - [zuul/zuul-jobs] 871664: upload-container-image: Add option to stage in separate repository https://review.opendev.org/c/zuul/zuul-jobs/+/871664 | ||
| - [zuul/zuul-jobs] 871665: promote-artifactory-image: Add role https://review.opendev.org/c/zuul/zuul-jobs/+/871665 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 871671: Refresh dependencies for changes in pipelines https://review.opendev.org/c/zuul/zuul/+/871671 | 23:35 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!