| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | 00:30 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849593: [wip] upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | 01:04 | |
| -@gerrit:opendev.org- Ian Wienand proposed: | 01:20 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849593: [wip] upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849593: [wip] upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | 01:38 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | 01:47 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 01:52 | |
| -@gerrit:opendev.org- Ian Wienand proposed: | 01:57 | |
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 02:04 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 02:32 | |
| -@gerrit:opendev.org- Ian Wienand proposed: | 02:50 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: | 04:02 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: | 04:27 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: | 05:03 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| -@gerrit:opendev.org- Ian Wienand proposed: | 05:18 | |
| - [zuul/zuul-jobs] 849589: upload-pypi: support API token upload https://review.opendev.org/c/zuul/zuul-jobs/+/849589 | ||
| - [zuul/zuul-jobs] 849598: ensure-twine: make python3 default, ensure pip installed https://review.opendev.org/c/zuul/zuul-jobs/+/849598 | ||
| - [zuul/zuul-jobs] 849593: upload-pypi: basic testing https://review.opendev.org/c/zuul/zuul-jobs/+/849593 | ||
| - [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | ||
| @iwienand:matrix.org | do other people see "Gerrit Code Review" and a broken image logo as "previews" of changes? | 06:04 |
|---|---|---|
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 06:07 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 06:20 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: [wip] test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 06:27 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: upload-pypi: test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 07:01 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: upload-pypi: test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 08:34 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul-jobs] 849597: upload-pypi: test sandbox upload https://review.opendev.org/c/zuul/zuul-jobs/+/849597 | 08:53 | |
| @avass:vassast.org | I dug some more and I think we're seeing this issue: https://sourceware.org/bugzilla/show_bug.cgi?id=24941 | 11:02 |
| it looks like that is fixed in glibc 2.33, but the zuul containers are using 2.31 | ||
| @avass:vassast.org | from what I understand the other solution is to stop using fork in ansible and instead use forkserver, or spawn: https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/multiprocessing.py#L17 | 11:17 |
| -@gerrit:opendev.org- Artem Goncharov proposed: | 12:15 | |
| - [zuul/zuul] 849033: Initial implementation of the gitea driver https://review.opendev.org/c/zuul/zuul/+/849033 | ||
| - [zuul/zuul] 849670: Improve gitea trigger testing https://review.opendev.org/c/zuul/zuul/+/849670 | ||
| @bingberg:matrix.org | What is the logic behind where you should put the playbooks? | 13:36 |
| We have a config repo with the playbook "playbooks/integration-test.yml" and a job set to "run: playbooks/integration-test.yml" however when running the gate for our project top1 we get: | ||
| Error Unable to find playbook /var/.../top1/playbooks/integration-test.yml | ||
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849680: Implement gitea reporter interface https://review.opendev.org/c/zuul/zuul/+/849680 | 13:41 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849680: Implement gitea reporter interface https://review.opendev.org/c/zuul/zuul/+/849680 | 13:42 | |
| @jim:acmegating.com | bingberg: they should be in the repo where the job definition references them. | 13:50 |
| @avass:vassast.org | There's no way to gracefully shutdown nodepool right? It looks like just killing that has a small chance to cause `NODE_ERROR` for jobs. | 14:20 |
| @avass:vassast.org | * There's no way to gracefully shutdown nodepool right? It looks like just killing it that has a small chance to cause `NODE_ERROR` for jobs. | 14:21 |
| @avass:vassast.org | * There's no way to gracefully shutdown nodepool right? It looks like just killing it has a small chance to cause `NODE_ERROR` for jobs. | 14:21 |
| @jpew:matrix.org | Is there a way to set the disk space limit per job instead of per-executor? | 15:42 |
| @jpew:matrix.org | Or anything finer grained that per-executor for that matter | 15:43 |
| @jpew:matrix.org | * Or anything finer grained than the executor for that matter | 15:44 |
| @clarkb:matrix.org | jpew: the disk space limits should be per build on the executor. However, they run with a slight delay as the limiter has to scan the build dirs | 15:59 |
| @clarkb:matrix.org | > <@avass:vassast.org> There's no way to gracefully shutdown nodepool right? It looks like just killing it has a small chance to cause `NODE_ERROR` for jobs. | 16:00 |
| OpenDev has always simply restarted them. In fact we're doing that automatically now and I'm not sure we've seen any `NODE_ERROR`s. I think the expectation is that this is always safe because nodepool hands off to zuul then zuul locks things and that keeps the paths open? You do potentially lose active boots though that have to be deleted when nodepool starts again | ||
| @jpew:matrix.org | Clark: Ya, I was just wondering if the limit could be controller with a finer grain. We have some builds that need a lot, and some that don't | 16:03 |
| @jpew:matrix.org | But, perhaps given what you've said the implementation of that would be hard | 16:03 |
| @avass:vassast.org | > <@clarkb:matrix.org> OpenDev has always simply restarted them. In fact we're doing that automatically now and I'm not sure we've seen any `NODE_ERROR`s. I think the expectation is that this is always safe because nodepool hands off to zuul then zuul locks things and that keeps the paths open? You do potentially lose active boots though that have to be deleted when nodepool starts again | 16:26 |
| We've seen it happen during one restart for the last six days. So it's very rare at least. I can probably dog into the logs tomorrow and see what happened. | ||
| @avass:vassast.org | * We've seen it happen during one restart for the last six days. So it's very rare at least. I can probably dig into the logs tomorrow and see what happened. | 16:26 |
| @avass:vassast.org | That it was only three jobs that were affected by it. | 16:27 |
| @avass:vassast.org | * It was only three jobs that were affected by it. | 16:30 |
| -@gerrit:opendev.org- Artem Goncharov proposed: | 16:32 | |
| - [zuul/zuul] 849719: Implement gitea commit status reporter https://review.opendev.org/c/zuul/zuul/+/849719 | ||
| - [zuul/zuul] 849720: Implement gitea comment trigger https://review.opendev.org/c/zuul/zuul/+/849720 | ||
| @jim:acmegating.com | it depends on how many providers you have; more would allow for unclean shutdowns. | 16:32 |
| @avass:vassast.org | There's only the one :) | 16:33 |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849033: Initial implementation of the gitea driver https://review.opendev.org/c/zuul/zuul/+/849033 | 16:35 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: | 18:48 | |
| - [zuul/zuul] 849727: Add gitea pr dyn_reconf test https://review.opendev.org/c/zuul/zuul/+/849727 | ||
| - [zuul/zuul] 849728: Implement gitea refs https://review.opendev.org/c/zuul/zuul/+/849728 | ||
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849670: Improve gitea trigger testing https://review.opendev.org/c/zuul/zuul/+/849670 | 18:50 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849680: Implement gitea reporter interface https://review.opendev.org/c/zuul/zuul/+/849680 | 18:50 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849719: Implement gitea commit status reporter https://review.opendev.org/c/zuul/zuul/+/849719 | 18:50 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849720: Implement gitea comment trigger https://review.opendev.org/c/zuul/zuul/+/849720 | 18:51 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849727: Add gitea pr dyn_reconf test https://review.opendev.org/c/zuul/zuul/+/849727 | 18:51 | |
| -@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 849728: Implement gitea refs https://review.opendev.org/c/zuul/zuul/+/849728 | 18:51 | |
| @iwienand:matrix.org | btw the nodepool/dib/cgroups v2/containerfile thing had a fairly uninspiring conclusion so far of "have you considered running systemd in the container" so far :/ | 23:07 |
| @clarkb:matrix.org | ianw: is there a tldr to what the problem is? docker runs with one version of cgroups and podman another? | 23:08 |
| @mordred:inaugust.com | > <@iwienand:matrix.org> btw the nodepool/dib/cgroups v2/containerfile thing had a fairly uninspiring conclusion so far of "have you considered running systemd in the container" so far :/ | 23:09 |
| no, I have not considered running systemd in the container. One of the nice things about containers is that I don't have to consider systemd inside of them. | ||
| @iwienand:matrix.org | Clark: not really; https://github.com/containers/podman/issues/14884 is closed now but it's not clear that it was actually fixed for our use case of a daemon forking a process that calls podman | 23:09 |
| @iwienand:matrix.org | i will get back to it. i think that "cat /sys/fs/cgroup/cgroup.procs | sudo tee /sys/fs/cgroup/init/cgroup.procs" is the wrong way to move processes; maybe. maybe you have to do individual writes of the pids, not all of them together | 23:12 |
| @iwienand:matrix.org | (that was another suggestion) | 23:12 |
| @clarkb:matrix.org | I'm a bit confused about that because I thought that all docker was doing was running processes in new cgroups | 23:12 |
| @clarkb:matrix.org | we run with host networking so that is part of the root cgroup but the user and process spaces should be new? | 23:13 |
| @clarkb:matrix.org | and it seems that the cgroup in question is the process cgroup based on the other issue | 23:13 |
| @iwienand:matrix.org | honestly i don't have enough depth on the problem to agree or disagree, as it's only a cgroups v2 thing, so something to do with unified hierarchies etc. | 23:14 |
| @clarkb:matrix.org | oh right v2. Its possible my understanding is v1 based (the lats time I manually edited cgroups was with v1) | 23:15 |
| @clarkb:matrix.org | rereading the other issue the cgroup inspection is expected to be done in the container context not the host so root here is whatever the container cgroup is? | 23:21 |
| @clarkb:matrix.org | https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#processes is helpful for anyone else reading those issues | 23:25 |
| @clarkb:matrix.org | ianw: if I'm reading that correctly the suggestion is basically to move dib (or init/systemd) into a new sub cgroup | 23:28 |
| @clarkb:matrix.org | In our case I think we wouldn't need to use systemd if we can just move dib. Though something like systemd likely does it automatically (amazing that we'd need ti though) | 23:28 |
| @iwienand:matrix.org | yeah i think there are other ways to move it. perhaps if the element calling podman moves its shell to a separate new cgroup. feels hacky | 23:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!