| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 851941: AWS: Use snapshot instead of image import https://review.opendev.org/c/zuul/nodepool/+/851941 | 00:29 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 849273: Dockerfile: move into separate group when running under cgroupsv2 https://review.opendev.org/c/zuul/nodepool/+/849273 | 00:52 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 851942: docs: update console streaming docs https://review.opendev.org/c/zuul/zuul/+/851942 | 02:00 | |
| @bingberg:matrix.org | So we have a situation where we have a dependency projects `dep1`, `dep2` etc and several consumer projects `con1`, `con2` etc. We wish to gate changes in the dependency projects such that the consumer projects will be compatible with any changes in the dependency projects | 09:30 |
|---|---|---|
| In regular zuul configuration this could be modeled by the `dep1` project having a .zuul.yaml file which triggers the integration tests specified by a template e.g. | ||
| // .zuul.yaml in dep1 | ||
| project: | ||
| templates: | ||
| - con1-integration-tests | ||
| // .zuul.yaml in dep2 | ||
| project: | ||
| templates: | ||
| - con1-integration-tests | ||
| - con2-integration-tests | ||
| The consumer projects would then have the definition of their respective integration tests and release branches with different dependencies than master would work properly. | ||
| Due to several reasons we would like the dependency configuration to be wholly located inside of the consumer projects, i.e. we would instead like to write something like: | ||
| // .zuul.yaml in con1 | ||
| project: | ||
| name: dep1 | ||
| templates: | ||
| - con1-integration-tests | ||
| project: | ||
| name: dep2 | ||
| templates: | ||
| - con1-integration-tests | ||
| // .zuul.yaml in con2 | ||
| project: | ||
| name: dep2 | ||
| templates: | ||
| - con2-integration-tests | ||
| This is not allowed since untrusted projects can't modify other project configurations. Changing the consumer projects to a config-project (or writing it to a separate config project) is also not an option since config projects only examines the master branch and therefore release branches would not be running with their dependencies. | ||
| Does anyone have an idea how we could achieve this without exposing the dependents in the dependency projects? | ||
| @bingberg:matrix.org | * So we have a situation where we have a dependency projects `dep1`, `dep2` etc and several consumer projects `con1`, `con2` etc. We wish to gate changes in the dependency projects such that the consumer projects will be compatible with any changes in the dependency projects | 09:31 |
| In regular zuul configuration this could be modeled by the `dep1` project having a .zuul.yaml file which triggers the integration tests specified by a template e.g. | ||
| ``` | ||
| // .zuul.yaml in dep1 | ||
| project: | ||
| templates: | ||
| - con1-integration-tests | ||
| // .zuul.yaml in dep2 | ||
| project: | ||
| templates: | ||
| - con1-integration-tests | ||
| - con2-integration-tests | ||
| ``` | ||
| The consumer projects would then have the definition of their respective integration tests and release branches with different dependencies than master would work properly. | ||
| Due to several reasons we would like the dependency configuration to be wholly located inside of the consumer projects, i.e. we would instead like to write something like: | ||
| ``` | ||
| // .zuul.yaml in con1 | ||
| project: | ||
| name: dep1 | ||
| templates: | ||
| - con1-integration-tests | ||
| project: | ||
| name: dep2 | ||
| templates: | ||
| - con1-integration-tests | ||
| // .zuul.yaml in con2 | ||
| project: | ||
| name: dep2 | ||
| templates: | ||
| - con2-integration-tests | ||
| ``` | ||
| This is not allowed since untrusted projects can't modify other project configurations. Changing the consumer projects to a config-project (or writing it to a separate config project) is also not an option since config projects only examines the master branch and therefore release branches would not be running with their dependencies. | ||
| Does anyone have an idea how we could achieve this without exposing the dependents in the dependency projects? | ||
| @tobias.henkel:matrix.org | zuul-maint: I think the freezes are just a symptom, I found frequent segfaults of ansible processes since we updated zuul from 5.2.5 to 6.1.0 | 12:43 |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 851988: Implement cleanup of leaked resources in k8s driver https://review.opendev.org/c/zuul/nodepool/+/851988 | 12:44 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed on behalf of Tobias Henkel: [zuul/zuul] 756077: Increase merge retries and delays in between https://review.opendev.org/c/zuul/zuul/+/756077 | 12:45 | |
| -@gerrit:opendev.org- Joshua Watt proposed: [zuul/zuul] 851931: doc: Fix Nodepool monitoring stats https://review.opendev.org/c/zuul/zuul/+/851931 | 13:09 | |
| @tobias.henkel:matrix.org | changes in our ansible venv during that update: | 13:14 |
| ``` | ||
| Jinja2: 3.0.3 -> 3.1.2 | ||
| azure-core: 1.24.1 -> 1.24.2 | ||
| azure-storage-blob: 12.12.0 -> 12.13.0 | ||
| boto3: 1.24.17 -> 1.24.25 | ||
| botocore: 1.27.17 -> 1.27.25 | ||
| cffi: 1.15.0 -> 1.15.1 | ||
| charset-normalizer: 2.0.12 -> 2.1.0 | ||
| cryptography: 37.0.2 -> 37.0.4 | ||
| dogpile.cache: 1.1.6 -> 1.1.7 | ||
| google-auth: 2.8.0 -> 2.9.0 | ||
| openstacksdk: 0.99.0 -> 0.61.0 | ||
| passlib: None -> 1.7.4 | ||
| requests: 2.28.0 -> 2.28.1 | ||
| typing_extensions: 4.2.0 -> 4.3.0 | ||
| urllib3: 1.26.9 -> 1.26.10 | ||
| ``` | ||
| @tobias.henkel:matrix.org | maybe it's the cffi or crytography bump, both do native stuff that might cause segfaults | 13:14 |
| @fungicide:matrix.org | tobiash: did you happen to get a core file or a stacktrace from the segfaults? anything to hint at the origin? | 13:54 |
| @tobias.henkel:matrix.org | unfortunately not yet | 13:55 |
| @tobias.henkel:matrix.org | the combination of k8s and bwrap makes that difficult | 13:55 |
| @tobias.henkel:matrix.org | at least currently there is no persistent r/w path mapped into the bwrap context we could use as target for the core files | 13:56 |
| @tobias.henkel:matrix.org | we first need to prepare that by updating executors with additional rw/paths configured | 13:56 |
| @fungicide:matrix.org | bingberg: "config projects only examines the master branch and therefore release branches would not be running with their dependencies." is not true. config projects are capable of describing jobs for any/all branches of other projects. that's the main use case for https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.branches | 13:59 |
| @fungicide:matrix.org | also it's not clear to me what your branch model is, but keep in mind that the branch(es) used for determining which job configuration to apply comes from the trigger event. configuration in single-branch projects (including config projects) applies to any triggering branch in the project using it unless narrowed or overridden by specific job attributes | 14:08 |
| @bingberg:matrix.org | > <@fungicide:matrix.org> bingberg: "config projects only examines the master branch and therefore release branches would not be running with their dependencies." is not true. config projects are capable of describing jobs for any/all branches of other projects. that's the main use case for https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.branches | 14:27 |
| We did have a look at if the branch specifier could be powerful enough to express what we wanted, but did not succeed. Perhaps we have misunderstood something? | ||
| By only looking at master I was referring to the configuration loading as specified in https://zuul-ci.org/docs/zuul/latest/project-config.html#configuration-loading and https://zuul-ci.org/docs/zuul/latest/config/project.html# where it specifies that it will apply project definitions from the master branch of the config project to all branches of the subproject. While the untrusted project will apply the project definition only to the matching branch. | ||
| What we are trying to achieve is to have changes in the dependency only trigger integration jobs for their dependents for branches where they are actually dependencies. | ||
| I.e. something like in a config project (if it was possible) | ||
| ``` | ||
| // some config project | ||
| project: | ||
| name: dep1 | ||
| templates: | ||
| - con1-integration-tests | ||
| branch: release/alpha | ||
| project: | ||
| name: dep1 | ||
| templates: | ||
| - con1-integration-tests | ||
| - con2-integration-tests | ||
| branch: master | ||
| ``` | ||
| This would trigger the con1 and con2 tests when changing dep1 in master, but only the con1 test when changing dep1 in release/alpha. Now branch is not a valid modifier for a project and we could not figure out a way to express the same behavior by applying it directly to jobs. | ||
| @clarkb:matrix.org | > <@bingberg:matrix.org> We did have a look at if the branch specifier could be powerful enough to express what we wanted, but did not succeed. Perhaps we have misunderstood something? | 14:40 |
| > | ||
| > By only looking at master I was referring to the configuration loading as specified in https://zuul-ci.org/docs/zuul/latest/project-config.html#configuration-loading and https://zuul-ci.org/docs/zuul/latest/config/project.html# where it specifies that it will apply project definitions from the master branch of the config project to all branches of the subproject. While the untrusted project will apply the project definition only to the matching branch. | ||
| > | ||
| > What we are trying to achieve is to have changes in the dependency only trigger integration jobs for their dependents for branches where they are actually dependencies. | ||
| > | ||
| > I.e. something like in a config project (if it was possible) | ||
| > | ||
| > ``` | ||
| > // some config project | ||
| > project: | ||
| > name: dep1 | ||
| > templates: | ||
| > - con1-integration-tests | ||
| > branch: release/alpha | ||
| > project: | ||
| > name: dep1 | ||
| > templates: | ||
| > - con1-integration-tests | ||
| > - con2-integration-tests | ||
| > branch: master | ||
| > ``` | ||
| > | ||
| > This would trigger the con1 and con2 tests when changing dep1 in master, but only the con1 test when changing dep1 in release/alpha. Now branch is not a valid modifier for a project and we could not figure out a way to express the same behavior by applying it directly to jobs. | ||
| You can specify in the con1 job definition(s) that the job applies to master and release/alpha. Then in the con2 jobs only specify they apply to master. | ||
| @bingberg:matrix.org | > <@clarkb:matrix.org> You can specify in the con1 job definition(s) that the job applies to master and release/alpha. Then in the con2 jobs only specify they apply to master. | 14:41 |
| but the con2 jobs do apply to release/alpha, just not as a result of changes in dep1 but changes to dep2 (and changes to con2 itself as well) | ||
| @clarkb:matrix.org | ok for that you either need different jobs (or variants) | 14:45 |
| @clarkb:matrix.org | when you apply the job to con1 to you update the branches it applies to in order to restrict it further from the default that applies to con2 (this creates a variant) | 14:46 |
| @fungicide:matrix.org | bingberg: i would expect that last example you pasted to result in syntax errors. it looks like you're trying to apply a branch specifier on a project rather than on a job or job variant | 16:01 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] 852042: Don't reconfigure tenant if we exclude all config https://review.opendev.org/c/zuul/zuul/+/852042 | 16:57 | |
| @y2kenny:matrix.org | Hi, what is the suggested way to access the secure zookeeper backing Zuul using things like zkCli/zkServer, etc.? Or do people generally enable plaintext zk for maintenance? | 19:43 |
| @y2kenny:matrix.org | what is the replacement for probe commands like this: "echo "ruok" | timeout 2 nc -w 2 localhost 2181 | grep imok" I suspect I need some ssl enabled nc like ncat but I am not sure. | 20:11 |
| @fungicide:matrix.org | Kenny Ho: we configure ours to allow unauthenticated connections from localhost, then ssh into one of the zk servers and use zksh cli commands there | 20:17 |
| @y2kenny:matrix.org | is this ssl.clientAuth=need in the zoo.cfg? (I just saw something on stackoverflow but not sure if that's right) | 20:18 |
| @fungicide:matrix.org | maybe a bit hackish, but we just have it listen on two ports, one with tls and one without: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zookeeper/templates/zoo.cfg.j2#L26-L27 | 20:20 |
| @fungicide:matrix.org | and of course we block the non-tls socket with iptables on the server | 20:20 |
| @y2kenny:matrix.org | OOoh... I will give that a try | 20:21 |
| @y2kenny:matrix.org | I run my zk on k8s pods so port access is restricted. Hopefully this will work. Thanks for the tip. (I just realized I put my zk storage onto spinning rust instead of nvme... which may have contributed to zuul performance issue.) | 20:22 |
| @fungicide:matrix.org | here you can see we allow 2281/tcp (the tls socket) connections from our zuul and nodepool groups, but nothing to the plaintext 2181/tcp: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/group_vars/zookeeper.yaml#L6-L8 | 20:23 |
| @y2kenny:matrix.org | so I was hoping to spin up a few more zk node that has nvme and then spin down the misconfigured zk node | 20:23 |
| @fungicide:matrix.org | that way we don't have to fiddle with tls setup and cert auth for zk-shell commands over the loopback interface | 20:23 |
| @fungicide:matrix.org | replacing zk cluster members should be pretty straightforward. we've already done it at leats once, owing to the fact that our current cluster members are named zk04, zk05 and zk06 | 20:24 |
| @fungicide:matrix.org | * replacing zk cluster members should be pretty straightforward. we've already done it at least once, owing to the fact that our current cluster members are named zk04, zk05 and zk06 | 20:25 |
| @clarkb:matrix.org | Ya we dropped 01 and added 04 then iterated through one by one like that | 20:25 |
| @fungicide:matrix.org | oh, but if you only have a single zk server now, that is probably a bit more work to add a cluster to it | 20:26 |
| @y2kenny:matrix.org | yea... I only have single zk right now. I will probably need to change the cfg for the election stuff | 20:26 |
| @y2kenny:matrix.org | but first thing is I need to be able to use all those zk*.sh tool to make sure whatever I am doing is correct. | 20:27 |
| @fungicide:matrix.org | but yes, should still be doable without downtime | 20:27 |
| @y2kenny:matrix.org | I will report back on how things go. Thanks for the tips. | 20:29 |
| @fungicide:matrix.org | er, well hopefully anyway. i don't know for sure that you can reconfigure zk to join a cluster without restarting the service | 20:30 |
| @fungicide:matrix.org | once you have a cluster, of course, replacing servers is entirely hitless as long as you allow for replication time between swaps | 20:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!