| @tony-ang:matrix.org | 0 | 02:13 |
|---|---|---|
| i'm newbe in zuul-ci & installing and configuring zuul-ci according to the document below. | ||
| https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html | ||
| However, the documentation is not integrated with gitlab and it is difficult to configure properly. | ||
| Is there anyone who has integrated zuul-ci and gitlab to install and configure it with test? | ||
| @mnaser:matrix.org | Before I dig too deep, it seems like getting the commit message inside Zuul is not something that is natively available in the list of vars, right? | 02:29 |
| https://0034caf17e384f8e6dbd-bb6e0ede9bf725bf4119cf3bb77d65f6.ssl.cf2.rackcdn.com/834082/2/gate/ansible-collection-atmosphere-tox-build/e92989f/zuul-info/inventory.yaml | ||
| @tristanc_:matrix.org | mnaser: what about zuul.message? it should be the commit message base64 encoded | 02:33 |
| @mnaser:matrix.org | oh you're right, i didnt catch that one | 02:33 |
| @mnaser:matrix.org | ah, i found the wrong/old documents on google :( | 02:33 |
| @iwienand:matrix.org | corvus: a couple of little ones that came out of the zuul-registry work https://review.opendev.org/q/topic:zr-collect-logs | 02:41 |
| @jim:acmegating.com | Tony Ang: it's true the quickstart doesn't talk about how to set up gitlab. after you use it to get zuul up and running with gerrit and you understand the basics, you may be able to look at the gitlab documentation to add a new gitlab connection to the system: https://zuul-ci.org/docs/zuul/latest/drivers/gitlab.html | 02:52 |
| @jim:acmegating.com | mnaser: https://zuul-ci.org/docs/zuul/latest/job-content.html#var-zuul.message has an example | 02:52 |
| @mnaser:matrix.org | > <@jim:acmegating.com> mnaser: https://zuul-ci.org/docs/zuul/latest/job-content.html#var-zuul.message has an example | 02:53 |
| Yup, worked that one out, but now I am trying to find a nice way of getting the files of the most recent commit in a node-less job :) | ||
| @mnaser:matrix.org | https://review.opendev.org/c/vexxhost/ansible-collection-atmosphere/+/834092 -- I'm thinking of using get_uri to make an API call to Gerrit or something along those lines | 02:53 |
| @jim:acmegating.com | mnaser: or maybe see if you can do that in a trusted playbook so you can run git (may be possible if you generalize it sufficiently) | 02:55 |
| @mnaser:matrix.org | yeah, I'm trying to avoid anything trusted for this -- it might be opendev-specific but Looks like https://opendev.org/api/v1/repos/vexxhost/ansible-collection-atmosphere/git/commits/32e6aec07058cb0246ea856aaffe5bf675091df6 could be a thing seems like a thing | 02:55 |
| @mnaser:matrix.org | perhaps making a role which can get commit info for $PLATFORM (i.e. github, gitea, etc) | 02:56 |
| @mnaser:matrix.org | oooor it looks like the commit ID seen inside zuul isnt the same that is inside gerrit (and that's replicated) | 02:58 |
| @jim:acmegating.com | correct, it may be different; you'd have to look it up by change i think | 02:59 |
| @mnaser:matrix.org | looks like i can do this: https://opendev.org/api/v1/repos/vexxhost/ansible-collection-atmosphere/git/refs/changes%2F92%2F834092%2F2 then get the ref from there | 02:59 |
| @jim:acmegating.com | i'm being summoned afk; good luck :) | 03:00 |
| @mnaser:matrix.org | :) | 03:00 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/zuul-registry] 831846: testing: add DEBUG flag to testing container https://review.opendev.org/c/zuul/zuul-registry/+/831846 | 03:09 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/zuul-jobs] 829853: encrypt-file: always import expiring keys https://review.opendev.org/c/zuul/zuul-jobs/+/829853 | 03:16 | |
| @tony-ang:matrix.org | > <@jim:acmegating.com> Tony Ang: it's true the quickstart doesn't talk about how to set up gitlab. after you use it to get zuul up and running with gerrit and you understand the basics, you may be able to look at the gitlab documentation to add a new gitlab connection to the system: https://zuul-ci.org/docs/zuul/latest/drivers/gitlab.html | 03:19 |
| Thanks for replying. I'll take a look at that document and set it up. | ||
| @tony-ang:matrix.org | The part that I do not understand is: | 05:00 |
| (The supported options in zuul.conf connections are:) | ||
| to set gitlab integration : | ||
| <gitlab connection> | ||
| <gitlab connection>.driver(required) | ||
| gitlab | ||
| to set trigger : pipeline.trigger.<gitlab source> | ||
| to set Webhook : http://<zuul-web>/api/connection/<connection-name>/payload. | ||
| what is <gitlab connection>, <gitlab source> & <connection-name> here? | ||
| Any hints? (Can I put in any name or be chosen freely??) | ||
| like this in zuul.conf?? (I don't know how to set it in the zuul.conf.) | ||
| <gitlab connection> | ||
| driver: gitlab | ||
| api_token_name: GITLAB_PAT | ||
| <pipeline> | ||
| trigger | ||
| <gitlab source> | ||
| @sean-k-mooney:matrix.org | you are not the only one the way that the docs use placeholders conuse many that first wee it | 05:02 |
| @sean-k-mooney:matrix.org | here is the gerrit conenction from the example https://opendev.org/zuul/zuul/src/branch/master/doc/source/examples/etc_zuul/zuul.conf#L13 | 05:07 |
| @sean-k-mooney:matrix.org | in this case <gerrit connection> is gerrit | 05:07 |
| @sean-k-mooney:matrix.org | so fro the gitlab connection you would typically write | 05:08 |
| @sean-k-mooney:matrix.org | [connection “gitlab”] | 05:09 |
| @sean-k-mooney:matrix.org | driver: gitlab | 05:09 |
| @sean-k-mooney:matrix.org | … | 05:09 |
| @sean-k-mooney:matrix.org | the value in “” is the name you willl used to refer to it in your tenant file | 05:10 |
| @sean-k-mooney:matrix.org | so you can set it to anything that meakse sense to you such as a team name or company name | 05:11 |
| @sean-k-mooney:matrix.org | its just the name that logically mapps to the connection that will be used for the tenant file | 05:11 |
| @sean-k-mooney:matrix.org | and in the zuul jobs if you fully qualify a repo with its connection name | 05:12 |
| @sean-k-mooney:matrix.org | gitlab source i belive matches whatever name you put in “” | 05:15 |
| @sean-k-mooney:matrix.org | so gitlab in ths case | 05:16 |
| @sean-k-mooney:matrix.org | in the example https://zuul-ci.org/docs/zuul/latest/drivers/gitlab.html#reference-pipelines-configuration they are assuming the connection is called gitlab.com | 05:17 |
| @sean-k-mooney:matrix.org | it would be better if they provided an example of the zuul.conf for each driver | 05:17 |
| @sean-k-mooney:matrix.org | as i agree teh docs are not the simplest to parse | 05:17 |
| @tony-ang:matrix.org | 05:50 | |
| Thank you so much for the detailed explanation. | ||
| I will set it up and test it. | ||
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 07:14 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 834109: Pass requestor data to Nodes https://review.opendev.org/c/zuul/nodepool/+/834109 | 07:56 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 08:03 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 829867: Report gross/total tenant resource usage stats https://review.opendev.org/c/zuul/zuul/+/829867 | 08:10 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 829867: Report gross/total tenant resource usage stats https://review.opendev.org/c/zuul/zuul/+/829867 | 08:14 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 830628: Items in extra paths should be loaded in dependent changes https://review.opendev.org/c/zuul/zuul/+/830628 | 08:19 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 08:26 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 831925: Inject bundle id to inventroy zuul.items https://review.opendev.org/c/zuul/zuul/+/831925 | 09:03 | |
| @q:fricklercloud.de | hmm, running "zuul delete-state" in a docker-compose based aio environment turns out to be non-trivial | 09:36 |
| @q:fricklercloud.de | also the zuul-client container seems to be pretty old and doesn't know about that command | 09:37 |
| @q:fricklercloud.de | Bug-Report: When I'm on e.g. https://zuul.opendev.org/t/opendev/builds?project=opendev%2Fbase-jobs and click "Remove all filters", I get "something went wrong" at the URL https://zuul.opendev.org/t/opendev/builds?skip=0 , after a reload all is fine again | 09:52 |
| @q:fricklercloud.de | same thing for other filters. removing the filter by clicking on the "x" next to it works fine however | 09:53 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed on behalf of Ian Wienand: [zuul/nodepool] 826541: [WIP] Revert "Remove openstacksdk from siblings job" https://review.opendev.org/c/zuul/nodepool/+/826541 | 11:13 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [zuul/nodepool] 834152: Fix flavor handling for openstacksdk 1.0 https://review.opendev.org/c/zuul/nodepool/+/834152 | 11:13 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed on behalf of Ian Wienand: [zuul/nodepool] 826541: Revert "Remove openstacksdk from siblings job" https://review.opendev.org/c/zuul/nodepool/+/826541 | 11:28 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 831925: Inject bundle id to inventroy zuul.items https://review.opendev.org/c/zuul/zuul/+/831925 | 12:39 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 833794: Log debug messages when remote ref is missing https://review.opendev.org/c/zuul/zuul/+/833794 | 13:07 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/nodepool] 834170: Prioritize requests with labels that can be served https://review.opendev.org/c/zuul/nodepool/+/834170 | 13:29 | |
| @jim:acmegating.com | q: delete-state is provided by the scheduler, not zuul-client. you can run it in docker-compose with "docker-compose exec". | 13:40 |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 829867: Report gross/total tenant resource usage stats https://review.opendev.org/c/zuul/zuul/+/829867 | 13:51 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/nodepool] 834170: Prioritize requests with labels that can be served https://review.opendev.org/c/zuul/nodepool/+/834170 | 14:47 | |
| @q:fricklercloud.de | > <@jim:acmegating.com> q: delete-state is provided by the scheduler, not zuul-client. you can run it in docker-compose with "docker-compose exec". | 15:09 |
| so the documentation should be more clear then about the difference between the "zuul" client in the zuul_scheduled container and the "zuul-client" client. | ||
| also, "docker-compose exec" doesn't work if I stop the zuul_* containers beforehand, as I'm supposed to do by the documentation for the delete-state command | ||
| I finally managed to get it working by running another copy of the zuul_scheduler container, manually mounting all the needed volumes and setting the correct network and overriding /etc/hosts in order to be able to resolve the zookeeper container hostname | ||
| @jim:acmegating.com | q: then you want `docker-compose run` | 15:24 |
| @jim:acmegating.com | q: heres' the command from the history on zuul02.opendev.org: `docker-compose run --rm scheduler zuul delete-state` | 15:24 |
| @jim:acmegating.com | q: here's the documentation for the delete-state "zuul" client command: https://zuul-ci.org/docs/zuul/latest/client.html#delete-state maybe you want to add a note to the top of that file reminding folks that it's different than the zuul-client package? | 15:26 |
| @clarkb:matrix.org | corvus: I'm pulling up the gitlab timer changes now | 15:36 |
| @clarkb:matrix.org | corvus: question on https://review.opendev.org/c/zuul/zuul/+/833918 | 15:51 |
| @jim:acmegating.com | Clark: replied | 15:59 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 16:12 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul-jobs] 834196: encrypt-file: roll back extended file stat https://review.opendev.org/c/zuul/zuul-jobs/+/834196 | 16:17 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul-jobs] 834197: Revert "encrypt-file: roll back extended file stat" https://review.opendev.org/c/zuul/zuul-jobs/+/834197 | 16:22 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 16:24 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 834198: Add spec for removing restricted Ansible https://review.opendev.org/c/zuul/zuul/+/834198 | 16:29 | |
| @jim:acmegating.com | zuul-maint: ^ we've discussed this off and on before; i think now is a good time to make a decision on that one way or the other; please let me know your thoughts. | 16:30 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 834198: Add spec for removing restricted Ansible https://review.opendev.org/c/zuul/zuul/+/834198 | 16:33 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 834198: Add spec for removing restricted Ansible https://review.opendev.org/c/zuul/zuul/+/834198 | 16:36 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 16:56 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 833183: Log errors in timer trigger https://review.opendev.org/c/zuul/zuul/+/833183 | 16:58 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 16:59 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 833184: Continue generating timer events on error https://review.opendev.org/c/zuul/zuul/+/833184 | 17:01 | |
| -@gerrit:opendev.org- Artem Goncharov proposed on behalf of Dr. Jens Harbott: [zuul/nodepool] 834152: Fix flavor handling for openstacksdk 1.0 https://review.opendev.org/c/zuul/nodepool/+/834152 | 17:08 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 833918: Fix gitlab timer events https://review.opendev.org/c/zuul/zuul/+/833918 | 17:15 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 17:42 | |
| - [zuul/nodepool] 830527: Add additional tests to the aws driver https://review.opendev.org/c/zuul/nodepool/+/830527 | ||
| - [zuul/nodepool] 834057: Remove unused AWS implementation https://review.opendev.org/c/zuul/nodepool/+/834057 | ||
| @tristanc_:matrix.org | dmsimard: here is a spec to support recent ansible version: https://review.opendev.org/c/zuul/zuul/+/834198 . Didn't you mention a mechanism to run different version, is it execution environment? | 17:43 |
| @dmsimard:matrix.org | Execution environments are really just container images with particular versions of ansible-core, ansible-runner as well as appropriate or necessary ansible collections -- AWX uses them to run jobs in k8s pods nowadays | 17:46 |
| @dmsimard:matrix.org | I'll have a read on the spec and see if I can provide any insight | 17:46 |
| @dmsimard:matrix.org | There's an ansible-builder tool to help build them if you want (it even uses bindep!), I have a fairly simple example here: https://github.com/ansible-community/images/tree/main/execution-environments | 17:48 |
| @clarkb:matrix.org | Is anyone able to review https://review.opendev.org/c/zuul/zuul-jobs/+/834196 ? that is affecting opendev's ability to update our gitea deployment right now | 17:53 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 17:56 | |
| @clarkb:matrix.org | tristanC: thanks! | 18:04 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [zuul/zuul-jobs] 834196: encrypt-file: roll back extended file stat https://review.opendev.org/c/zuul/zuul-jobs/+/834196 | 18:06 | |
| @dmsimard:matrix.org | tristanC: added a few comments, thanks for pointing it out to me :) | 18:14 |
| @jim:acmegating.com | dmsimard: thanks! | 18:17 |
| @dmsimard:matrix.org | happy to help, I happen to be the one doing the releases of the 'ansible' package nowadays so feel free to reach out if you have any questions | 18:18 |
| @fungicide:matrix.org | dmsimard: one thing which could be helpful is suggestions for any ways you're aware of to make it safer to run untrusted ansible playbooks beyond basic containerization functionality like process namespacing et cetera. has there been any progress in recent ansible releases toward new ways to control risks in that use case? | 18:39 |
| @fungicide:matrix.org | looking for ways to avoid putting all our executor security eggs in the kernel/container basket basically (which is effectively the situation we're in at the moment) | 18:40 |
| @dmsimard:matrix.org | @fungi Yes, I remember asking about this myself many years ago already... I don't have a good answer at this time but I know who I can ask so let me get back to you on that. | 18:58 |
| @dmsimard:matrix.org | Broadly speaking, I know that AWX has dropped bwrap in favor of execution environments (containers) in k8s pods (which also makes k8s a hard requirement) though I am unfamiliar with the security pros and cons of doing it that way or whether there are expectations about single vs multiple tenancy | 18:59 |
| @dmsimard:matrix.org | * fungi: Yes, I remember asking about this myself many years ago already... I don't have a good answer at this time but I know who I can ask so let me get back to you on that. | 19:00 |
| @fungicide:matrix.org | thanks, that's definitely interesting. if you have any details on how containers in kubernetes are inherently more secure than containers outside kubernetes, that would be good to know | 19:02 |
| @dmsimard:matrix.org | I don't know about that but I meant to talk about the container/execution environment approach vs bubblewrap | 19:03 |
| @fungicide:matrix.org | yep, thanks, i definitely appreciate the details! | 19:04 |
| @fungicide:matrix.org | whatever you find out would be of interest | 19:04 |
| @fungicide:matrix.org | dmsimard: from a purely technical implementation perspective, i'm also curious how you solve the executor git efficiency problem if workspaces can no longer share a common cache (or do you have a cache, and how do you expose that to them?) | 19:21 |
| @jim:acmegating.com | well, i don't think awx is a 1:1 match for zuul functionality | 19:23 |
| @jim:acmegating.com | ie, i'm not quite sure it has the same focus on git repos | 19:23 |
| @fungicide:matrix.org | oh, i probably misunderstood, i thought he meant awx's use of zuul | 19:24 |
| @jim:acmegating.com | ah, i read that as comparing notes on the parts that are similar: ie, "a thing that runs ansible playbooks for people" | 19:24 |
| @jim:acmegating.com | but one perspective for zuul: a hypothetical COE executor engine shifts the risk from the executor to the k8s cluster. so one would be less concerned with the executor being compromised, but the k8s cluster could be. that would shift the potential exposure from all zuul secrets to the zuul secrets which happened to be used on that k8s cluster during the exposure period. | 19:25 |
| @fungicide:matrix.org | absolutely, it would effectively move the ephemeral build containers to a separate system, but raises obvious efficiency challenges in the process | 19:29 |
| @fungicide:matrix.org | (locality of cache, et cetera) | 19:30 |
| @fungicide:matrix.org | being able to rely on git's hardlink-based cloning within the same filesystem is a huge performance boost | 19:31 |
| @dmsimard:matrix.org | awx does have the concept of "projects" which are git repos that can be retrieved before a job runs or fetched on a regular basis (i.e, cron) but I'm not knowledgeable enough about awx to khow they eventually make it into the container | 20:25 |
| @dmsimard:matrix.org | * awx does have the concept of "projects" which are git repos that can be retrieved before a job runs or fetched on a regular basis (i.e, cron) but I'm not knowledgeable enough about awx to know how they eventually make it into the container | 20:26 |
| @dmsimard:matrix.org | > <@dmsimard:matrix.org> fungi: Yes, I remember asking about this myself many years ago already... I don't have a good answer at this time but I know who I can ask so let me get back to you on that. | 20:47 |
| I asked around and it doesn't look like there's a feature around allowing (or preventing) particular modules or plugins to run although something that I am told could be interesting to look into is to leverage [ansible-runner](https://ansible-runner.readthedocs.io/en/stable/index.html) which may provide more flexibility than shelling out to the ansible-playbook CLI | ||
| @dmsimard:matrix.org | In terms of security, execution environments aren't perfect (given containers, container runtimes and platforms have their own vulnerabilities..) though one thing to consider is that the attack surface can be smaller if you only install what you need in them (vs the entire ansible kitchen sink) | 20:51 |
| @shrews:matrix.org | fwiw, I don’t ever see that being a feature in runner | 20:52 |
| @dmsimard:matrix.org | yeah, I don't know how it would even work in practice :/ | 20:52 |
| @dmsimard:matrix.org | a few years ago I remember asking about an ansible configuration that'd be like a whitelist/blacklist of modules | 20:53 |
| @dmsimard:matrix.org | awx has something kind of like that, I think for adhoc commands ? but it's enforced by awx, not ansible | 20:54 |
| @clarkb:matrix.org | > <@shrews:matrix.org> fwiw, I don’t ever see that being a feature in runner | 20:54 |
| that == restricted access/runtime? | ||
| @shrews:matrix.org | module/plugin allowed list | 20:57 |
| @clarkb:matrix.org | got it | 20:57 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul-jobs] 834194: Fix encrypt files stat validation https://review.opendev.org/c/zuul/zuul-jobs/+/834194 | 21:10 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!