| @chgans:matrix.org | > <@clarkb:matrix.org> In that case I really do expect zuul would work excellently. We run jobs with that number of repos quite often | 00:36 |
|---|---|---|
| Specificities of AOSP projects: | ||
| @chgans:matrix.org | Specificities of AOSP projects: | 00:40 |
| - changes are tracked by gerrit using 'batch'. This groups changes across several repo into a single change | ||
| - use of the `repo` tool in place of git to checkout the whole AOSP project (see https://source.android.com/setup/develop/repo) | ||
| - gatting job is common to all repos, gatting job apply to the AOSP project as a whole | ||
| @chgans:matrix.org | > <@chgans:matrix.org> Specificities of AOSP projects: | 00:43 |
| > - changes are tracked by gerrit using 'batch'. This groups changes across several repo into a single change | ||
| > - use of the `repo` tool in place of git to checkout the whole AOSP project (see https://source.android.com/setup/develop/repo) | ||
| > - gatting job is common to all repos, gatting job apply to the AOSP project as a whole | ||
| batch plugin: https://gerrit.googlesource.com/plugins/batch | ||
| @chgans:matrix.org | Just found this presentation: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/presentation-media/Zuul-as-a-build-system-15.11.2018.pdf Apparently | 03:01 |
| > single build of all components (30 repos) | ||
| > Android Repo tool | ||
| @chgans:matrix.org | looks like that zuul support gerrit's `change.submitWholeTopic` https://zuul-ci.org/docs/zuul/latest/config/queue.html#attr-queue.dependencies-by-topic | 04:13 |
| -@gerrit:opendev.org- Tim Beermann proposed: [zuul/zuul] 845124: github: added workflow_dispatch trigger https://review.opendev.org/c/zuul/zuul/+/845124 | 09:16 | |
| @q:fricklercloud.de | is anyone else seeing such notices when running nodepool commands? | 10:03 |
| 2022-06-09 09:47:14,338 INFO nodepool.ComponentRegistry: Noticed new builder component nb010000000020 | ||
| 2022-06-09 09:47:14,343 INFO nodepool.ComponentRegistry: Component builder nb010000000020 updated: {'hostname': 'nb01', 'kind': 'builder', 'model_api': 0, 'state': 'stopped', | ||
| +'version': '6.0.1.dev34 5c4ca2f'} | ||
| 2022-06-09 09:47:14,343 INFO nodepool.ComponentRegistry: Noticed new builder component nb030000000021 | ||
| 2022-06-09 09:47:14,346 INFO nodepool.ComponentRegistry: Component builder nb030000000021 updated: {'hostname': 'nb03', 'kind': 'builder', 'model_api': 0, 'state': 'stopped', | ||
| +'version': '6.0.1.dev34 5c4ca2f'} | ||
| 2022-06-09 09:47:14,347 INFO nodepool.ComponentRegistry: Noticed new builder component nb020000000019 | ||
| 2022-06-09 09:47:14,350 INFO nodepool.ComponentRegistry: Component builder nb020000000019 updated: {'hostname': 'nb02', 'kind': 'builder', 'model_api': 0, 'state': 'stopped', | ||
| +'version': '6.0.1.dev34 5c4ca2f'} | ||
| 2022-06-09 09:47:14,350 INFO nodepool.ComponentRegistry: System minimum data model version 0; this component 0 | ||
| I'm confused by the state:stopped because in fact they seem to be running | ||
| @q:fricklercloud.de | slightly related: how can I see which image builds have been paused with "nodepool image-pause"? | 10:04 |
| -@gerrit:opendev.org- Tim Beermann proposed: [zuul/zuul] 845124: github: added workflow_dispatch trigger https://review.opendev.org/c/zuul/zuul/+/845124 | 11:56 | |
| @jim:acmegating.com | Christian Gagneraud: cde note that because zuul is intended to be responsible for providing a speculative future state of git repos, jobs shouldn't use tools like repo to check out git repos; instead they should use the repos that zuul provides. in other words, tools like repo and batch overlap with functionality that zuul implements internally (and is critical to zuul's operation). using gerrit topics along with submit-whole-topic is a good way of getting batch change functionality with zuul (automaticy) | 12:29 |
| @chgans:matrix.org | corvus: that might be the reason why tungsten fabric gave up on zull (if i understand correctly) | 13:12 |
| @chgans:matrix.org | for AOSP work, repo is a central tool. checking out AOSP is very very expensive operation. So re-use of existing source tree is important. Incremental builds is importnat too. | 13:13 |
| @jim:acmegating.com | Christian Gagneraud: yes, zuul also does a lot of work to ensure efficiency and git repo re-use | 13:15 |
| @jim:acmegating.com | * Christian Gagneraud: zuul also does a lot of work to ensure efficiency and git repo re-use | 13:16 |
| @jsokolvewd:matrix.org | > <@jim:acmegating.com> Christian Gagneraud: cde note that because zuul is intended to be responsible for providing a speculative future state of git repos, jobs shouldn't use tools like repo to check out git repos; instead they should use the repos that zuul provides. in other words, tools like repo and batch overlap with functionality that zuul implements internally (and is critical to zuul's operation). using gerrit topics along with submit-whole-topic is a good way of getting batch change functionality with zuul (automaticy) | 13:16 |
| I personally side-step this by initially doing checkout using repo (50+ repositories, including external dependencies), then doing checkouts again using zuul checkouts as a source (5 repositories that are under my CI) | ||
| @jim:acmegating.com | Jakub Sokół: that's reasonable, and not too dissimilar to how the zuul-jobs roles handle git caches on test nodes. | 13:24 |
| @jim:acmegating.com | i do believe that repo isn't necessary with zuul, and that with a git repo cache on a build node + zuul's internal cache on the executor, the result should be reasonably efficient and complete. but as long as the end result is that the complete state of the repos that zuul prepared end up on the build node(s), that's all that really matters. :) | 13:28 |
| @jim:acmegating.com | (for max efficiency, if using vms, i would use repo to build a cache on the image, then use standard zuul-jobs roles to push updates from the executor to the build node) | 13:29 |
| @jim:acmegating.com | (which i think is basically the Jakub Sokół process, except i've inserted an image build into the sequence :) | 13:31 |
| @g_gobi:matrix.org | Hi, | 14:04 |
| Is there a way to run jobs in parallel? | ||
| @clarkb:matrix.org | They should be default as long as there are test resources to do so | 14:04 |
| @g_gobi:matrix.org | Clark: Can you please explain a bit. I can't understand that. | 14:05 |
| @clarkb:matrix.org | If an event triggers multiple jobs they will be run in parallel as long as nodepool can supply enough test nodes to do so and you haven't configured job dependencies that force a specific order | 14:06 |
| @g_gobi:matrix.org | I have two tests. Each will take a hour. I want to know, Is there a way to run both jobs in parallel and also how will be logs look like in dashboard. | 14:06 |
| @clarkb:matrix.org | Yes the default is that they will run in parallel. | 14:07 |
| @g_gobi:matrix.org | No, I Want to run 2 tests in single pipeline. | 14:07 |
| @g_gobi:matrix.org | Like how we configure lint and code-format jobs to run in parallel. | 14:09 |
| @g_gobi:matrix.org | In zuul, we use ansiblw it run lint and code-fmt as sequential. | 14:09 |
| -@gerrit:opendev.org- Joseph Kostreva proposed: [zuul/zuul-jobs] 842723: prepare-workspace: Add variable prepare_workspace_rsync_opts https://review.opendev.org/c/zuul/zuul-jobs/+/842723 | 14:10 | |
| @clarkb:matrix.org | You can look at zuul's zuul Dashboard to see what logs look like. My internet connection is poor right now so can't get a full url but https://zuul.opendev.org to start | 14:10 |
| @clarkb:matrix.org | > <@g_gobi:matrix.org> In zuul, we use ansiblw it run lint and code-fmt as sequential. | 14:11 |
| You may want to run two separate jobs. Or otherwise use Ansible techniques to run things in parallel for example on different hosts. I would suggest two different jobs. This is what we do | ||
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/builds is the url to start from. Select a build and you can view logs from there | 14:12 |
| @g_gobi:matrix.org | > <@clarkb:matrix.org> You may want to run two separate jobs. Or otherwise use Ansible techniques to run things in parallel for example on different hosts. I would suggest two different jobs. This is what we do | 14:13 |
| You mean two different jobs in single pipeline? | ||
| @clarkb:matrix.org | Yes | 14:13 |
| @jim:acmegating.com | that's from the zuul dashboard Clark linked above right now -- it shows 6 jobs running in parallel for one change; linters, docs, then 4 versions of python | 14:17 |
| @g_gobi:matrix.org | Can I have a link for this pipeline configuration? | 14:18 |
| @g_gobi:matrix.org | - project: | 14:22 |
| name: proj-name | ||
| default-branch: master | ||
| merge-mode: squash-merge | ||
| gate: | ||
| jobs: | ||
| - lint (example) | ||
| - code-fmt (example) | ||
| So, if we configure like that. It will run both lint and code-fmt like above (parallel). Am I right? | ||
| @clarkb:matrix.org | Correct | 14:23 |
| @jim:acmegating.com | Gobi G: here's zuul's own config: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L286 | 14:24 |
| @jim:acmegating.com | (the zuul-jobs repo config is available too, but it's enormous and automatically generated, so not a very good example) | 14:24 |
| @g_gobi:matrix.org | Thank you Clark and corvus . I'll try and let you guys know if I face any issues. | 14:26 |
| @g_gobi:matrix.org | It works as you guys said.👏 | 14:36 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 845284: Clarify missing merge requirement message https://review.opendev.org/c/zuul/zuul/+/845284 | 15:13 | |
| @jpew:matrix.org | I'm confused about when zuul decided to show the output of a shell command in the console stream :/ | 21:00 |
| @jpew:matrix.org | For some commands it does and for others it doesn't | 21:00 |
| @sean-k-mooney:matrix.org | i tought that was just determined by ansible | 21:01 |
| @jpew:matrix.org | Ah `shell` works, but not `ansible.builtin.shell` | 21:08 |
| @jpew:matrix.org | Looks like it was fixes already; excellent | 21:21 |
| -@gerrit:opendev.org- Joseph Kostreva proposed: [zuul/zuul-jobs] 842723: prepare-workspace: Add variable prepare_workspace_rsync_opts https://review.opendev.org/c/zuul/zuul-jobs/+/842723 | 22:33 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!