| @chgans:matrix.org | > <@jim:acmegating.com> Christian Gagneraud: cde note that because zuul is intended to be responsible for providing a speculative future state of git repos, jobs shouldn't use tools like repo to check out git repos; instead they should use the repos that zuul provides. in other words, tools like repo and batch overlap with functionality that zuul implements internally (and is critical to zuul's operation). using gerrit topics along with submit-whole-topic is a good way of getting batch change functionality with zuul (automaticy) | 00:43 |
|---|---|---|
| the repo tool does more than dealing with git repo, the definition of the source tree is specified by a manifest file, see https://android.googlesource.com/platform/manifest/+/refs/heads/android-12.0.0_r32/default.xml for example. Please note the use of 'copy-file' and 'link-file' | ||
| @iwienand:matrix.org | Christian Gagneraud: maybe you could do something like have a zuul job that edits the manifest file and points it to repos that zuul has checked out? | 01:08 |
| @iwienand:matrix.org | in terms of caching, etc. just for reference what opendev/openstack does is makes .qcow2 vm images daily and clones everything into these images. these are then a on-disk cache for each testing node -- so we start from these source trees and then only need to pull deltas across the network, the theory being at most 1 days worth of changes | 01:12 |
| @iwienand:matrix.org | we're taking the penalty up front of bigger images that we upload, to reduce cloning time during CI runs. it works for us because we use many different cloud providers, so it's not like all our testing nodes are connected by 10g ethernet to the canonical source, etc. If we were are heterogeneous setup fast, big shared storage or something we might make different choices. Obviously endless variations here, it would really depend on storage trade-offs, how close you can get things on the network, etc. etc. | 01:16 |
| @chgans:matrix.org | interesting approach, a bit heavy maybe. In my mind, builder nodes will have to re-use volumes, eg a volume for the source tree, a volume for build tree. As we want to handle incremental builds. Building from scratch can take 2 hours or more depnding on number of CPUs. We typically use 16 vCPU VMs, and we might switch to 32 vCPUs | 01:29 |
| @chgans:matrix.org | Is there documentation on the internals of Zuul? That could help me understand the problems | 01:29 |
| @iwienand:matrix.org | I have some experience in the past with heavy builds and ccache, shared storage etc. Honestly it was never that effective because so much could bust the cache. this was certainly not android-ish though, so things may be very different. incremental builds are sort of interesting because a big thing about the CI is essentially ensuring everything starts from a "blank slate". | 01:45 |
| @iwienand:matrix.org | in terms of internals -- a lot of this level of thing is done via zuul-jobs | 01:46 |
| @chgans:matrix.org | > <@iwienand:matrix.org> I have some experience in the past with heavy builds and ccache, shared storage etc. Honestly it was never that effective because so much could bust the cache. this was certainly not android-ish though, so things may be very different. incremental builds are sort of interesting because a big thing about the CI is essentially ensuring everything starts from a "blank slate". | 01:47 |
| Agree about blank state, but incremental builds allow you to get answers within 10 to 20 minutes. Built from scratch can stiff be performed (eg nightly, ...). | ||
| @chgans:matrix.org | > <@iwienand:matrix.org> in terms of internals -- a lot of this level of thing is done via zuul-jobs | 01:47 |
| thanks, i'll dig into this aspect | ||
| @iwienand:matrix.org | to get an idea, you can really pull up any build | 01:47 |
| @iwienand:matrix.org | https://zuul.openstack.org/builds and look at the console output | 01:48 |
| @chgans:matrix.org | pull up any build, what do you mean? | 01:48 |
| @iwienand:matrix.org | the base/pre.yaml playbooks give you an idea of what every job is running to get itself setup | 01:48 |
| @iwienand:matrix.org | a good way to get insight is to just search for the tasks at codesearch.opendev.org -- that will usually get you into the actual code behind them | 01:49 |
| @iwienand:matrix.org | zuul-jobs is a library; lots of components that help you do things like setup source on disk, setup log files, etc. etc. they could either be used directly, or used as templates to do custom things. i imagine you'd start just by hacking in whatever works, and as you think "oh, this android source setup could be made more generic" it's the type of thing you could clean up and contribute back to zuul jobs, for the next luckly/unlucky person who has to do that :) | 01:52 |
| @iwienand:matrix.org | i imagine you'd be most interested in things like prepare-workspace-git, etc. | 01:54 |
| @chgans:matrix.org | yes, thanks for that, i'm starting to understand who this works! :) | 01:55 |
| @chgans:matrix.org | looking at https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git right now | 01:55 |
| @chgans:matrix.org | but i can't find where git commands are run... | 01:55 |
| @chgans:matrix.org | NVM, found it! :) | 01:56 |
| @iwienand:matrix.org | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git/tasks/main.yaml#L11 would be about it | 01:57 |
| @iwienand:matrix.org | really the magic of it is | 01:58 |
| @iwienand:matrix.org | with_items: "{{ zuul.projects.values() | list }}" | 01:58 |
| @iwienand:matrix.org | so what has happened is that zuul has started ansible, and has setup the "zuul.projects" variable for you with the projects you are testing in this change | 01:58 |
| @iwienand:matrix.org | everything that follows on there is really just about cloning those repos | 01:59 |
| @iwienand:matrix.org | and if you're wondering what zuul sets for you; it is all in the official docs in "human readable" form; but in hacker form, on every job results page, under logs, you can expand zuul-info -> inventory.yaml and you'll see the variables | 02:02 |
| @chgans:matrix.org | interesting! i didn't see where the checkout is done tho. gerrit gives you a change set that you'll have to fetch... OK, digging more. | 02:05 |
| @iwienand:matrix.org | one thing to keep in mind is that there is the "executor", which is the initial environment, where zuul has setup the source trees for you | 02:07 |
| @iwienand:matrix.org | that is the bwrap environment that runs on the zuul executor node. this stuff you're looking at is about getting the source code to the actual node you're going to test on | 02:08 |
| @iwienand:matrix.org | the executor is really just for shuffling code, copying logs, etc. your job is going to have something (or, indeed multiple somethings) big attached to it (a vm, container, whatever) by nodepool where the tests actually run. these steps are where you would integrate the code zuul is telling you to test onto the actual testing environment. | 02:12 |
| @chgans:matrix.org | OK, thanks, lot of things to dig into! ðĪŠ | 02:41 |
| @chgans:matrix.org | Will the videos be available online?Man, i just missed the summit! ð | 04:49 |
| @chgans:matrix.org | * Man, i just missed the summit! ð Will the videos be available online? | 04:49 |
| @jim:acmegating.com | > <@chgans:matrix.org> Man, i just missed the summit! ð Will the videos be available online? | 04:59 |
| Yes they will! | ||
| @jim:acmegating.com | Zuul does all the fetching from Gerrit internally; job playbooks should never talk to gerrit -- instead, they should (using the roles ianw pointed out) push the git repo state that the executor prepared for the job to the nodes. I think ianw's suggestion of doing some kind of post-processing with repo makes sense in the zuul paradigm. | 05:03 |
| @jim:acmegating.com | we do something similar with submodules for the gerrit project on gerrit's zuul | 05:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!