| *** jamesmcarthur has quit IRC | 00:22 | |
| *** jamesmcarthur has joined #zuul | 00:26 | |
| *** jamesmcarthur has quit IRC | 00:31 | |
| *** jamesmcarthur has joined #zuul | 00:47 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| mnaser | does the zuul-cloned repos include tags by any chance? | 01:01 |
|---|---|---|
| mnaser | aka can operations that grab things like git tags be done on those git checkouts? | 01:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix undefined attrs in registry push/pull roles https://review.openstack.org/637072 | 01:08 |
| clarkb | mnaser: yes it should have those too | 01:11 |
| clarkb | that is how openstack release jobs work | 01:11 |
| mnaser | ok, i'm porting some jobs to zuul and a script that runs git describe seems to return "fatal: No names found, cannot describe anything." .. i'll dig in a bit more | 01:12 |
| *** jamesmcarthur has joined #zuul | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:20 | |
| mnaser | interesting | 01:27 |
| mnaser | "fatal: No names found, cannot describe anything." when i run "git tag" | 01:27 |
| *** jamesmcarthur has joined #zuul | 01:34 | |
| mnaser | GAH | 01:38 |
| mnaser | there was no tags oops | 01:38 |
| *** jamesmcarthur has quit IRC | 01:46 | |
| *** ruffian_sheep has joined #zuul | 01:50 | |
| *** jamesmcarthur has joined #zuul | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:11 | |
| *** jamesmcarthur has joined #zuul | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:32 | |
| *** jamesmcarthur has joined #zuul | 02:48 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #zuul | 03:00 | |
| ruffian_sheep | Sometimes the service can be shut down and restarted directly, but sometimes there will be a timeout. Is this situation normal for the service? I have not made any changes. Just restart the service | 03:06 |
| ruffian_sheep | http://paste.openstack.org/show/745130/ | 03:06 |
| *** jamesmcarthur has quit IRC | 03:18 | |
| *** jamesmcarthur has joined #zuul | 03:19 | |
| mnaser | hmm, fetch-output and stage-output are using different paths by default | 04:27 |
| mnaser | is this by purpose? | 04:27 |
| mnaser | i think ideally we want both to be the same default? | 04:27 |
| *** jamesmcarthur has quit IRC | 04:49 | |
| *** jamesmcarthur has joined #zuul | 04:50 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** jamesmcarthur has joined #zuul | 05:10 | |
| *** jamesmcarthur has quit IRC | 05:15 | |
| *** pabelanger has quit IRC | 05:15 | |
| *** mhu has quit IRC | 05:15 | |
| *** mhu has joined #zuul | 05:15 | |
| *** jamesmcarthur has joined #zuul | 05:31 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** jamesmcarthur has joined #zuul | 05:51 | |
| *** jamesmcarthur has quit IRC | 05:56 | |
| *** rlandy|bbl has quit IRC | 06:04 | |
| *** jamesmcarthur has joined #zuul | 06:12 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** jamesmcarthur has joined #zuul | 06:33 | |
| *** jamesmcarthur has quit IRC | 06:38 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 06:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 06:50 |
| *** saneax has joined #zuul | 06:51 | |
| *** jamesmcarthur has joined #zuul | 06:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add dockerized test setup https://review.openstack.org/636424 | 06:54 |
| *** quiquell|off is now known as quiquell|rover | 06:54 | |
| *** saneax has quit IRC | 06:55 | |
| *** jamesmcarthur has quit IRC | 06:59 | |
| *** saneax has joined #zuul | 06:59 | |
| quiquell|rover | corvus: thanks we can try to put both versions of the comment the clear text with unsafe and the base64 will do at a follow up | 07:05 |
| quiquell|rover | corvus: a suspect yaml tagging will be just metadata at other parser | 07:07 |
| *** bjackman has joined #zuul | 07:07 | |
| quiquell|rover | corvus: one concern would be dumping to json but UnsafeTag has __str__ so this should work | 07:08 |
| *** jamesmcarthur has joined #zuul | 07:15 | |
| *** jamesmcarthur has quit IRC | 07:20 | |
| *** chkumar|pto is now known as chkumar|ruck | 07:29 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:36 | |
| *** jamesmcarthur has joined #zuul | 07:37 | |
| *** jamesmcarthur has quit IRC | 07:42 | |
| *** jamesmcarthur has joined #zuul | 07:58 | |
| *** jamesmcarthur has quit IRC | 08:03 | |
| *** manjeets has quit IRC | 08:09 | |
| *** manjeets has joined #zuul | 08:11 | |
| *** gtema has joined #zuul | 08:14 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:17 | |
| *** jamesmcarthur has joined #zuul | 08:19 | |
| *** jamesmcarthur has quit IRC | 08:24 | |
| *** jamesmcarthur has joined #zuul | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:45 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 08:48 |
| *** jpena|off is now known as jpena | 08:57 | |
| *** jamesmcarthur has joined #zuul | 09:02 | |
| *** jamesmcarthur has quit IRC | 09:07 | |
| *** jamesmcarthur has joined #zuul | 09:23 | |
| *** jamesmcarthur has quit IRC | 09:28 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 09:40 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 09:40 |
| *** jamesmcarthur has joined #zuul | 09:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 09:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 09:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support boolean values in get_default https://review.openstack.org/637158 | 09:46 |
| *** jamesmcarthur has quit IRC | 09:49 | |
| *** saneax has quit IRC | 09:59 | |
| *** saneax has joined #zuul | 09:59 | |
| *** saneax has quit IRC | 10:00 | |
| *** saneax has joined #zuul | 10:01 | |
| *** saneax has quit IRC | 10:03 | |
| *** saneax has joined #zuul | 10:04 | |
| *** saneax has quit IRC | 10:04 | |
| *** saneax has joined #zuul | 10:05 | |
| *** saneax has quit IRC | 10:06 | |
| *** jamesmcarthur has joined #zuul | 10:06 | |
| *** saneax has joined #zuul | 10:07 | |
| *** saneax has quit IRC | 10:10 | |
| *** jamesmcarthur has quit IRC | 10:11 | |
| *** saneax has joined #zuul | 10:11 | |
| *** saneax has quit IRC | 10:12 | |
| *** saneax has joined #zuul | 10:12 | |
| *** saneax has quit IRC | 10:23 | |
| *** saneax has joined #zuul | 10:24 | |
| *** jamesmcarthur has joined #zuul | 10:28 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| jkt | is it safe to use a zuul-executor on a separate node which is from git master, and the rest of zuul on another node running the latest release? | 10:32 |
| jkt | (I'm trying to avoid having to bring that whole JS toolchain to my centos 7 machine which runs my Zuul) | 10:33 |
| jkt | and I would like to try these parallel git preparations that tobiash mentioned a few days ago | 10:33 |
| *** saneax has quit IRC | 10:45 | |
| *** jamesmcarthur has joined #zuul | 10:49 | |
| *** jamesmcarthur has quit IRC | 10:54 | |
| *** saneax has joined #zuul | 10:59 | |
| *** saneax has quit IRC | 11:03 | |
| *** jamesmcarthur has joined #zuul | 11:10 | |
| *** jamesmcarthur has quit IRC | 11:15 | |
| tobiash | jkt: most of the time we try to stay backwards compatible so there is a high chance that this just works | 11:16 |
| jkt | tobiash: thanks | 11:18 |
| tobiash | jkt: but you should be aware of the differences if you do that and judge on each change if it's just targeting one service or more at once | 11:23 |
| tobiash | jkt: so it is not really recommended ;) | 11:24 |
| tobiash | jkt: so I usually run the whole system at the same version (latest master) | 11:26 |
| *** gtema has quit IRC | 11:26 | |
| *** ruffian_sheep has quit IRC | 11:28 | |
| *** jamesmcarthur has joined #zuul | 11:31 | |
| *** rfolco|off is now known as rfolco | 11:33 | |
| *** jamesmcarthur has quit IRC | 11:36 | |
| *** saneax has joined #zuul | 11:50 | |
| *** jamesmcarthur has joined #zuul | 11:53 | |
| *** jamesmcarthur has quit IRC | 11:57 | |
| *** electrofelix has joined #zuul | 11:57 | |
| jkt | tobiash: is there an easy way of pregenerating a release-like tarball with built webapp/JS thingy, and feeding that to pip on my real zuul node? | 12:01 |
| tobiash | jkt: I normally just pip install from the source dir | 12:02 |
| jkt | tobiash: I'm afraid that installing the JS dev stack on centos 7 is not really trivial | 12:03 |
| tobiash | jkt: I think the tools/install-js-tools.sh already covers centos | 12:05 |
| jkt | tobiash: thanks :) | 12:06 |
| *** jamesmcarthur has joined #zuul | 12:14 | |
| *** gtema has joined #zuul | 12:14 | |
| *** jamesmcarthur has quit IRC | 12:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 12:19 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 12:19 |
| chkumar|ruck | Hello | 12:27 |
| chkumar|ruck | Is there a way to access project canonical git url with in the playbook while running it in ci? | 12:28 |
| *** jpena is now known as jpena|lunch | 12:32 | |
| jkt | chkumar|ruck: if you mean within the build node itself, then note that the build node might not have access and/or credentials to access the git server | 12:34 |
| *** jamesmcarthur has joined #zuul | 12:35 | |
| chkumar|ruck | jkt: in tripleo ci jobs, we have this action plugin https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/library/zuul_deps.py | 12:36 |
| chkumar|ruck | jkt: where it clones specific projects based on depends-on | 12:36 |
| chkumar|ruck | jkt: it works fine with openstack project, but if there is github hosted project, it does not work | 12:37 |
| chkumar|ruck | jkt: there I wanted to access the project url for the github project if depends-on is added | 12:37 |
| chkumar|ruck | jkt: here it is called https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L191 | 12:38 |
| jkt | chkumar|ruck: it looks like there should be zuul.projects.$project.canonical_hostname and .name, but they do not contain everything that one might need (e.g., ssh:// vs https:// prefix) | 12:39 |
| jkt | but anyway, I thought that the executors do this "on their own" as long as you list all projects that you ever need in job.required_projects | 12:39 |
| chkumar|ruck | jkt: job.required projects seems to be a good idea | 12:40 |
| chkumar|ruck | let me try that | 12:40 |
| *** jamesmcarthur has quit IRC | 12:40 | |
| jkt | my expectation is that Zuul will then check out correct refs based on depends-on etc | 12:41 |
| *** EmilienM is now known as EvilienM | 12:43 | |
| *** jamesmcarthur has joined #zuul | 12:57 | |
| *** bjackman has quit IRC | 12:58 | |
| *** jamesmcarthur has quit IRC | 13:01 | |
| *** electrofelix has quit IRC | 13:14 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: List changed files for all commits between refs https://review.openstack.org/631797 | 13:17 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API https://review.openstack.org/635714 | 13:17 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add new merger job to get role definitions from a repository https://review.openstack.org/637181 | 13:17 |
| *** jamesmcarthur has joined #zuul | 13:18 | |
| *** bjackman has joined #zuul | 13:18 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** jamesmcarthur has joined #zuul | 13:40 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** rlandy has joined #zuul | 13:42 | |
| *** jamesmcarthur has quit IRC | 13:44 | |
| *** jamesmcarthur has joined #zuul | 13:45 | |
| *** jamesmcarthur has quit IRC | 13:50 | |
| *** jamesmcarthur has joined #zuul | 14:06 | |
| *** jamesmcarthur has quit IRC | 14:10 | |
| *** gtema has quit IRC | 14:15 | |
| *** gtema has joined #zuul | 14:18 | |
| *** pabelanger has joined #zuul | 14:27 | |
| *** jamesmcarthur has joined #zuul | 14:27 | |
| *** gtema has quit IRC | 14:29 | |
| *** jamesmcarthur has quit IRC | 14:31 | |
| jkt | I'm looking at Zuul's new support for artifacts. Is that being used in some openstack jobs already? | 14:42 |
| jkt | (there's one downside to projects defining jobs in their repos -- it makes it hard to grep for these :) ) | 14:42 |
| jkt | the docs are a bit short on this, so I wonder how exactly artifacts work. Accoridng to this: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.artifacts | 14:49 |
| jkt | it looks that my job will only get the `zuul.artifacts` variable set *iff* there's a preceding job in the same pipeline | 14:50 |
| jkt | but if the change that is being tested is a consumer of some artifacts which come from another project, then I'm out of luck, right? | 14:50 |
| jkt | I was hoping for something nifty such as Zuul remembering what artifacts were produced in a build which corresponds to the current tip of the other project's branch, etc | 14:51 |
| *** quiquell|rover is now known as quiquell|off | 14:51 | |
| jkt | I think that the "zuulish" way of doing this is to have a post-merge job which promotes/registers/etc whatever got produced during gate. This promotion/registration should probably do something like setting a symlink in a file server to point to the just-produced artifacts | 14:55 |
| jkt | then, in my consumer project, I take a look at zuul.artifacts; if it is set, I can get my dependencies from there; if not, just fetch them from that well-known place | 14:56 |
| jkt | does that sound about right? | 14:56 |
| *** bjackman has quit IRC | 15:01 | |
| tobiash | jkt: as far as I remember zuul will store the artifact info in the sql database | 15:06 |
| tobiash | jkt: so you need to add an sql connection if not done yet and add the sql reporter to every pipeline | 15:07 |
| tobiash | jkt: this will give out also the builds tab in zuul-web | 15:07 |
| *** jamesmcarthur has joined #zuul | 15:08 | |
| jkt | tobiash: I have not used these zuul's artifacts, but I do have an sql reported, and I can see the build tab | 15:09 |
| jkt | tobiash: my question is related to whether these artifacts get populated from branch state if there's no preceding job in the queue | 15:10 |
| tobiash | jkt: I think that should be the case | 15:10 |
| tobiash | but probably corvus knows better | 15:10 |
| jkt | tobiash: I would love that to be the case, but the docs do not mention that feature | 15:11 |
| clarkb | I dont think it works that way | 15:11 |
| jkt | also, how would zuul infer the artifacts for a branch state of the other project (the one which provides artifacts) | 15:12 |
| clarkb | the docker image roles that make use of this default to pulling the 'latest' tag iirc | 15:12 |
| clarkb | so the registry is responsible for that (along with the roles) not the artifact system itself | 15:12 |
| jkt | clarkb: amd there's the "gate" queue which builds these images, and the "promote" which uploads them to the registry, right? | 15:12 |
| *** jamesmcarthur has quit IRC | 15:13 | |
| clarkb | yes and the promote job moves the latest tag along | 15:13 |
| clarkb | so if there are no preceeding artifacts you grab the latest aiui | 15:13 |
| jkt | my use case involves depending on external projects that I cannot gate on | 15:13 |
| jkt | about once a month, these projects change their function signatures, which requires a change in my downstream projects | 15:14 |
| jkt | so far, I've been using git submodules for these, pinning a specific ref of the external, unstable projects | 15:14 |
| jkt | if there's a change, I can easily adapt our code and update the submodule ref in one commit | 15:15 |
| jkt | but as I'm moving to zuul v3, I'm thinking on how to improve this | 15:15 |
| jkt | we already have two projects consuming these external deps, and it makes sense to keep the pinned version in a lockstep among these two leaf/child projects | 15:15 |
| jkt | so I was thinking of bringing in a thrid projects, let's call it common-dependencies, letting that project produce artifacts, with two leaf projects consuming them | 15:16 |
| jkt | when I adapt to an incompatible change in the external deps, I can add a Depends-on: <change in the common-deps>, and that would ensure that I will get a fresh artifact in leaf's build | 15:17 |
| jkt | recent Gerrit supports "topics", and cross-project changes sharing the same topic can only be submitted together | 15:18 |
| jkt | I don't think Zuul recognizes these, does it? | 15:18 |
| clarkb | not yet, there is a mailing list thread or two about adding support for commiting changes together. | 15:19 |
| clarkb | I think its mostly down to finding someone to implement it | 15:19 |
| jkt | how come I missed that? I browsed the archive only yesterday I think | 15:19 |
| *** cmurphy is now known as cmorpheus | 15:22 | |
| clarkb | the conversations werent specific to that gerrit feature but more gsnsralized support for submitting more than one "change" at atime. I'dhave to dig archives myself to find it | 15:23 |
| tobiash | jkt, clarkb: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-November/000615.html | 15:29 |
| *** jamesmcarthur has joined #zuul | 15:29 | |
| *** kmalloc is now known as needscoffee | 15:32 | |
| *** jamesmcarthur has quit IRC | 15:34 | |
| *** chkumar|ruck is now known as chandankumar | 15:38 | |
| *** jamesmcarthur has joined #zuul | 15:51 | |
| *** jamesmcarthur has quit IRC | 15:53 | |
| *** jamesmcarthur has joined #zuul | 15:53 | |
| corvus | jkt: the idea with provides/requires + artifacts is for artifacts to be speculative, just like git repos. so if change A depends-on change B, and change A is in gate, then obviously B is either ahead of A, in which case artifacts will be populated with the results of B, or it previously landed, in which case it's expected that the artifacts will already have been published, so they won't be populated. | 16:20 |
| jkt | corvus: thanks | 16:21 |
| jkt | so there's an assumption that there's "a registry" somewhere, got it | 16:21 |
| corvus | jkt: in check, it's similar -- if B has landed, zuul will not populated the artifacts. But if B is still an open change, zuul will populate artifacts from the most recent run by querying the sql database | 16:21 |
| jkt | I was hoping that zuul could be that registry for me | 16:21 |
| corvus | jkt: different artifacts require different kinds of storage. i'm working on a set of jobs now that use this system with a docker image registry | 16:22 |
| jkt | ah, by Iregistry" I meant a single field holding a "blessed" URL which is known to work, not really actual content storage | 16:23 |
| corvus | (but this system could work similarly with a deb/rpm repo, etc) | 16:23 |
| jkt | basically, just storing URLs, similarly to what the artifact feature is doing now -- but also for already merged changes | 16:23 |
| corvus | jkt: well, the storage is still there, and you can look it up for any change using the web ui or api. zuul just doesn't populate the field with data from merged changes when it runs new jobs... | 16:25 |
| corvus | (because once a change merges, it isn't speculative anymore) | 16:26 |
| jkt | can I ask zuul for a job result based on a git ref which triggered it? | 16:27 |
| jkt | I'm still thinking about post-merge vs. promote, too | 16:27 |
| corvus | jkt: yes | 16:27 |
| jkt | so right now there's this build at the top of http://zuul.openstack.org/builds : http://zuul.openstack.org/build/22d5f1e5ac6446a3a32193c9dcea45a5 | 16:29 |
| clarkb | fwiw the github cache seems to have maybe improved things but I still see us going out to ansible on getPRBySha far more often than I would expect based on logs | 16:29 |
| clarkb | I'm working on the cleanup that tobiash suggested yesterday around project vs project_name then will see if I can't sort out why that is still hitting the github api so frequently | 16:30 |
| jkt | but using its "newrev", I'm making a search such as http://zuul.openstack.org/builds?change=1f104a093c1f27ddb1967e72cd452d43120b4cc2, and it returns no builds | 16:30 |
| clarkb | Maybe the cache update in getPullBySha isn't working right | 16:30 |
| *** saneax has quit IRC | 16:30 | |
| corvus | jkt: i haven't completely wrapped my head around your use case yet, but what you described at 14:55 ('I think that the "zuulish" way') is a correct description of how provides/requires+artifacts was designed to work. the thing i worry about hitting the api is making the jobs too dependent on zuul. in general, if changes are already merged, you shouldn't need to ask zuul how to build something. but i | 16:31 |
| corvus | admit, that concern is a bit vague and general. :) | 16:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Rename project to project_name in getPullBySha https://review.openstack.org/637218 | 16:32 |
| jkt | yeah, I'm currently only using "check", not "gate", so my point of view might be a bit biased | 16:32 |
| jkt | also, I'm reading through that thread that tobiash mentioned | 16:32 |
| jkt | I have not seen a mention of gerrit's new topic restrictions in there | 16:32 |
| corvus | jkt: that's a search by change number; theoretically, this should be the search: http://zuul.openstack.org/builds?newrev=1f104a093c1f27ddb1967e72cd452d43120b4cc2 | 16:33 |
| corvus | jkt: but it returns too many results, so it seems like it may be broken | 16:33 |
| jkt | basically, there's a way to submit either all changes, or no changes, in upstream gerrit if projects are appropriately configured | 16:33 |
| corvus | jkt: zuul doesn't have support for simultaneous, or bi-directional, depends-on. it assumes it's going to merge things in strict sequence. we could expand zuul to support the other case if we get a volunteer to write that -- there are a few steps involved. one of them, perhaps surprisingly, is actually having zuul perform the merges rather than gerrit. | 16:35 |
| corvus | (so gerrit's support for this becomes somewhat irrelevant) | 16:35 |
| jkt | thanks | 16:37 |
| corvus | jkt: here's a search for the latest builds of the master branch of openstack/neutron; your build from earlier is at the top: http://zuul.openstack.org/builds?project=openstack%2Fneutron&branch=master | 16:38 |
| corvus | jkt: the jobs i'm working on to use provides/requires+artifacts with a docker registry are being tested here: https://review.openstack.org/637037 (the system doesn't work yet, but you can trace those jobs and roles if you want to see more examples. the parent job is in project-config, and the roles are in zuul-jobs) | 16:40 |
| jkt | corvus: on the other hand, it is IMHO an interesting point that Gerrit now effectively implements that point (B) from http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-November/000634.html | 16:41 |
| corvus | jkt: yes, though i believe it was implemented before that discussion too, so i think the conclusions there are still relevant | 16:44 |
| SpamapS | corvus: there's one problem with relying on "latest" in containers. Often times latest has moved, but things like kubernetes will not know it has changed, so ideally there's a way to find out what the latest *unique* tag is, so you can look at the git repo, and tell kubernetes to update. | 16:50 |
| SpamapS | I've done that with a series of tags.. pipeline_post is my default, then I have newrev_{sha} for when I need to make k8s pick up the new image. | 16:51 |
| SpamapS | But that only works in post. I'd love to be able to discern the unique tag from gate just by looking at the git repo + zuul. | 16:52 |
| clarkb | SpamapS: I think our gitea k8s config has a serial number like bind zone files and that gets incremented | 16:54 |
| corvus | SpamapS: for the jobs i'm building, i'm tagging in the intermediate registry with the build uuid. if you can find a build record in zuul, you can find the image. | 16:54 |
| corvus | and yes, that's how we're currently doing deploys -- it's a separate git commit to cause a deploy to happen. it's not the only way to interpret 'gitops' but it is one way :) | 16:55 |
| SpamapS | Right, so I can probably go git sha -> github -> pr -> zuul search by change -> build | 16:56 |
| SpamapS | corvus: I think I actually prefer that gitops paradigm. | 16:56 |
| SpamapS | I have a lot of issues with the way I do it, not the least of which is that whole prod/master branch disaster. | 16:56 |
| SpamapS | Going to be switching to deploy-on-tag I think, which is essentially the same thing. | 16:57 |
| SpamapS | and would have the same breadcrumbs. | 16:57 |
| corvus | yeah, i like the breadcrumbs, it's why i lean in that direction | 17:00 |
| corvus | (i'm just being careful to indicate that zuul is not opinionated about *this* thing :) | 17:00 |
| clarkb | I've written a small test class for GithubShaCache and it passes :/ | 17:08 |
| clarkb | I mean I expected it to, but was hoping that would offer insight into why the cache doesn't seem to always be used | 17:09 |
| tobiash | clarkb: maybe we should log cache hits and misses so we can easier debug this | 17:11 |
| clarkb | tobiash: ya that is probably going to be the next step. I'm going to clean up this test suite and push it since I've written it anyway so may as well have continuing value | 17:12 |
| tobiash | ++ | 17:12 |
| corvus | clarkb: can you paste some log lines that are making you suspicious? | 17:12 |
| corvus | (and yeah, that test class will be helpful if we do anything to make the cache more sophisticated) | 17:13 |
| corvus | new topic -- i think we need a role to read data from zuul_return and use it to set_fact. that way we can easily transfer data from a pre playbook to a post. does that sound useful? does that exist already? :) | 17:13 |
| clarkb | corvus: http://paste.openstack.org/show/745171/ log lines (there are ~170 of those according to wc -l) | 17:15 |
| corvus | clarkb: do you know one of the prs it's looking for? | 17:16 |
| corvus | might be useful to see if we're getting status updates on closed prs, or somesuch. | 17:16 |
| corvus | or, i guess, what sha it's looking for :) | 17:17 |
| corvus | maybe it's a superceded sha or something | 17:17 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Test GithubShaCache https://review.openstack.org/637228 | 17:17 |
| SpamapS | corvus: is there any reason not to just set any zuul returns from pre's on all post's somewhere under zuul? | 17:18 |
| clarkb | corvus: http://paste.openstack.org/show/745172/ is a more recent one where I've gathered more data | 17:18 |
| clarkb | that one is "special" in that the queue grew afterwards | 17:18 |
| clarkb | re superceded sha that could be it | 17:19 |
| clarkb | I'll cross check that shaqw | 17:19 |
| clarkb | er | 17:19 |
| corvus | SpamapS: that's a good question. we do that for later jobs, so why not later playbooks? i can't come up with much of an argument against it, other than a general wariness of altering ansible expectations too much. but i can't even convince myself with that argument. | 17:19 |
| clarkb | sha | 17:19 |
| corvus | SpamapS: i think that was the push i needed to avoid writing a role and instead write a code patch. :) | 17:20 |
| SpamapS | corvus: I've taken a rigor of not referencing zuul.* inside roles.. I only map it in playbooks. This makes my roles pretty reusable outside zuul, and helps me know where I'm relying on zuul magic. So for me, there's really no conflict. :) | 17:20 |
| clarkb | corvus: https://github.com/ansible/ansible/pull/52177 looking at that I think the reason we didn't see it is that sha was a merge commit sha | 17:21 |
| clarkb | corvus: so it didn't previously exist in our cache | 17:21 |
| clarkb | and ansible is running checks on merge commits | 17:22 |
| clarkb | https://app.shippable.com/github/ansible/ansible/runs/107670/summary/console is the check that ran | 17:22 |
| corvus | clarkb: say more words about merge commit -- i'm not following. | 17:23 |
| clarkb | corvus: http://paste.openstack.org/show/745172/ has more details logs and points out the sha is 25323155d2996831744df11dfc6a8ff28ba8201f. That sha doesn't exist until the PR above merges. | 17:23 |
| clarkb | so our earlier cache population won't have it in the table | 17:23 |
| clarkb | then shippable runs those checks on that merge commit and sets a status. Because this commit is brand new post merging we end up doing the full scan | 17:24 |
| corvus | does 25323155d2996831744df11dfc6a8ff28ba8201f show up on that page at all? | 17:24 |
| clarkb | corvus: yes https://github.com/ansible/ansible/pull/52177 as the very end "gundalow merged commit 2532... into ansible:devel" | 17:24 |
| corvus | clarkb: ok, and shippable ran after that merge event, yeah? | 17:25 |
| clarkb | corvus: yup | 17:25 |
| corvus | clarkb: so i wonder why the merge event itself didn't update the cache | 17:25 |
| clarkb | let me do more grepping to see if we processed such an event | 17:26 |
| corvus | wow you just can't grep for 52177 in the log can you :) | 17:26 |
| clarkb | I started with the sha but I'll believe you that 52177 is noisy :) | 17:27 |
| clarkb | another thing I notice, we processed three status events for that sha, all three did the 1.5 minute long lookup | 17:27 |
| clarkb | (so that probably helps explain why the queue grows at that point) | 17:28 |
| tobiash | maybe we just don't call getPull when a pr gets merged | 17:30 |
| clarkb | tobiash: ya looking at the code I think that may be the case? | 17:30 |
| tobiash | didn't look yet | 17:30 |
| tobiash | currently looking at the event docs of github | 17:31 |
| corvus | why do none of these services understand timestamps? | 17:31 |
| corvus | oh, right, because everyone is in san francisco | 17:31 |
| tobiash | clarkb, corvus: I think I know why it's not cached | 17:31 |
| tobiash | because we explicitly remove it if it's other than 'open'... | 17:32 |
| clarkb | oh | 17:32 |
| clarkb | so we remove it, then we get the status event | 17:32 |
| clarkb | ha | 17:32 |
| tobiash | and of course a merge pr has "state": "closed", | 17:32 |
| tobiash | so we may need to switch to ttl based cleanup | 17:33 |
| clarkb | or combo of both | 17:34 |
| clarkb | you stay in cache indefinitely while open, but once not open start counting down | 17:34 |
| corvus | give me a minute to find the definitive order of events here | 17:35 |
| tobiash | should we use this: https://cachetools.readthedocs.io/en/stable/#cachetools.TTLCache ? | 17:37 |
| corvus | clarkb, tobiash: this is the sequence: http://paste.openstack.org/show/745174/ | 17:43 |
| corvus | i got that info from github delivery log | 17:44 |
| corvus | though i think i'm missing the events with 25323 as the sha | 17:44 |
| corvus | those are the events in our log that mention pr 52177 | 17:44 |
| clarkb | corvus: my 25323 logs come after the pull request closed action | 17:45 |
| clarkb | so I think there is another set of events after that that will be interesting to look at | 17:45 |
| corvus | okay, i see 3 deliveries for that, so i'll find those now | 17:47 |
| SpamapS | Wouldn't LRU invalidation handle the ttl/open/closed cases? | 17:47 |
| SpamapS | If we're not using dogpile... and memcached... might make sense to consider.. this is sort of what they're for. | 17:47 |
| * SpamapS hasn't looked closely. | 17:47 | |
| SpamapS | Especially since we don't have a scale-out scheduler yet, but we will, and when that happens, having cache outside process will be important. | 17:48 |
| corvus | SpamapS: we're just storing a mapping, we're not trying to store the data | 17:48 |
| corvus | i think external storage would be overkill | 17:49 |
| SpamapS | ah, k | 17:49 |
| SpamapS | so yeah, then why isn't LRU sufficient? | 17:50 |
| clarkb | lru probably would be, I think we just wanted easy and didn't anticipate this workflow from ansible (though its a totally valid workflow especially if you don't have a zuul) | 17:51 |
| clarkb | the reason lru probably works well is CI results tend to be clustered to other events. So earlier events like a push, recheck, or merge will update cache, then not much later we get the ci status event | 17:53 |
| clarkb | and in that time we likely don't evict the mapping | 17:53 |
| tobiash | the question is what about superceded sha's? | 17:54 |
| tobiash | annotating the function with lru won't handle that | 17:55 |
| tobiash | oh, I think our cache doesn't handle this as well | 17:55 |
| SpamapS | So you're saying that a sha gets disconnected from any PR's, and we don't invalidate that sha -> pr mapping? | 17:56 |
| corvus | got it! | 17:56 |
| corvus | http://paste.openstack.org/show/745176/ | 17:56 |
| corvus | the sha *isn't* superceded | 17:56 |
| corvus | there's a new field -- merge_commit_sha | 17:56 |
| corvus | so we should not remove on closed events, and we should populate the mapping with the merge_commit_sha as well | 17:58 |
| corvus | it should be fine to have more than one sha go to the same pr -- it's only one sha going to more than one pr that's too confusing for us. | 17:58 |
| clarkb | oh interesting | 17:59 |
| clarkb | then use lru to remove them? or ttl? | 17:59 |
| clarkb | if we don't remove on closed events we probably need some other expiry right? | 17:59 |
| corvus | i still like lru for this because of what you said | 17:59 |
| *** jpena is now known as jpena|off | 18:01 | |
| SpamapS | Yeah, LRU fits the access pattern well. The only thing I worry about is missed events. | 18:21 |
| SpamapS | TTL+LRU would actually be the best. In case something stays popular, but doesn't get any updating events, you want to invalidate it just because it is old. | 18:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 18:27 |
| corvus | SpamapS, clarkb, fungi: ^ there's a quick update to the registry role to keep my work moving; it should be forward-compatible with doing that inside of zuul itself later | 18:30 |
| SpamapS | corvus: neat | 18:31 |
| SpamapS | I just hit a situation yesterday where I have one image build, and two deploys to do in post, and I need to start pulling in your work to have a promote pipeline. | 18:31 |
| clarkb | corvus: does that need a from_json filter? | 18:31 |
| corvus | clarkb: not in local testing | 18:31 |
| corvus | though perhaps with that i could have squashed it into one task | 18:32 |
| corvus | lemme see | 18:32 |
| corvus | yep | 18:32 |
| corvus | will redo | 18:33 |
| clarkb | ok | 18:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 18:33 |
| corvus | SpamapS: the promote pipeline and associated roles seem to be working well so far. i don't *think* we'll need to change them to integrate the buildset/intermediate registry stuff, but i'm not 100% sure about that. | 18:34 |
| *** bjackman has joined #zuul | 18:54 | |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** jamesmcarthur has joined #zuul | 19:23 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 19:30 |
| *** jamesmcarthur has joined #zuul | 19:33 | |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** jamesmcarthur has joined #zuul | 19:38 | |
| *** jamesmcarthur has quit IRC | 19:40 | |
| *** jamesmcarthur has joined #zuul | 19:48 | |
| -openstackstatus- NOTICE: The StoryBoard service on storyboard.openstack.org is offline momentarily for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002666.html | 20:05 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** jamesmcarthur has joined #zuul | 20:38 | |
| *** jamesmcarthur has quit IRC | 20:48 | |
| *** jamesmcarthur has joined #zuul | 20:52 | |
| *** jamesmcarthur_ has joined #zuul | 20:55 | |
| *** jamesmcarthur_ has quit IRC | 20:57 | |
| *** jamesmcarthur has quit IRC | 20:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Correct host variable in push-to-intermediate-registry https://review.openstack.org/637310 | 21:43 |
| *** rlandy has quit IRC | 21:58 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Uncap mypy and ignore out yaml ambiguity https://review.openstack.org/637313 | 22:08 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: WIP: Add type hints to zuul.model https://review.openstack.org/637314 | 22:08 |
| gundalow | corvus: clarkb did I break something? | 22:10 |
| SpamapS | ^^ just me, noodling around with mypy | 22:10 |
| clarkb | gundalow: no, more github api weirdness coupled with ansible workflow creating previously unknown problems for our zuul instance | 22:11 |
| clarkb | gundalow: I think what ansible is doing is a valid workflow, just creates fun for caching stuff on our end | 22:11 |
| corvus | gundalow: you're helping us test and improve zuul scaling :) | 22:12 |
| gundalow | Edge cases | 22:13 |
| gundalow | Glad I'm of use :p | 22:13 |
| corvus | gundalow: you are much appreciated :) | 22:13 |
| clarkb | gundalow: in particular github ci status events are per commit which forces zuul to map commit shas to PRs somehow. Currently thats done with a full scan of all open PRs (we're trying to cache that better) | 22:13 |
| clarkb | gundalow: we thought we had that fixed but then we discovered ansible runs CI jobs on post merge commits which is a totally valid thing to do particularly since you don't have a zuul gating stuff | 22:14 |
| clarkb | and our cache was discarding "closed" PRs | 22:14 |
| corvus | now we will cache even harder :) | 22:14 |
| gundalow | Hum, I don't think that's a new new thing. | 22:18 |
| corvus | gundalow: nope, but we just noticed it recently | 22:19 |
| corvus | (it's a performance issue, not a break, so it crept up on us) | 22:20 |
| gundalow | Cool. Well good spot :) | 22:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Correct host variable in push-to-intermediate-registry https://review.openstack.org/637310 | 22:31 |
| *** EvilienM is now known as EmilienM | 22:56 | |
| *** panda is now known as panda|off | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!