| pabelanger | question about periodic pipeline in zuulv3, when a timer trigger happens, does ever branch of ever project get enqueued into the pipeline for zuul to look for configuration? I just notice 3100 jobs created in rdoproject, and trying to understand what is happening | 00:12 |
|---|---|---|
| pabelanger | I admit, I haven't look at zuul.openstack.org to see what happens | 00:13 |
| clarkb | I think its completely branchless | 00:13 |
| pabelanger | yah, I am struggling to figure out what is happening at https://review.rdoproject.org/zuul3/status.html | 00:13 |
| clarkb | you basically can an up to date repo wtih every branch at current HEAD but you don't get an event for every branch, instead its on event that then triggers the jobs youhave listed under periodic | 00:14 |
| pabelanger | but, I don't have access to zuul debug.ogs | 00:14 |
| pabelanger | logs* | 00:14 |
| clarkb | is it possible you have 3100 periodic jobs defined? | 00:14 |
| pabelanger | no, I don't think we have any defined right now | 00:15 |
| pabelanger | we'll have some next week, once we migrate the remaining jobs, but can't see any projects with it configured | 00:15 |
| pabelanger | I'll have to work with tristanC next week to see what is happening | 00:16 |
| pabelanger | okay, we do have some periodic jobs | 00:18 |
| pabelanger | let me look at configuration | 00:18 |
| pabelanger | yah, I'll need to poke at logs, because a project (openstack/octavia) was enqueued 6 times for some reason | 00:21 |
| pabelanger | running the same jobs | 00:21 |
| pabelanger | which is the same amount of branches that project has: https://review.rdoproject.org/r/#/admin/projects/openstack/octavia,branches | 00:22 |
| clarkb | pabelanger: where can I look at that repo? | 00:23 |
| pabelanger | clarkb: you mean https://review.rdoproject.org/r/gitweb?p=openstack%2Foctavia.git;a=shortlog;h=HEAD | 00:24 |
| pabelanger | or for zuul configuration | 00:25 |
| clarkb | pabelanger: the zuul config | 00:27 |
| pabelanger | clarkb: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/projects.yaml;h=40495a491ad30dc20dadab0703a7e31bdf9f463a;hb=HEAD | 00:27 |
| pabelanger | and https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=tree;f=zuul.d;h=ab48e50dd2935c222cd87d6f650df9042a008987;hb=HEAD is where untrusted config is | 00:28 |
| pabelanger | so legacy jobs / legacy project-templates | 00:29 |
| clarkb | I don't see where the job is defined | 00:32 |
| pabelanger | https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/projects.yaml;h=40495a491ad30dc20dadab0703a7e31bdf9f463a;hb=HEAD#l3479 | 00:32 |
| pabelanger | is the job that is running in periodic now | 00:32 |
| clarkb | that is where the job is in the pipeline but where is that job | 00:33 |
| pabelanger | https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/zuul-legacy-jobs.yaml;h=280c18797af258945b6f1aef0128434bafc02714;hb=HEAD#l214 | 00:33 |
| pabelanger | I'm stumped | 00:34 |
| clarkb | ya that all looks fine | 00:34 |
| pabelanger | but also brain is fried | 00:34 |
| pabelanger | I know SF is patching zuul for some additional things, wonder if this is a result, otherwise, I'm going to see about looking at zuul.o.o and see what happens there | 00:35 |
| pabelanger | but hopefully tristanC will be able to see in debug logs what is happening | 00:35 |
| *** tranzemc has quit IRC | 00:57 | |
| tristanC | pabelanger: it seems like a standard zuul behavior, it adds a fake timer event to every project | 01:00 |
| *** tranzemc has joined #zuul | 01:03 | |
| tristanC | though there is a missing "skip merger:cat job" for project using "include: []" configuraiton | 01:07 |
| *** yolanda_ has joined #zuul | 01:07 | |
| *** yolanda has quit IRC | 01:08 | |
| tristanC | clarkb: http://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/timer/__init__.py#n82 does "foreach project; foreach branch; addEvent()" | 01:18 |
| tristanC | and for some reason, this does a merger:cat job, even for project explicitely configured to not include any in repo object | 01:19 |
| clarkb | hrm it must do that so that each branches pipeline config will be considered | 01:20 |
| clarkb | however that shouldn't result in 3100 jobs if you don't have jobs configured | 01:20 |
| tristanC | it doesn't, but while the merger:cat job are running, items appears in the status page | 01:21 |
| tristanC | and since we only have one merger, it takes some time to flush | 01:21 |
| clarkb | ah | 01:21 |
| clarkb | then ya I think the fix for that is either have it not consider config if it is told to not load config or increase number of merges if you need to load the config | 01:21 |
| tristanC | clarkb: most project are using "include: []" to skip merger:cat job when scheduler restart or reload | 01:22 |
| tristanC | clarkb: but i think the scheduler handle event needs an extra "if project.include" before doing the merger:cat call for timer event | 01:23 |
| clarkb | that will prevent any config from being loaded by the projects though | 01:23 |
| tristanC | which we want for most projects: https://softwarefactory-project.io/cgit/config/tree/zuul/rdo.yaml#n166 | 01:26 |
| tristanC | pabelanger: anyway, pretty sure there is a missing merger:cat job skip somewhere, but as clarkb said, we should also add more merger to speed things up | 01:28 |
| pabelanger | great, sounds like a plan for next week then! And agree, was going to bring up increasing zuul-merger / zuul-executor next week. Thanks for looking tristanC / clarkb | 01:33 |
| *** yolanda_ has quit IRC | 02:46 | |
| *** yolanda_ has joined #zuul | 02:47 | |
| *** mhu has joined #zuul | 03:57 | |
| *** _ari_ has joined #zuul | 03:57 | |
| *** pabelanger_ has joined #zuul | 03:57 | |
| *** weshay_ has joined #zuul | 03:57 | |
| *** weshay has quit IRC | 03:59 | |
| *** _ari__ has quit IRC | 04:00 | |
| *** mhu` has quit IRC | 04:00 | |
| *** myoung|off has quit IRC | 04:00 | |
| *** pabelanger has quit IRC | 04:01 | |
| *** myoung has joined #zuul | 04:02 | |
| tobiash | I also noticed that it issues a cat job for every branch in every project | 05:20 |
| tobiash | But I think we may not need them at all because we're doing tenant reconfigs on branch updates anyway so we should have an up to date version of the config at any time | 05:22 |
| tobiash | I noticrd that because one of our projects created a periodic pipeline that triggers every minute... | 05:23 |
| *** pabelanger_ is now known as pabelanger | 18:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!