| *** tosky has quit IRC | 00:03 | |
| *** jamesmcarthur has joined #zuul | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #zuul | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:34 | |
| *** jamesmcarthur has joined #zuul | 01:00 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #zuul | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** jamesmcarthur has joined #zuul | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:55 | |
| *** jamesmcarthur has joined #zuul | 02:00 | |
| *** zenkuro has quit IRC | 02:17 | |
| *** lyr has quit IRC | 02:18 | |
| *** sshnaidm has quit IRC | 02:19 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #zuul | 02:22 | |
| *** lyr has joined #zuul | 02:22 | |
| *** mhu has quit IRC | 02:23 | |
| *** tobberydberg has quit IRC | 02:23 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** mhu has joined #zuul | 02:28 | |
| *** sshnaidm has joined #zuul | 02:29 | |
| *** tobberydberg has joined #zuul | 02:35 | |
| *** jamesmcarthur has joined #zuul | 02:52 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** jamesmcarthur has joined #zuul | 03:08 | |
| *** ykarel has joined #zuul | 04:24 | |
| *** sanjayu_ has joined #zuul | 05:06 | |
| *** vishalmanchanda has joined #zuul | 05:10 | |
| *** jamesmcarthur has quit IRC | 05:24 | |
| *** jamesmcarthur has joined #zuul | 05:29 | |
| *** evrardjp_ has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** jamesmcarthur has quit IRC | 05:34 | |
| *** jfoufas1 has joined #zuul | 05:40 | |
| *** sanjayu_ has quit IRC | 05:51 | |
| *** jamesmcarthur has joined #zuul | 05:52 | |
| *** ykarel has quit IRC | 05:58 | |
| *** ykarel has joined #zuul | 06:05 | |
| *** jamesmcarthur has quit IRC | 06:12 | |
| *** jamesmcarthur has joined #zuul | 06:18 | |
| *** jamesmcarthur has quit IRC | 06:23 | |
| *** jamesmcarthur has joined #zuul | 06:35 | |
| *** sanjayu_ has joined #zuul | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:49 | |
| *** jamesmcarthur has joined #zuul | 06:52 | |
| *** jamesmcarthur has quit IRC | 06:56 | |
| *** sanjayu_ has quit IRC | 07:04 | |
| *** jamesmcarthur has joined #zuul | 07:20 | |
| *** AJaeger has joined #zuul | 07:27 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** jamesmcarthur has joined #zuul | 07:29 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** jamesmcarthur has joined #zuul | 07:30 | |
| *** jamesmcarthur has quit IRC | 07:35 | |
| *** jcapitao has joined #zuul | 07:35 | |
| *** saneax has joined #zuul | 07:36 | |
| *** ykarel_ has joined #zuul | 07:45 | |
| *** saneax has quit IRC | 07:45 | |
| *** ykarel has quit IRC | 07:45 | |
| *** sanjayu_ has joined #zuul | 07:45 | |
| *** ykarel_ is now known as ykarel | 07:45 | |
| *** danpawlik5 has quit IRC | 07:51 | |
| *** dpawlik7 has joined #zuul | 07:53 | |
| *** ykarel_ has joined #zuul | 07:58 | |
| *** rpittau|afk is now known as rpittau | 07:59 | |
| *** AJaeger has quit IRC | 07:59 | |
| *** jamesmcarthur has joined #zuul | 08:00 | |
| *** ykarel has quit IRC | 08:01 | |
| *** ykarel_ is now known as ykarel | 08:03 | |
| *** jamesmcarthur has quit IRC | 08:05 | |
| *** jamesmcarthur has joined #zuul | 08:20 | |
| *** yourname is now known as avass | 08:30 | |
| *** tosky has joined #zuul | 08:44 | |
| *** hashar has joined #zuul | 08:52 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** icey has joined #zuul | 10:59 | |
| *** icey has quit IRC | 10:59 | |
| *** icey has joined #zuul | 11:00 | |
| *** dpawlik7 has quit IRC | 11:06 | |
| *** dpawlik7 has joined #zuul | 11:09 | |
| *** nils has joined #zuul | 11:29 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: gitlab: Add access token name, Update docs, Fix webhook https://review.opendev.org/c/zuul/zuul/+/771184 | 11:29 |
|---|---|---|
| *** msuszko has quit IRC | 11:38 | |
| *** msuszko has joined #zuul | 11:40 | |
| *** jcapitao is now known as jcapitao_lunch | 11:54 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-client master: Add create-auth-token command https://review.opendev.org/c/zuul/zuul-client/+/771188 | 11:57 |
| *** zenkuro has joined #zuul | 12:08 | |
| *** jpena is now known as jpena|lunch | 12:28 | |
| *** bhagyashris has quit IRC | 12:47 | |
| *** bhagyashris has joined #zuul | 12:47 | |
| *** sshnaidm is now known as sshnaidm|ruck | 12:52 | |
| *** rlandy has joined #zuul | 12:54 | |
| *** hashar has quit IRC | 12:56 | |
| *** vishalmanchanda has quit IRC | 12:56 | |
| *** tobberydberg has quit IRC | 12:56 | |
| *** holser has quit IRC | 12:56 | |
| *** snapiri has quit IRC | 12:56 | |
| *** johanssone has quit IRC | 12:56 | |
| *** logan- has quit IRC | 12:56 | |
| *** hashar has joined #zuul | 12:59 | |
| *** vishalmanchanda has joined #zuul | 12:59 | |
| *** tobberydberg has joined #zuul | 12:59 | |
| *** holser has joined #zuul | 12:59 | |
| *** snapiri has joined #zuul | 12:59 | |
| *** logan- has joined #zuul | 12:59 | |
| *** johanssone has joined #zuul | 12:59 | |
| *** logan- has quit IRC | 12:59 | |
| *** logan- has joined #zuul | 13:00 | |
| *** masterpe has quit IRC | 13:03 | |
| *** mordred has quit IRC | 13:05 | |
| *** Eighth_Doctor has quit IRC | 13:05 | |
| *** bhavikdbavishi has joined #zuul | 13:09 | |
| *** jcapitao_lunch is now known as jcapitao | 13:10 | |
| *** bhavikdbavishi has quit IRC | 13:13 | |
| *** jpena|lunch is now known as jpena | 13:28 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-client master: Add create-auth-token command https://review.opendev.org/c/zuul/zuul-client/+/771188 | 13:30 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Fail mirror-workspace-git-repos if checkout failed https://review.opendev.org/c/zuul/zuul-jobs/+/771220 | 13:31 |
| tobiash | zuul-maint: this fixes an issue that could errors slipping through the gate ^ | 13:41 |
| avass | tobiash: lgtm | 13:46 |
| tobiash | zuul-maint: there is also another fix that fixes the tox job with latest tox versions: https://review.opendev.org/c/zuul/zuul-jobs/+/770819 | 13:54 |
| *** mordred has joined #zuul | 13:55 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fail mirror-workspace-git-repos if checkout failed https://review.opendev.org/c/zuul/zuul-jobs/+/771220 | 13:57 |
| *** Eighth_Doctor has joined #zuul | 13:57 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-client master: Add create-auth-token command https://review.opendev.org/c/zuul/zuul-client/+/771188 | 14:01 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: gitlab: Add access token name, Update docs, Fix webhook https://review.opendev.org/c/zuul/zuul/+/771184 | 14:02 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: gitlab: Add access token name, Update docs, Fix webhook https://review.opendev.org/c/zuul/zuul/+/771184 | 14:07 |
| avass | lgtm | 14:13 |
| avass | but i noticed that the 'tox_constraints_env' isn't documented and is only set as a fact if 'tox_constraints_file' is set | 14:13 |
| avass | maybe 'tox_constraints_env' should either be made 'private' by prepending a '_' like other roles do and to mimic python a bit or it should be supported and added to the docs | 14:14 |
| *** sduthil has joined #zuul | 14:14 | |
| avass | it's also used in ensure-javascript-packages which is a bit weird: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-javascript-packages/tasks/main.yaml#L47 | 14:17 |
| *** maxamillion[m] has joined #zuul | 14:22 | |
| *** sanjayu__ has joined #zuul | 14:30 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Pass environment variables to 'tox envlist config' task https://review.opendev.org/c/zuul/zuul-jobs/+/770819 | 14:31 |
| *** maxamillion has quit IRC | 14:31 | |
| *** sanjayu_ has quit IRC | 14:33 | |
| *** zbr2 has joined #zuul | 14:35 | |
| *** zbr has quit IRC | 14:37 | |
| *** zbr2 is now known as zbr | 14:37 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: negative test https://review.opendev.org/c/zuul/zuul-jobs/+/522438 | 14:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771225 | 14:41 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 14:41 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 14:42 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 14:45 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: negative test https://review.opendev.org/c/zuul/zuul-jobs/+/522438 | 14:49 |
| *** dpawlik7 has quit IRC | 14:52 | |
| *** ykarel has quit IRC | 14:57 | |
| tristanC | it seems like our third party for zuul/zuul-jobs is no longer working, reporting is failing with `error: fatal: Applying label "Verified": -1 is restricted` | 14:57 |
| tobiash | corvus: our zuul is drowning in logs, most prominent by zuul.layout (~80%). This package only emits debug messages but we need those at least partially for debugging job freezing issues. What do you think about promoting some of those messaged to info so this would be sufficient for most job freezing analysis? | 15:05 |
| tobiash | I could think about promoting the 'matched' messages and maybe 'froze job' messages | 15:05 |
| tobiash | especially the 'did not match' messages are becoming a problem especially during reconfigs of large tenants with large queues and many branches | 15:07 |
| avass | ++ | 15:07 |
| tobiash | our profiling showed that up to 30% of cpu time of our scheduler goes into logging and it's almost permanently cpu bound nowadays | 15:09 |
| corvus | tobiash: just brainstorming: can you set the logs to info and then use the 'debug' pipeline attribute to get job freezing logs on request? | 15:09 |
| tobiash | corvus: we encourage our users to do that but that's not possible for periodic and tag triggered pipelines | 15:10 |
| corvus | tobiash: if the issue is cpu consumed by logging -- are you sure that changing the log level will reduce usage? | 15:10 |
| tobiash | corvus: profiling showed that it's spending mich during formatting, so I guess this would improve | 15:10 |
| corvus | i mean it should reduce it some, but will it reduce it "enough", i wonder | 15:10 |
| fungi | tristanC: what account are you using? i'll see if any permissions/acls have changed | 15:11 |
| corvus | tobiash: okay, i guess formatters don't run if no logger they're attached to matches, so that makes sense. | 15:11 |
| tobiash | yes | 15:11 |
| tobiash | my thinking is that we could make at least part of job freezing post mortem debuggable by at least logging the matches | 15:12 |
| corvus | tobiash: let me think about this a bit; i'm not sure moving those to info is going to be great. | 15:12 |
| tristanC | fungi: the user is sf-project-io | 15:13 |
| tobiash | otherwise I guess I'd have to patch out the 'does not match' messages in our zuul for now until we have a better solution | 15:13 |
| corvus | tobiash: i find that when i'm actually debugging an issue, the 'does not match' is usually the line i need. | 15:14 |
| tobiash | in our case most problems are that more elements than expected match due to branching confusion :) | 15:15 |
| corvus | makes sense | 15:15 |
| tobiash | I know that this would be not a 100% solution | 15:15 |
| tobiash | I'm just trying to keep our scheduler alive atm | 15:15 |
| tristanC | tobiash: we are also flooded with 'does not match' message, especially with changes on zuul-jobs which go through every pipeline, even when it has no project config | 15:15 |
| tobiash | tristanC: I guess you mean the change did not match? | 15:16 |
| tobiash | or does it really job freezing in every pipeline? | 15:17 |
| *** dpawlik7 has joined #zuul | 15:17 | |
| fungi | tristanC: did you maybe recently change your configuration to start trying to leave verified votes rather than just comments? looking at changes for which that account has left results it seems to have only ever been commenting not voting | 15:17 |
| tristanC | tobiash: $tenant.$pipeline: [e: $event-id] Event <GerritTriggerEvent comment-added opendev.org/zuul/zuul-jobs ...> does not match ... | 15:17 |
| tristanC | fungi: that account used to be able to vote verified -1, how are we supposed to report third-party ci failure? | 15:18 |
| fungi | tristanC: to leave verified votes on that repo (unless we change the acl, which hasn't been altered in ages), an account would need to be a member of the "Voting Third-Party CI" group in our gerrit. are you sure it was leaving -1 (or even +1) votes in the past? can you find an example? | 15:19 |
| tristanC | fungi: oh, it seems like it did not, though perhaps before the gerrit upgrade, the vote failure was not preventing comment | 15:21 |
| fungi | tristanC: that sounds likely, yes | 15:22 |
| fungi | i think you're correct that old gerrit versions silently ignored votes which it wasn't allowing you to apply | 15:22 |
| corvus | to be clear, if you're drowning in debug log messages, you can turn off debug logging. my understanding is that tobiash would like to do that, but it turns out he needs some debug-level messages for normal usage, thus the suggestion to promote them to info. but still, i'd like to keep info a little higher-level than that. perhaps we can find another way to make the info available: 1) save it as a zuul var | 15:23 |
| corvus | in job inventory; 2) use the upcoming job freeze api to provide it; or 3) log it at info level, but do it less often (perhaps only when it changes from the last time it was logged :) | 15:23 |
| corvus | tobiash: i'm warming up to #3, what do you think about that? | 15:23 |
| tobiash | corvus: can you elaborate on what you mean by do it less often? | 15:24 |
| tobiash | I also can think about inhibiting those logs during reconfigs and not during normal job freezing | 15:25 |
| corvus | tobiash: i think we log it every time we freeze, which happens frequently, but if we keep track of what we logged previously, we could omit it if it's the same | 15:25 |
| tobiash | since reconfigs are the most expensive parts of operations I've seen | 15:25 |
| tristanC | fungi: thanks for your help! | 15:25 |
| corvus | tobiash: that would be another good solution, which is similar. | 15:25 |
| corvus | tobiash: your suggestion might be easier to implement :) | 15:26 |
| *** dpawlik7 has quit IRC | 15:26 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 15:26 |
| corvus | the main thing is that right now you're drowing in debug logs, and the solution to that is to switch to info, but what we must not do is then cause people to drown in info logs :) | 15:26 |
| fungi | tristanC: sure, also if we as a community feel like verified +1/-1 votes from third-party ci systems is something we want, i'd recommend we add a dedicated group in our acls to enable that for specific accounts | 15:27 |
| *** dpawlik has joined #zuul | 15:27 | |
| corvus | tobiash: options #1 and #2 are nice because they may expose more of this directly to users who may be able to self-service | 15:27 |
| tobiash | I've also seen that we format those messages for storing it in the item in debug mode regardless if the pipeline is in debug mode, currently working on making that lazy | 15:27 |
| tobiash | corvus: I think 1) is already good enough since we have the job inheritance (which is essentially everything what matched) | 15:28 |
| corvus | tobiash: idea #4: store it in sql database | 15:28 |
| tristanC | fungi: why is verify label is restricted in the first place? | 15:29 |
| corvus | tobiash: good point; maybe we should look into that. | 15:29 |
| tobiash | corvus: hrm, thinking about that I think the most important analysis task we have that can only be done with the job freezing logs is 'why hasn't job x been run' in pipeline x | 15:30 |
| corvus | tobiash: we could just stick the entire 'debug' attribute in the inventory (and you could drop your lazy change) | 15:30 |
| corvus | tobiash: yeah, it'd have to be the whole item, not just that job | 15:31 |
| corvus | gotta run | 15:31 |
| tobiash | and this probably still needs the didn't match messages :/ | 15:31 |
| fungi | tristanC: mostly to deter abuse, some projects don't want cu systems leaving spurious -1 votes because the gerrit ui doesn't really distinguish them from the primary ci system's verified -1 vote and it results in reviewers skipping reviewing changes which have a random broken ci system voting on them | 15:31 |
| tobiash | I guess I need to think further in that topic | 15:31 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 744416 https://review.opendev.org/c/zuul/zuul/+/753578 | 15:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make Zookeeper mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/756716 | 15:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make ConnectionRegistry mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/757095 | 15:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 756716 and 757095 https://review.opendev.org/c/zuul/zuul/+/757148 | 15:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Instantiate executor client, merger, nodepool and app within Scheduler https://review.opendev.org/c/zuul/zuul/+/757149 | 15:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 756304 https://review.opendev.org/c/zuul/zuul/+/757097 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/c/zuul/zuul/+/717299 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Component Registry in ZooKeeper https://review.opendev.org/c/zuul/zuul/+/759187 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Move management and result events to model https://review.opendev.org/c/zuul/zuul/+/761163 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of management events https://review.opendev.org/c/zuul/zuul/+/761164 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of result events https://review.opendev.org/c/zuul/zuul/+/761165 | 15:40 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Add and fix fields in driver trigger event models https://review.opendev.org/c/zuul/zuul/+/761166 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of trigger events https://review.opendev.org/c/zuul/zuul/+/761167 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Interface to get a driver's trigger event class https://review.opendev.org/c/zuul/zuul/+/761168 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Clear list of Zookeeper connections after tests https://review.opendev.org/c/zuul/zuul/+/761169 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Increase default test wait timeout to 120s https://review.opendev.org/c/zuul/zuul/+/763754 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implementation of Zookeeper backed event queues https://review.opendev.org/c/zuul/zuul/+/761170 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implementation of Zookeeper event watcher https://review.opendev.org/c/zuul/zuul/+/761171 | 15:41 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to Zookeeper backed trigger event queues https://review.opendev.org/c/zuul/zuul/+/761172 | 15:42 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to Zookeeper backed management event queues https://review.opendev.org/c/zuul/zuul/+/761738 | 15:42 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Use logical timestamp to detect outdated changes https://review.opendev.org/c/zuul/zuul/+/763755 | 15:42 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make buildset mandatory on build https://review.opendev.org/c/zuul/zuul/+/770900 | 15:42 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implement ZooKeeper builds API https://review.opendev.org/c/zuul/zuul/+/770901 | 15:42 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to ZooKeeper backed job execution and result events https://review.opendev.org/c/zuul/zuul/+/770902 | 15:42 |
| *** rlandy is now known as rlandy|mtg | 15:58 | |
| *** jamesmcarthur has quit IRC | 16:16 | |
| *** sanjayu__ has quit IRC | 16:17 | |
| *** jamesmcarthur has joined #zuul | 16:17 | |
| openstackgerrit | Alfredo Moralejo proposed zuul/zuul-jobs master: Set zuul-jobs-test-base-roles-gentoo-17-0-systemd non-voting https://review.opendev.org/c/zuul/zuul-jobs/+/771248 | 16:19 |
| openstackgerrit | Alfredo Moralejo proposed zuul/zuul-jobs master: Rename config repos file config for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/770815 | 16:20 |
| openstackgerrit | Alfredo Moralejo proposed zuul/zuul-jobs master: Rename config repos file config for CentOS Stream https://review.opendev.org/c/zuul/zuul-jobs/+/770815 | 16:24 |
| *** jfoufas1 has quit IRC | 16:28 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 16:30 |
| *** sanjayu__ has joined #zuul | 16:35 | |
| *** sanjayu_ has joined #zuul | 16:42 | |
| *** zbr1 has joined #zuul | 16:43 | |
| *** sanjayu__ has quit IRC | 16:45 | |
| *** zbr has quit IRC | 16:45 | |
| *** zbr1 is now known as zbr | 16:45 | |
| *** hashar has quit IRC | 16:48 | |
| *** rpittau is now known as rpittau|afk | 16:49 | |
| *** rlandy|mtg is now known as rlandy | 16:54 | |
| tobiash | corvus: re logs, I'll re-do more detailed profiling tomorrow, my last profile is already quite old and when the system was less loaded. we're currently have event loop rtt of ~5min | 16:58 |
| tobiash | let's see first what else can be optimized | 16:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: runner: add execute sub-command https://review.opendev.org/c/zuul/zuul/+/630944 | 16:59 |
| *** ykarel has joined #zuul | 17:07 | |
| *** jfoufas1 has joined #zuul | 17:08 | |
| *** pabelanger has joined #zuul | 17:09 | |
| *** jfoufas1 has quit IRC | 17:09 | |
| pabelanger | We are getting ready to revamp some testing in zuul.a.c, and wanted to see if we can update a little how speculative containers work. In the case of gate pipeline, right now a change, will wait for the change a head of it to finish all container jobs, before starting. This cases a little bit of an issue for us, but think we can address it. Both changes have a 'build-container-image' job, for the | 17:11 |
| pabelanger | same image. Given, we are rebuilding the image in each change, does it make sense to wait for change A to finish, then change B to start? | 17:11 |
| pabelanger | I hope that makes sense | 17:11 |
| corvus | pabelanger: look at what zuul/zuul does; it names artifacts for the container image | 17:12 |
| corvus | so provides/requires is more granular | 17:12 |
| corvus | (and zuul/nodepool) | 17:12 |
| pabelanger | okay, I think we did that too. Our issue, is each project has the same container-image name. It isn't a 1:1 image to project. More like 1 to many | 17:13 |
| pabelanger | so, to level container is network-ee, and multiple projects build it, as those projects (collections) live inside of it | 17:13 |
| pabelanger | s/to level/top level | 17:13 |
| *** ykarel has quit IRC | 17:15 | |
| corvus | pabelanger: if two projects build the same container, then they don't require that container (and therefore you shouldn't put a 'requires: network-ee' on the job) | 17:15 |
| corvus | iow, if a job doesn't consume a container (only produces it) then don't put "requires: $containername" on that job | 17:16 |
| pabelanger | right, we don't do that. We only required, the next layer image up. | 17:17 |
| pabelanger | let me work on POC, and see if I can debug. From the sounds of it, it should work properly | 17:18 |
| fungi | i suppose you could also split it into a separate job which just builds the container, then have the jobs consuming it all only wait for that to complete? | 17:19 |
| pabelanger | yup, that is what we do today | 17:20 |
| pabelanger | I might have an idea, what is happening. Let me test | 17:21 |
| fungi | oh, wait, after rereading your problem description i think i understand better now | 17:21 |
| fungi | so it's just one job creating the image, but it runs a build on each change even if those would produce an effectively identical image | 17:22 |
| pabelanger | fungi: yah, that is right | 17:23 |
| corvus | yeah, that's the behavior we had with our initial pass of container building, before we changed to more granular provides/requires | 17:23 |
| *** jcapitao has quit IRC | 17:36 | |
| pabelanger | okay, I believe I fixed it | 17:46 |
| pabelanger | my issue was https://zuul-ci.org/docs/zuul-jobs/docker-image.html#yoursite-buildset-registry | 17:46 |
| pabelanger | requires: docker-image in buildset-registry, always waited for that, in the change a head of it | 17:46 |
| pabelanger | removing that, and falling back to our existing provides / requires seems to have worked | 17:46 |
| corvus | yeah, if you use granular provides/requires you need to get rid of the generic ones | 17:47 |
| pabelanger | cool, with that change. I think we are now ready :) | 17:48 |
| corvus | the generic ones are great when you're starting out because they're at least correct if not efficient; makes it harder for someone starting out to end up with an untested change | 17:48 |
| corvus | with the granular ones, you have to make sure you mapped out the deps correctly | 17:48 |
| *** vishalmanchanda has quit IRC | 17:59 | |
| *** jpena is now known as jpena|off | 18:07 | |
| *** sanjayu_ has quit IRC | 18:24 | |
| *** sanjayu_ has joined #zuul | 18:29 | |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** jamesmcarthur has joined #zuul | 19:22 | |
| *** jamesmcarthur has quit IRC | 19:23 | |
| *** jamesmcarthur has joined #zuul | 19:23 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** jamesmcarthur has joined #zuul | 19:26 | |
| *** vishalmanchanda has joined #zuul | 19:30 | |
| *** nils has quit IRC | 19:32 | |
| *** sanjayu_ has quit IRC | 19:41 | |
| *** jamesmcarthur has quit IRC | 20:06 | |
| *** jamesmcarthur has joined #zuul | 20:07 | |
| *** jamesmcarthur has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #zuul | 20:15 | |
| *** hanson has joined #zuul | 20:16 | |
| *** hanson has quit IRC | 20:18 | |
| *** Deknos has left #zuul | 20:45 | |
| *** jamesmcarthur has quit IRC | 21:16 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: zuul-client: test create-auth-token subcommand https://review.opendev.org/c/zuul/zuul/+/771226 | 21:28 |
| *** jamesmcarthur has joined #zuul | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** vishalmanchanda has quit IRC | 22:09 | |
| *** jamesmcarthur has joined #zuul | 23:37 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** rlandy has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!