| @morucci:matrix.org | Hi, we are attaching more and more projects using the Github driver, all projects get our Github app attached to avoid rate limit issues, however sometime it happen that a project don't have the app installed and usually we really quickly have a situation where the scheduler seems stuck on waiting for the quota to release (not processing jobs for all the others projects/tenants). Another situation leading to a scheduler stuck is when a Depends-on is set to a Github reference (repo) not part of a tenant config and for this one it seems that Zuul wants to honor the depends-on but hits quota issue. On that Zuul installation we are still using an old Zuul version (8.1), and we are wondering if the quota handling have been improved in the recent versions to limit the impact on the scheduler when we hit quota issue ? | 08:05 |
|---|---|---|
| @bingberg:matrix.org | We're setting up real time log ingestion into elastic search for zuul executors which are not reachable from the web frontend due to network topology issues, the documentation around the subject is a bit sparse but is there any way to interface with zuuls regular log streaming? As far as I understand from reading the code the zuul_console task itself only expose three variables: path, port and state and doesn't seem to have a way to send the output to an external service. | 12:27 |
| @fungicide:matrix.org | > <@morucci:matrix.org> Hi, we are attaching more and more projects using the Github driver, all projects get our Github app attached to avoid rate limit issues, however sometime it happen that a project don't have the app installed and usually we really quickly have a situation where the scheduler seems stuck on waiting for the quota to release (not processing jobs for all the others projects/tenants). Another situation leading to a scheduler stuck is when a Depends-on is set to a Github reference (repo) not part of a tenant config and for this one it seems that Zuul wants to honor the depends-on but hits quota issue. On that Zuul installation we are still using an old Zuul version (8.1), and we are wondering if the quota handling have been improved in the recent versions to limit the impact on the scheduler when we hit quota issue ? | 13:25 |
| when you say "quota" are you talking about github rate limits, or cloud resource quotas? | ||
| @fungicide:matrix.org | > <@bingberg:matrix.org> We're setting up real time log ingestion into elastic search for zuul executors which are not reachable from the web frontend due to network topology issues, the documentation around the subject is a bit sparse but is there any way to interface with zuuls regular log streaming? As far as I understand from reading the code the zuul_console task itself only expose three variables: path, port and state and doesn't seem to have a way to send the output to an external service. | 13:26 |
| the log stream is pull-oriented, not push. the streamer is an ietf rfc 1288 "finger" server, essentially | ||
| @fungicide:matrix.org | what we've (opendev) done in the past is to run a service which acts on buildset completion to fetch the associated build manifests and then use them to determine which log files were published for those builds and where to get them. service pulls down the logs, post-processes them with whatever filters we want, then feeds them to elasticsearch (or opensearch) ingestion | 13:29 |
| @fungicide:matrix.org | with that approach, you can index far more than just the console stream | 13:30 |
| @fungicide:matrix.org | dpawlik runs a system like that today, and might be able to give you some pointers | 13:31 |
| @morucci:matrix.org | @fungi, yes sorry, I mean github rate limit quota | 13:49 |
| @fungicide:matrix.org | fbo: i think in opendev we do still see github rate limit problems if we restart schedulers or trigger full reconfigures too many times in rapid succession, though yes i think there were some improvements made in semi-recent releases to reduce api calls to github in order to help the situation. i'd need to go looking through release notes to see when that was though | 13:57 |
| @fungicide:matrix.org | https://zuul-ci.org/docs/zuul/latest/releasenotes.html#relnotes-9-2-0 talks about using an api token with the github driver's app mode | 13:59 |
| @fungicide:matrix.org | that's the only thing that jumps out at me since the version you're running, but it's possible there's more that i missed or which didn't make it into a release note maybe | 14:00 |
| @morucci:matrix.org | Thanks fungi for the feedback I'll investigate more the github driver code | 14:05 |
| @jim:acmegating.com | for the past few days we've been working on database issues in opendev. current zuul master has very poor performance under mysql/mariadb with large datasets. we can fix this for mysql 8 with a simple change, but that makes things worse for mysql 5.7 and does not improve them for mariadb. i believe i know how to improve things for all three dbms systems, but it requires some slightly larger changes to zuul. at the moment, i recommend folks running mysql/mariadb avoid upgrading zuul until this is resolved (hopefully next week). | 16:28 |
| @jim:acmegating.com | Clark: given the above ^ and similar email i just sent to zuul-discuss, i think we can remove the release note in https://review.opendev.org/914990 | 16:34 |
| @jim:acmegating.com | basically treat zuul master as highly unstable for a week, then if we end up with any advice at the end of the process, we can add a release note then. | 16:35 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 914990: Restore mysql index hint https://review.opendev.org/c/zuul/zuul/+/914990 | 16:37 | |
| @clarkb:matrix.org | ++ | 16:37 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 914715: Handle annotated and signed tags when packing refs https://review.opendev.org/c/zuul/zuul/+/914715 | 17:10 | |
| @clarkb:matrix.org | Thats a debugging iteration of the change. I don't currently have a zuul test setup locally and while I figure that out I figured I could probably get results this way | 17:10 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 914715: Handle annotated and signed tags when packing refs https://review.opendev.org/c/zuul/zuul/+/914715 | 20:13 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 915188: Handle artifacts created by non change refs https://review.opendev.org/c/zuul/zuul/+/915188 | 21:23 | |
| @clarkb:matrix.org | vlotorev: corvus I don't know if that is a correct solution to the artifact problem yet but I went ahead and wrote smoething up so that we can better capture the path forward in code review | 21:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!