| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831876: Trigger pipeline runs when semaphores release https://review.opendev.org/c/zuul/zuul/+/831876 | 00:02 | |
| @jim:acmegating.com | Clark: ^ | 00:02 |
|---|---|---|
| @clarkb:matrix.org | corvus: thanks lgtm. I guess if we a rolling restart puts that event in a queue and the old scheduler doesn't know what to do with it we log an "error" but continue on otherwise | 00:09 |
| @jim:acmegating.com | yep | 00:10 |
| @jim:acmegating.com | same as with postconfig event | 00:10 |
| @clarkb:matrix.org | * corvus: thanks lgtm. I guess if during a rolling restart we put that event in a queue and the old scheduler doesn't know what to do with it we log an "error" but continue on otherwise | 00:10 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831876: Trigger pipeline runs when semaphores release https://review.opendev.org/c/zuul/zuul/+/831876 | 00:12 | |
| @jim:acmegating.com | Clark: ^ missed a call | 00:12 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831876: Trigger pipeline runs when semaphores release https://review.opendev.org/c/zuul/zuul/+/831876 | 00:27 | |
| @jim:acmegating.com | Clark: ^ and one last one | 00:27 |
| @jim:acmegating.com | my local test run is done, so that should be it :) | 00:27 |
| @jim:acmegating.com | zuul-maint: if anyone has a minute to +3 https://review.opendev.org/831876 i think that's worth rolling into the opendev zuul rolling restart (which is in progress, but paused between executors and schedulers/web right now and i plan to resume it tomorrow) | 01:50 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 831895: functional-test : ensure glean started https://review.opendev.org/c/zuul/nodepool/+/831895 | 03:27 | |
| -@gerrit:opendev.org- Ian Wienand proposed: | 03:41 | |
| - [zuul/nodepool] 788684: Switch to using openstackclient https://review.opendev.org/c/zuul/nodepool/+/788684 | ||
| - [zuul/nodepool] 788876: Switch functional test to Focal nodeset https://review.opendev.org/c/zuul/nodepool/+/788876 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 04:36 | |
| - [zuul/zuul] 831246: Add more pipeline processing stats https://review.opendev.org/c/zuul/zuul/+/831246 | ||
| - [zuul/zuul] 831249: Add even more pipeline processing stats https://review.opendev.org/c/zuul/zuul/+/831249 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 831609: Add a tenant reconfiguration metric https://review.opendev.org/c/zuul/zuul/+/831609 | 04:36 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 831906: Ignore deleted nodes when looking for lost params https://review.opendev.org/c/zuul/zuul/+/831906 | 09:01 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 831876: Trigger pipeline runs when semaphores release https://review.opendev.org/c/zuul/zuul/+/831876 | 09:05 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Albin Vass: [zuul/zuul] 831737: Simplify reportBuildEnd call https://review.opendev.org/c/zuul/zuul/+/831737 | 09:05 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 831925: Inject bundle id to inventroy zuul.items https://review.opendev.org/c/zuul/zuul/+/831925 | 10:45 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 831925: Inject bundle id to inventroy zuul.items https://review.opendev.org/c/zuul/zuul/+/831925 | 10:47 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable filtering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 12:40 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable filtering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 12:43 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 830840: Make it possible to configure job retries with zuul_return https://review.opendev.org/c/zuul/zuul/+/830840 | 13:13 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 830995: Correctly reset failing cycle behind failing item https://review.opendev.org/c/zuul/zuul/+/830995 | 13:26 | |
| -@gerrit:opendev.org- yatin proposed on behalf of Slawek Kaplonski: [zuul/zuul-jobs] 762650: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 13:44 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tobias Henkel: [zuul/zuul] 757452: WIP: Support ansible 2.10 https://review.opendev.org/c/zuul/zuul/+/757452 | 14:04 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul-client] 831946: Add web authentication https://review.opendev.org/c/zuul/zuul-client/+/831946 | 14:10 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 831948: Document how to enable web authentication with zuul-client https://review.opendev.org/c/zuul/zuul/+/831948 | 14:11 | |
| @mhuin:matrix.org | zuul-maint: 831946 and 831948 might make it more straightforward for CLI users when handling authentication | 14:12 |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul-client] 819118: Support for "basic" authentication https://review.opendev.org/c/zuul/zuul-client/+/819118 | 14:15 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul-client] 831946: Add web authentication https://review.opendev.org/c/zuul/zuul-client/+/831946 | 14:16 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 796270: Tests: add non-voting unit testing for Python 3.10 https://review.opendev.org/c/zuul/zuul/+/796270 | 14:20 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tobias Henkel: [zuul/zuul] 757452: WIP: Support ansible 2.10 https://review.opendev.org/c/zuul/zuul/+/757452 | 14:34 | |
| @jpew:matrix.org | I'm trying to get diskimage builder working with Zuul and nodepool; It seems like the suggested method is to build very bare images using DIB; whats the recommended method to install all the extra software we need (`build-essentials`, `git`, `docker` etc.) | 14:39 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831959: Clarify anonymous nodeset docs https://review.opendev.org/c/zuul/zuul/+/831959 | 14:53 | |
| @clarkb:matrix.org | > <@jpew:matrix.org> I'm trying to get diskimage builder working with Zuul and nodepool; It seems like the suggested method is to build very bare images using DIB; whats the recommended method to install all the extra software we need (`build-essentials`, `git`, `docker` etc.) | 15:04 |
| OpenDev does this with a dedicated element that lists the packages out. https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/infra-package-needs | ||
| @jpew:matrix.org | Clark: Ok, so you do install that stuff when you build the image | 15:05 |
| @clarkb:matrix.org | We don't pre install docker or build essential. But do install git and some other stuff | 15:06 |
| -@gerrit:opendev.org- yatin proposed on behalf of Slawek Kaplonski: [zuul/zuul-jobs] 762650: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 15:06 | |
| @clarkb:matrix.org | A lot of what we install is to bootstrap Ansible and zuul. Python, git, rsync, ssh, etc | 15:08 |
| @clarkb:matrix.org | Then if jobs need a compiler or container runtime we expect them to pull that in using Ansible in the job | 15:09 |
| @jpew:matrix.org | You give the user blanked sudo permissions, then revoke them? | 15:10 |
| @jpew:matrix.org | * You give the user ALL sudo permissions, then revoke them? | 15:10 |
| @jpew:matrix.org | * You give the user all sudo permissions, then revoke them? | 15:11 |
| @clarkb:matrix.org | Some jobs revoke them yes. Particularly those like unittests and linting jobs where we set the expectation that root shouldn't be required to execute the actual tests. Others don't drop the extra permissions because they are required. | 15:12 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 831873: Use JobData for Build result data https://review.opendev.org/c/zuul/zuul/+/831873 | 15:19 | |
| @blaisep-sureify:matrix.org | Just FYI, the latest quickstart docker swarm ran better on my fedora server, but ZK still showed one DB issue over night. https://paste.opendev.org/show/bUu7FEEmKlXNmDprQHrP/ | 15:26 |
| No action required and I'm happy to help investigate if you need me to. | ||
| @blaisep-sureify:matrix.org | * Hi Clark , Just FYI, the latest quickstart docker-compose tutorial ran better on my fedora server, but ZK still showed one DB issue over night. | 15:28 |
| https://paste.opendev.org/show/bUu7FEEmKlXNmDprQHrP/ | ||
| No action required and I'm happy to help investigate if you need me to. | ||
| @jim:acmegating.com | Blaise Pabon: that's expected for an idle system; it should not be a problem. | 15:30 |
| @blaisep-sureify:matrix.org | Oh!! thank you!! | 15:30 |
| @jim:acmegating.com | (basically just the idle db connection hanging up; it'll get re-established when used) | 15:31 |
| -@gerrit:opendev.org- yatin proposed on behalf of Slawek Kaplonski: [zuul/zuul-jobs] 762650: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 16:12 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 16:27 | |
| - [zuul/zuul] 831106: Update patternfly-react https://review.opendev.org/c/zuul/zuul/+/831106 | ||
| - [zuul/zuul] 831107: Add option to show overall duration in buildset table https://review.opendev.org/c/zuul/zuul/+/831107 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com marked as active: [zuul/zuul] 831873: Use JobData for Build result data https://review.opendev.org/c/zuul/zuul/+/831873 | 17:20 | |
| @y2kenny:matrix.org | I think I have asked this before but I am still not very clear about it... when there is a burst of events (either from a single push of large amount of patches, or from a Timer trigger on a repo with massive number of branch/ref), which Zuul component is responsible for processing those events to determine if those events are no-op or not? Is it the scheduler, mergers or executors? | 17:52 |
| @clarkb:matrix.org | Kenny Ho: the scheduler and mergers work together to determine that | 17:53 |
| @clarkb:matrix.org | Kenny Ho: the scheduler is the brain but it needs information from the code review system and the merged config data to make decisions. | 17:53 |
| @y2kenny:matrix.org | Clark: ok so would the scheduler be a bottle neck if there's like a burst of 500 events? or does the scheduler has some kind of load balance thing within itself also? | 17:59 |
| @y2kenny:matrix.org | what I am seeing is that when the nightly Timer triggers, basically the whole world stops for Zuul and I am wondering if scaling the Merger will fixes that. | 18:00 |
| @y2kenny:matrix.org | basically the Timer pipeline will have a large number of events accumulated but then other pipelines are essentially frozen until the Timer related pipeline clears up | 18:01 |
| @y2kenny:matrix.org | does each pipeline has their own "queue" within the scheduler for the scheduler to round-robin between the pipelines? (I am just guessing the scheduler internals...please let me know if I am not having the right mental picture.) | 18:03 |
| @clarkb:matrix.org | Kenny Ho: scaling the mergers will likely help. There are also throughput bottlenecks in the scheduler, but I'm not sure that adding more scheduelrs will help with this. THe scheduler can only handle one pipeline at a time and it sounds like these are all ending up in the same pipeline | 18:10 |
| @clarkb:matrix.org | if you scale up the schedulers then multiple pipelines can be handled concurrently | 18:10 |
| @clarkb:matrix.org | I guess if you need other pipelines to run as well then adding a scheduler will help too | 18:10 |
| @y2kenny:matrix.org | ok so scaling scheduler is an option... I didn't know about that. Does the association of pipeline to the scheduler something that needs to be manually configured or will the schedulers talk among themselves? | 18:12 |
| -@gerrit:opendev.org- yatin proposed on behalf of Slawek Kaplonski: [zuul/zuul-jobs] 762650: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/c/zuul/zuul-jobs/+/762650 | 18:15 | |
| @y2kenny:matrix.org | sounds like I need to scale both merger and scheduler because I really don't want the periodic pipeline to block my check pipeline (I am seeing that only at midnight right now but I also only have a single periodic job with lots of refs. And midnight my time would be working hour in other timezone when we have more users.) | 18:16 |
| @clarkb:matrix.org | Kenny Ho: the scheduelrs negotiate that amongst themselves | 18:19 |
| @clarkb:matrix.org | you need zuulv5 or newer to run multiple schedulers | 18:19 |
| @clarkb:matrix.org | also might be worth reconsidering what time of day you trigger the timer jobs. We try to adjust them for our quiestest period | 18:20 |
| @y2kenny:matrix.org | ok cool. I am on v5, I will give that a try. | 18:22 |
| @fungicide:matrix.org | also if you don't already have performance graphing for zuul, it can be a huge help to see where different services are spending their time in such situations | 18:23 |
| @fungicide:matrix.org | zuul has a lot of moving parts, so resource and performance trending can help you identify what to scale out | 18:24 |
| @y2kenny:matrix.org | yea... that is definitely something on my mind... proper monitoring not just for Zuul but other things as well | 18:24 |
| @fungicide:matrix.org | for example, opendev's deployment is tracked in various zuul and nodepool dashboards here: https://grafana.opendev.org/?orgId=1&viewPanel=3&search=open | 18:26 |
| @fungicide:matrix.org | the general "nodepool," "zuul performance metrics" and "zuul status" dashboards have most of what we look at | 18:27 |
| @y2kenny:matrix.org | oh and I can see the corresponding dashboard config source... I will definitely reference that | 18:28 |
| @fungicide:matrix.org | yeah, we use https://pypi.org/project/grafyaml to tyrn yaml definitions in a git repository into grafana configs | 18:29 |
| @fungicide:matrix.org | * yeah, we use https://pypi.org/project/grafyaml to turn yaml definitions in a git repository into grafana configs | 18:30 |
| @jim:acmegating.com | there's a container image for it you can run too | 18:31 |
| @jim:acmegating.com | https://hub.docker.com/r/opendevorg/grafyaml | 18:31 |
| @fungicide:matrix.org | note that the "zuul performance metrics" dashboard is a bit of a work in progress, that's just been added in the last day or so and is still in flux | 18:34 |
| @fungicide:matrix.org | has some data which will presumably be included in 5.1.0 | 18:35 |
| @fungicide:matrix.org | we're still trying to nail down how best to represent it | 18:36 |
| @y2kenny:matrix.org | understood. | 18:38 |
| @jim:acmegating.com | it's about to 1.x in size as soon as the next deployment job runs.... | 18:39 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed on behalf of Ian Wienand: [zuul/nodepool] 826541: [WIP] Revert "Remove openstacksdk from siblings job" https://review.opendev.org/c/zuul/nodepool/+/826541 | 21:07 | |
| @vlotorev:matrix.org | Hi, before removing node label from nodepool I'd like to make sure no jobs are using this label. | 21:32 |
| Is there any way to find if there are any jobs using this label? | ||
| @clarkb:matrix.org | vlotorev: we use a search across our code repositories to do that sort of thing | 21:36 |
| @clarkb:matrix.org | It may be possible to scan the zuul api's job listings for the info though | 21:36 |
| @vlotorev:matrix.org | > we use a search across our code repositories to do that sort of thing | 21:38 |
| That would mean I need to clone all repos in tenant... | ||
| Do you have any public scrawlers scripts to do that? | ||
| @vlotorev:matrix.org | Scanning over zuul api looks more prominent though. | 21:40 |
| @clarkb:matrix.org | vlotorev: no scripts, we just use https://codesearch.opendev.org | 21:41 |
| @clarkb:matrix.org | if it returns results then we know it is in use | 21:41 |
| @vlotorev:matrix.org | * > we use a search across our code repositories to do that sort of thing | 21:41 |
| That would mean I need to clone all repos in tenant... | ||
| Do you have any public scrawlers scripts to do that? | ||
| @clarkb:matrix.org | if not then its probably safe enough to proceed | 21:41 |
| @vlotorev:matrix.org | Searching the code via codesearch is briliant :) | 21:45 |
| I don't have on for local repos, there are no way to search the code in non-public Gerrit with custom ACL for different projects. | ||
| @vlotorev:matrix.org | * Searching the code via codesearch is briliant :) | 21:45 |
| I don't have one for local repos, there are no way to search the code in non-public Gerrit with custom ACL for different projects. | ||
| @vlotorev:matrix.org | * Searching the code via codesearch is briliant :) | 21:46 |
| I don't have one for local repos, there is no way to search the code in non-public Gerrit with custom ACL for different projects. | ||
| @vlotorev:matrix.org | Clark: Thanks for ideas, zuul API is a good way to go. | 21:48 |
| @clarkb:matrix.org | vlotorev: if you jobs consistently use centrally defined nodesets you may also be able to push a change removing the related nodesets and see if that results in error | 21:51 |
| @vlotorev:matrix.org | Clark: nope, it's not my case. | 21:53 |
| @jim:acmegating.com | zuul or nodepool logs or monitoring might tell you the last time they were used too | 22:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!