| @fungicide:matrix.org | > <@clarkb:matrix.org> fungi: we did unrevert the zuul-jobs sibling thing that got reverted due to tox config parsing? | 00:05 |
|---|---|---|
| Clark: the fixes for it have merged but i was holding the unrevert until later tomorrow just so i don't cause new problems right in the middle of openstack release work | ||
| @clarkb:matrix.org | Sounds good | 00:23 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] Run jobs automatically on playbook changes https://review.opendev.org/c/zuul/zuul/+/755988 | 02:06 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] Add TCP keepalive to gitlab https://review.opendev.org/c/zuul/zuul/+/812619 | 10:00 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 12:43 | |
| - [zuul/zuul] Add support for sharded ZKObjects https://review.opendev.org/c/zuul/zuul/+/812467 | ||
| - [zuul/zuul] Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | 12:44 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | 12:47 | |
| @jpew:matrix.org | @corvus I havn't seen gitlab timeouts, but my gitlab use is very light ATM | 15:17 |
| @jpew:matrix.org | I'll keep an eye out for it though | 15:17 |
| @jjardon:matrix.org | Hi, when using gerrit's topic functionality, is there a way to configure zuul so it doesnt trigger a job for each commit but only for the entire topic? (So it behaves similar to github branches) | 15:35 |
| @clarkb:matrix.org | jjardon: I think that would depend on some sort of event exposing the topic updates? The way the per change triggers work is we look for patchset created events. If there is a topic updated event or similar it should be doable. Though I suppose it may need some driver updates to understand the difference and map at topic to an internal change type event | 15:37 |
| @jjardon:matrix.org | mmm, I'm not that familiar with gerrit but I will try to see if that event exposing topic updates exist in gerrit | 15:59 |
| @clarkb:matrix.org | otherwise I think that you'll definitely need to update the gerrit driver to inspect topic state on change updated/patchset updated events and decide if those updates represent a topic update that is actional | 15:59 |
| @clarkb:matrix.org | * otherwise I think that you'll definitely need to update the gerrit driver to inspect topic state on change updated/patchset updated events and decide if those updates represent a topic update that is actionable | 16:00 |
| @ologinov:synapse.sardinasystems.com | Hello everyone! Could you please help me with this ? | 16:08 |
| @ologinov:synapse.sardinasystems.com | it's default docker-compose example, this job was stuck | 16:08 |
| @ologinov:synapse.sardinasystems.com | I used fresh Zuul release(master branch) | 16:10 |
| @ologinov:synapse.sardinasystems.com | also zuul user didn't come to gerrit and didn't "verified +1" | 16:11 |
| @clarkb:matrix.org | Oleg Loginov: my process for debugging zuul events that don't do what I expect is to find the event that triggered the action in the log which gives you an event id which you can grep for across all service logs and that gives you a good picture of what is happening | 16:12 |
| @ologinov:synapse.sardinasystems.com | i see only INFO messages in logs, it works earlier | 16:15 |
| @ologinov:synapse.sardinasystems.com | * Clark: i see only INFO messages in logs, it works earlier | 16:15 |
| @clarkb:matrix.org | you should still be able to trace the event id at an INFO and above level. Though running at debug is probably a good idea if you are deploying master | 16:18 |
| @clarkb:matrix.org | Basically I would look at all the logs that show up for the event id associated with that event and see where it got stuck | 16:21 |
| @ologinov:synapse.sardinasystems.com | I can show it, if that's what you mean | 16:25 |
| @clarkb:matrix.org | Yes, if you can share it here that might help. Though it is possible that you can identify what the problem is by looking at it too | 16:26 |
| @ologinov:synapse.sardinasystems.com | Clark: thanks! I'm also still struggling with this | 16:29 |
| @ologinov:synapse.sardinasystems.com | https://pastebin.com/d90ddnqQ | 16:29 |
| @clarkb:matrix.org | That looks like the scheduler successfully handed the job off to the executor. You should grep for the event id in the executor and also possibly 493972060a5043d0aaf3d5c0f228d679 which is the job build uuid | 16:30 |
| @clarkb:matrix.org | What your paste shows is that the event came to the scheduler, it figured out what jobs to run, got nodes from nodepool for those jobs and then told the executor to execute the job. Now the executor should have the next steps in its log | 16:31 |
| @ologinov:synapse.sardinasystems.com | thanks!ok,I should check it | 16:31 |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] Web UI: make more filters selectable in build, buildset searches https://review.opendev.org/c/zuul/zuul/+/793159 | 16:35 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] [web] Early support for pagination in builds, buildsets search https://review.opendev.org/c/zuul/zuul/+/808041 | 16:35 | |
| @ologinov:synapse.sardinasystems.com | Where I can find log file for executor? I checked /var/log in executor container- nothing | 16:38 |
| @clarkb:matrix.org | Oleg Loginov: ours logs to /var/log/zuul/executor-debug.log. I think it will depend on your setup a bit probably similar | 16:39 |
| @ologinov:synapse.sardinasystems.com | Clark: this dir doesn't exist. I still user docker-compose example setup | 16:40 |
| @ologinov:synapse.sardinasystems.com | * Clark: this dir doesn't exist. I still use docker-compose example setup | 16:40 |
| @ologinov:synapse.sardinasystems.com | hmm,debug...should I enable this option for get it file? | 16:41 |
| @clarkb:matrix.org | Oleg Loginov: the test job that exercises the quickstart docker stuff uses `docker logs zuul-tutorial_executor_1 > {{ ansible_user_dir }}/logs/executor.log 2>&1` to get the logs. I think you should ask docker for them | 16:42 |
| @jim:acmegating.com | yeah, the quickstart logs go to stdout. if you want debug, add '-d' to the command line in the compose file. | 16:42 |
| @ologinov:synapse.sardinasystems.com | ok, thanks! | 16:43 |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] [zuul-web] Add pagination information when querying builds, buildsets endpoints https://review.opendev.org/c/zuul/zuul/+/808042 | 16:43 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] Pagination: remove compatibility code https://review.opendev.org/c/zuul/zuul/+/808043 | 16:44 | |
| @jim:acmegating.com | Clark, fungi: re report of increased znode usage in #opendev -- i see a bug in the change cache cleanup that will prevent cleanup of change cache znodes. i'm working on a fix (cc tobiash, swest) | 16:45 |
| @clarkb:matrix.org | corvus: glad we could identify an issue (I suppose it may not be the issue) so quickly. | 16:45 |
| @jim:acmegating.com | this also is missing an exception handler, so the error wasn't going to the log file, only to stderr :/ | 16:51 |
| @jim:acmegating.com | (which is why we didn't notice until it appeared on the graph) | 16:51 |
| @ologinov:synapse.sardinasystems.com | > <@jim:acmegating.com> yeah, the quickstart logs go to stdout. if you want debug, add '-d' to the command line in the compose file. | 16:52 |
| added info into top of paste https://pastebin.com/jtuuG33G | ||
| @ologinov:synapse.sardinasystems.com | > <@jim:acmegating.com> yeah, the quickstart logs go to stdout. if you want debug, add '-d' to the command line in the compose file. | 16:53 |
| * added info into top of paste https://pastebin.com/jtuuG33G This paste doesn't has info about uuid job, but has info about commit id | ||
| @clarkb:matrix.org | Oleg Loginov: that shows the executor configuring the git repos to run the job. I'm thinking you may want to restart the exuector with the -d flag to get debug logs and retry | 16:56 |
| @ologinov:synapse.sardinasystems.com | Clark: ok,thanks! | 16:56 |
| @ologinov:synapse.sardinasystems.com | Clark: I just restarted service only and job was success | 16:59 |
| @ologinov:synapse.sardinasystems.com | * Clark: I just restarted service with `-d` only and job was success | 16:59 |
| @clarkb:matrix.org | hrm ok, I guess if it happens again you'll have the extra logging to debug further | 17:03 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Resolve cache keys when gathering related changes https://review.opendev.org/c/zuul/zuul/+/812732 | 17:20 | |
| @jim:acmegating.com | Clark, fungi, tobiash, swest: ^ | 17:20 |
| @clarkb:matrix.org | change lgtm | 17:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley: [zuul/zuul-jobs] Revert "Revert "Include tox_extra_args in tox siblings tasks"" https://review.opendev.org/c/zuul/zuul-jobs/+/812005 | 17:31 | |
| @fungicide:matrix.org | so needs_changes is still a list of change objects but needed_by_changes is a list of cache keys? | 17:31 |
| @jim:acmegating.com | fungi: no, both cache keys, but only need_changes needs to be recursed into, and we need real change objects for that | 17:32 |
| @fungicide:matrix.org | nevermind, yeah i misread, we still .add() items from both lists directly | 17:32 |
| @jim:acmegating.com | er s/needchanges/neededby_changes/ | 17:32 |
| @jim:acmegating.com | ya | 17:32 |
| @jim:acmegating.com | i think we don't recurse into needs_changes because we assume we'll have already gotten that from the item ahead | 17:33 |
| @fungicide:matrix.org | that makes sense, thanks | 17:33 |
| @ologinov:synapse.sardinasystems.com | just for info: I noticed this string `2021-10-06 16:58:03,622 INFO zuul.ExecutorServer: Unregistering due to low disk space 0.7% < 5.0` then checked free space (was 100% busy and free 270M only), do it `docker volume prune` -> free enough disk space -> executor works success | 17:36 |
| @fungicide:matrix.org | awesome, that's working as designed then. the executor didn't want to pick up any new builds because there was a risk they would fail to be able to write to the workspace | 17:37 |
| @ologinov:synapse.sardinasystems.com | great, I'm expected to see this info with 'CRIT' or 'ERROR' status | 17:39 |
| @fungicide:matrix.org | well, it's normal for executors to temporarily unregister when running builds are consuming too much disk, and then they reregister for new builds once space frees up after those builds complete | 17:40 |
| @ologinov:synapse.sardinasystems.com | aha, i see | 17:40 |
| @fungicide:matrix.org | in our deployment, we graph the number of executors accepting builds, and you can see there are times when no executors are available: https://grafana.opendev.org/d/5Imot6EMk/zuul-status?viewPanel=23&orgId=1&from=now-24h&to=now | 17:42 |
| @fungicide:matrix.org | zuul has a number of tunable governors which decide to unregister executors under load, for example not enough free disk, not enough free ram, too many builds already starting... | 17:43 |
| @fungicide:matrix.org | in our case when activity levels have reached the point where the executors are all too busy to take on new builds, we usually also have a backlog of node requests so are keeping up at our available capacity regardless | 17:44 |
| @ologinov:synapse.sardinasystems.com | good,i see | 17:45 |
| @ologinov:synapse.sardinasystems.com | it's cool | 17:45 |
| @ecsantos:matrix.org | hello folks. will there be talks on zuul/nodepool in the upcoming ptg? | 19:26 |
| @clarkb:matrix.org | ecsantos: I am not aware of any plans for that. Zuul has not participated in any of the virtual PTGs so far. Instead electing to use IRC and now matrix as well as the mailing list to communicate through things | 19:36 |
| @clarkb:matrix.org | ecsantos: is there something specific you are looking for? | 19:36 |
| @ecsantos:matrix.org | Clark: thanks Clark! Nothing specific, just curious :) | 19:37 |
| @clarkb:matrix.org | ok, feel free to send email to the mailing list or bring things up here if you have questions, concerns, ideas :) | 19:46 |
| @fungicide:matrix.org | ecsantos: a number of zuul community members will be participating in the ptg, just no plans to discuss zuul-specific things (except maybe in the context of the opendev collaboratory or other openinfra projects which are relying on zuul to do things) | 20:00 |
| @goneri:matrix.org | nodepool quota computation went wrong after we started to manually spawn a couple of VM. It looks like https://review.opendev.org/c/zuul/nodepool/+/785821 address our problem. | 20:35 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: [zuul/zuul] Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | 21:45 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Add LocalZKContext for job freezing https://review.opendev.org/c/zuul/zuul/+/812750 | 21:45 | |
| @jim:acmegating.com | swest: ^ i inserted a change in there that i think may make dealing with the job freezing easier; i think that will fix the failing tests too | 21:45 |
| @fungicide:matrix.org | is test_node_request_canceled known to have occasional failures? looks like https://review.opendev.org/812732 hit one on the zuul-tox-py36 job | 22:27 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/zuul/build/578b4f2a624e49838ef4044fbffc321b | 22:34 |
| @jim:acmegating.com | fungi: news to me; i'll look into that, but it's very unlikely to be related so i'll go ahead and recheck. | 22:44 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Add RepoState object https://review.opendev.org/c/zuul/zuul/+/812760 | 23:00 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Fix race in test_node_request_canceled https://review.opendev.org/c/zuul/zuul/+/812761 | 23:09 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Fix race in test_node_request_canceled https://review.opendev.org/c/zuul/zuul/+/812761 | 23:09 | |
| @jim:acmegating.com | fungi: ^ that should take care of that | 23:09 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!