| opendevreview | Ian Wienand proposed zuul/nodepool master: functional-test : add an external connectivity check https://review.opendev.org/c/zuul/nodepool/+/803301 | 02:17 |
|---|---|---|
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:48 | |
| opendevreview | Ian Wienand proposed zuul/nodepool master: [wip] functional-test : add an external connectivity check https://review.opendev.org/c/zuul/nodepool/+/803301 | 04:53 |
| *** marios is now known as marios|ruck | 05:50 | |
| swest | corvus: all in all lgtm. some thoughts on 803245 | 06:17 |
| swest | tobiash: ^ | 06:31 |
| tobiash[m] | yepp, I think adding a release note for this makes sense | 06:31 |
| tobiash[m] | we could cleanup the possibly leaked events on the old path in the cleanup worker for a while | 06:33 |
| abitrolly[m] | <fungi "abitrolly: i know there are zuul"> Oh, `bubblewrap` is nice. I basically still can't find a tool that allows me to test run an arbitrary code from GitHub to see if it is alive. I wonder how ready it is.. | 06:47 |
| opendevreview | Ian Wienand proposed zuul/nodepool master: [wip] functional-test : add an external connectivity check https://review.opendev.org/c/zuul/nodepool/+/803301 | 06:48 |
| *** jpena|off is now known as jpena | 06:54 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:30 | |
| opendevreview | Merged zuul/zuul master: Don't use transactions when sharding large events https://review.opendev.org/c/zuul/zuul/+/803154 | 07:48 |
| opendevreview | Merged zuul/zuul master: Clean up leaked event queue side channel data https://review.opendev.org/c/zuul/zuul/+/803245 | 08:01 |
| opendevreview | Merged zuul/zuul master: Store build params separate from request https://review.opendev.org/c/zuul/zuul/+/803155 | 08:01 |
| opendevreview | Merged zuul/zuul master: Cleanup leaked build request parameters https://review.opendev.org/c/zuul/zuul/+/803262 | 08:01 |
| opendevreview | Merged zuul/zuul master: Move executor api cleanup to apscheduler https://review.opendev.org/c/zuul/zuul/+/803264 | 08:01 |
| opendevreview | Merged zuul/zuul master: Remove transaction support from sharding API https://review.opendev.org/c/zuul/zuul/+/803156 | 08:01 |
| opendevreview | Merged zuul/zuul master: Remove redundant get when removing a build request https://review.opendev.org/c/zuul/zuul/+/803158 | 08:01 |
| opendevreview | Merged zuul/zuul master: Vendor and use fixed Kazoo read/write lock https://review.opendev.org/c/zuul/zuul/+/803166 | 08:01 |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Store runtime related system attributes together https://review.opendev.org/c/zuul/zuul/+/801749 | 08:33 |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Cache system config in Zookeeper https://review.opendev.org/c/zuul/zuul/+/801859 | 08:33 |
| mhu1 | has anybody ever experienced mem leaks in nodepool-builder? I've just seen our process killed by OOMKiller, but not sure if it's just an outlier | 09:28 |
| mhu1 | it's running on a machine with qemu-kvm, with 8GB RAM, so there should be plenty of room | 09:32 |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 10:20 | |
| fungi | mhu1: our builders similarly have 8gb ram, but don't use much of it for anything besides cache: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68028&rra_id=all (we also configure them with an additional 8gb swap just in case, which seems to go mostly untouched) | 11:15 |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 11:16 |
| fungi | granted, our builders are running on xen domu guests at the moment, not kvm | 11:16 |
| fungi | though i doubt that difference would expose memory leaks | 11:18 |
| *** dviroel|out is now known as dviroel | 11:18 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| *** jpena|lunch is now known as jpena | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 12:13 |
| mhu1 | fungi, ok thanks for the feedback. I'll keep an eye on it and see if it happens again | 12:26 |
| *** jpena|lunch is now known as jpena | 12:33 | |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Cache system config in Zookeeper https://review.opendev.org/c/zuul/zuul/+/801859 | 13:49 |
| opendevreview | Simon Westphahl proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 13:49 |
| opendevreview | Felix Edel proposed zuul/zuul master: Add MergerApi https://review.opendev.org/c/zuul/zuul/+/787686 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Fix test_gerrit.TestPolling.test_config_update https://review.opendev.org/c/zuul/zuul/+/773023 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Execute merge jobs via ZooKeeper https://review.opendev.org/c/zuul/zuul/+/802299 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Store merge params separate from request https://review.opendev.org/c/zuul/zuul/+/803347 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Move merge request cleanup to Merger API https://review.opendev.org/c/zuul/zuul/+/803348 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Clean up leaked merge request params https://review.opendev.org/c/zuul/zuul/+/803349 | 14:05 |
| opendevreview | Felix Edel proposed zuul/zuul master: Don't use transactions when writing merge results https://review.opendev.org/c/zuul/zuul/+/803350 | 14:05 |
| felixedel | corvus: I've uploaded a few follow-up changes to the Merger API that remove the transactions when writing the merge request params and deal with the cleanup of lost merge request params (similar to the Executor API). I've also rebased the whole stack on top of master (which already includes the relevant Executor API changes) | 14:09 |
| Guest3154 | felixedel: great! | 14:10 |
| felixedel | If you want you can squash them but I think it wouldn't harm to keep them separated. | 14:11 |
| felixedel | And I think the last change https://review.opendev.org/c/zuul/zuul/+/803350/1 is failing the tests, but I haven't found the root cause yet. It seems to be related to the MergerResultEventFuture. Maybe I missed something when removing the transaction. | 14:13 |
| felixedel | If you find time, feel free to continue with those patches, otherwise I will do so tomorrow | 14:14 |
| Guest3154 | felixedel: sounds good | 14:17 |
| Guest3154 | swest, tobiash, felixedel, clarkb: re clock skew: do you think 5m is too short? i think we could easily lengthen it to 15m and be okay | 14:19 |
| Guest3154 | these cleanups aren't time sensitive -- they just need to happen to clean up any potential leaks | 14:19 |
| fungi | corvus: your nick is showing up as "Guest3154" through the oftc bridge | 14:30 |
| Guest3154 | fungi: ah. it looks like there was an "event" earlier, and i think the bridge lost some traffic too. :/ | 14:31 |
| fungi | neat | 14:32 |
| swest | corvus: yeah I think I'd increase it a bit. Maybe even to 30min. I don't expect a lot of leaked event data nodes | 14:34 |
| erbarr | Hi, I have a tenant where I'm loading the config project from my git, as an untrusted project I'm using opendev/ci-sandbox. In my config project, I have zuul.d with pipelines.yaml, projects.yaml, jobs.yaml. I believe I have defined a test job to just do an "ip a" in a check pipeline, however, I'm not seeing any check entry in the status page nor does the test job show up in the jobs list. I'm not seeing any errors pop up either, any | 14:38 |
| erbarr | idea what I could be doing wrong? | 14:38 |
| Guest3154 | erbarr: check the scheduler log, it describes the choices zuul makes about enqueing changes and starting jobs | 14:39 |
| erbarr | Guest3154: thanks, location of this? | 14:39 |
| *** Guest3154 is now known as notcorvus | 14:39 | |
| *** notcorvus is now known as corvus | 14:40 | |
| *** corvus is now known as Guest3197 | 14:40 | |
| Guest3197 | erbarr: are you the zuul system administrator? | 14:40 |
| clarkb | corvus you flapped back to Guest after identifying | 14:41 |
| clarkb | mhu1: fungi: one thing to keep in mind is that the image build itself could be ooming | 14:41 |
| erbarr | Guest3197, yea, I'm installing the whole thing | 14:41 |
| *** Guest3197 is now known as corvus | 14:42 | |
| *** corvus is now known as notcorvus | 14:42 | |
| *** notcorvus is now known as corvus | 14:42 | |
| *** corvus is now known as Guest3200 | 14:43 | |
| Guest3200 | erbarr: if you're running containers, then the container log; otherwise /var/log/zuul by default, or there's a zuul.conf option to supply your own log config | 14:43 |
| erbarr | I don't see a /var/log/zuul folder, I launched the scheduler, executor, and web, with the -f option though so I see the running process and I'm not seeing any errors. I had tried adding openstack/ironic as untrusted-project and that would lead to some errors with projects missing, which ended up in an ever growing list of run the scheduler and see what project is missing add it and try again, I was seeing no end to that. But anyways, I | 14:50 |
| erbarr | went simpler and no errors, just says that it loaded those files from my config project | 14:50 |
| Guest3200 | erbarr: -f means run in foregroud and that disabled logging, so it'll go to stdout or stderr. i'm assming you created a change. you should see log entries with gerrit events in the scheduler logs. check around those to see what zuul decided to do with it. | 14:52 |
| Guest3200 | correction: not disabled logging, disabled writing to the log file. | 14:53 |
| erbarr | ahh, ok, yea, I have a patch in ci-sandbox, are trigger events DEBUG log entries? | 14:54 |
| *** jcapitao3 is now known as jcapitao | 14:55 | |
| Guest3200 | erbarr: if nothing at INFO helps, you may need to add "-d" to get debug. zuul is a little less verbose about things it doesn't do at info level, generally focusing on things it does do. | 15:00 |
| Guest3200 | and you're looking for a thing it's not doing :) | 15:00 |
| erbarr | lol, yea, I do see the event with the -d option, I'm trying to follow it | 15:01 |
| Guest3200 | erbarr: there's an event id ([e: ...] in logs) that may help you trace it. but note there still could be some relevant log entries without it | 15:02 |
| erbarr | ok, thanks | 15:04 |
| *** jpena is now known as jpena|off | 15:06 | |
| *** Guest3200 is now known as corvus | 15:17 | |
| *** corvus is now known as notcorvus | 15:17 | |
| *** notcorvus is now known as corvus | 15:17 | |
| corvus | am i me? | 15:21 |
| corvus | yay i am | 15:21 |
| fungi | you are! | 15:29 |
| fungi | (or a convincing imposter) | 15:30 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul master: Add tenant name on NodeRequests for Nodepool https://review.opendev.org/c/zuul/zuul/+/788680 | 15:30 |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| opendevreview | Tristan Cacqueray proposed zuul/zuul master: [web] Update the TenantsPage to use React Hooks https://review.opendev.org/c/zuul/zuul/+/803375 | 16:14 |
| tristanC | corvus: here is a quick draft for how functional component with react hook looks like: https://review.opendev.org/c/zuul/zuul/+/803375 | 16:16 |
| corvus | tristanC: thanks -- that's very helpful -- i have a followup Q that's better for email, so i'll reply there | 16:22 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul master: [web] Update the TenantsPage to use React Hooks https://review.opendev.org/c/zuul/zuul/+/803375 | 16:27 |
| tristanC | corvus: in PS2 I added back error management by dispatching an action to the existing error reducer | 16:28 |
| *** marios|ruck is now known as marios|out | 16:30 | |
| *** ssbarnea[m] is now known as zbr | 16:31 | |
| corvus | tristanC: ++ replied in email | 16:36 |
| tristanC | great questions, let me add more details to the thread :-) | 16:41 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul master: [web] Update the TenantsPage to use React Hooks https://review.opendev.org/c/zuul/zuul/+/803375 | 16:46 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul master: [web] Update the TenantsPage to use React Hooks https://review.opendev.org/c/zuul/zuul/+/803375 | 17:27 |
| *** dviroel is now known as dviroel|brb | 17:34 | |
| *** dviroel|brb is now known as dviroel | 18:42 | |
| opendevreview | lotorev vitaly proposed zuul/zuul master: DNM: testing job.required-projects inheritance https://review.opendev.org/c/zuul/zuul/+/803391 | 19:01 |
| opendevreview | lotorev vitaly proposed zuul/zuul master: DNM: testing job.required-projects inheritance https://review.opendev.org/c/zuul/zuul/+/803391 | 19:08 |
| *** dviroel is now known as dviroel|out | 20:54 | |
| opendevreview | lotorev vitaly proposed zuul/zuul master: doc: Specify job.required-projects is overriden https://review.opendev.org/c/zuul/zuul/+/803269 | 21:11 |
| opendevreview | lotorev vitaly proposed zuul/zuul master: doc: Specify job.required-projects is extended by child jobs https://review.opendev.org/c/zuul/zuul/+/803269 | 21:12 |
| opendevreview | James E. Blair proposed zuul/zuul master: Clean up lost merge request results / data https://review.opendev.org/c/zuul/zuul/+/803398 | 21:35 |
| clarkb | corvus: is ^ that stack next in the list of sos related changes? | 21:36 |
| corvus | clarkb: yes -- but i do need to squash 3 changes together | 21:42 |
| clarkb | ok I'll wait then | 21:42 |
| corvus | felixedel: (clarkb fyi) earlier you suggested keeping the original changes and the transaction-removal changes separate, however, because of the removal of transaction support from the underlying shard class, the originals don't work anymore, so i think we do need to squash them. we also need to squash the change i just pushed up (803398), as it fixes a cleanup routine from the old code which doesn't work any more. | 21:43 |
| corvus | that passes for me locally, so i'll squash them now, and then fixup a few things with the "execute merge jobs via zk" change too in just a minute | 21:44 |
| corvus | well, few minutes -- this will involve some local test runs | 21:45 |
| corvus | fyi zuul folks -- we just accidentally had a really long debugging conversation in #opendev about stuck node requests after a zk restart. i think we have identified the issue and have a fix planned. i'll upload a change with an explanatory commit message in a bit | 23:24 |
| opendevreview | James E. Blair proposed zuul/zuul master: Add MergerApi https://review.opendev.org/c/zuul/zuul/+/787686 | 23:56 |
| opendevreview | James E. Blair proposed zuul/zuul master: Fix test_gerrit.TestPolling.test_config_update https://review.opendev.org/c/zuul/zuul/+/773023 | 23:56 |
| opendevreview | James E. Blair proposed zuul/zuul master: Execute merge jobs via ZooKeeper https://review.opendev.org/c/zuul/zuul/+/802299 | 23:56 |
| corvus | okay that's the squashed set | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!