| *** hamalq has quit IRC | 00:02 | |
| *** shanemcd has quit IRC | 00:02 | |
| *** shanemcd has joined #zuul | 00:03 | |
| *** y2kenny has quit IRC | 00:07 | |
| *** fsvsbs has quit IRC | 01:32 | |
| *** jangutter_ has joined #zuul | 02:14 | |
| *** jangutter has quit IRC | 02:17 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #zuul | 02:33 | |
| corvus | i'm beginning to doubt that i'm going to get useful data in a reasonable amount of time. | 03:13 |
|---|---|---|
| corvus | i think i have a handle on the kind of search to do next, and hopefully i can do it during a period where there aren't quite so many objects for it to churn through | 03:18 |
| *** ykarel|away has joined #zuul | 03:50 | |
| *** ykarel|away is now known as ykarel | 03:54 | |
| *** ajitha has joined #zuul | 03:59 | |
| *** jfoufas1 has joined #zuul | 04:27 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to ZooKeeper backed merge result events https://review.opendev.org/c/zuul/zuul/+/784195 | 05:55 |
| *** jangutter has joined #zuul | 06:05 | |
| *** fsvsbs has joined #zuul | 06:05 | |
| *** jangutter_ has quit IRC | 06:08 | |
| *** saneax has joined #zuul | 06:16 | |
| *** hashar has joined #zuul | 06:39 | |
| *** reiterative has quit IRC | 06:49 | |
| *** reiterative has joined #zuul | 06:49 | |
| *** jcapitao has joined #zuul | 07:00 | |
| *** tosky has joined #zuul | 07:45 | |
| openstackgerrit | Daniel Blixt proposed zuul/zuul-jobs master: WIP: Make build-sshkey handling windows compatible https://review.opendev.org/c/zuul/zuul-jobs/+/780662 | 07:47 |
| *** ykarel has quit IRC | 08:03 | |
| *** ykarel has joined #zuul | 08:05 | |
| *** fsvsbs has quit IRC | 08:14 | |
| *** nils has joined #zuul | 08:33 | |
| *** jangutter_ has joined #zuul | 08:57 | |
| *** ykarel is now known as ykarel|lunch | 08:58 | |
| *** jangutter has quit IRC | 09:00 | |
| *** ykarel|lunch is now known as ykarel | 10:10 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul master: WIP: Document tox environments https://review.opendev.org/c/zuul/zuul/+/766460 | 10:13 |
| *** jcapitao is now known as jcapitao_lunch | 10:39 | |
| *** hashar is now known as hasharLunch | 11:02 | |
| *** rlandy has joined #zuul | 11:42 | |
| *** jcapitao_lunch is now known as jcapitao | 11:51 | |
| *** hasharLunch is now known as hashar | 11:58 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul master: Document local testing https://review.opendev.org/c/zuul/zuul/+/766460 | 12:13 |
| *** bhagyashris has quit IRC | 12:28 | |
| *** bhagyashris has joined #zuul | 12:29 | |
| *** sanjayu_ has joined #zuul | 12:35 | |
| *** saneax has quit IRC | 12:36 | |
| *** jangutter_ is now known as jangutter | 12:38 | |
| jangutter | Newbie question on zuul config repos: we have 2 config repos for our system. One defines the pipelines and some projects, the other defines the base jobs. | 12:41 |
| jangutter | I submitted a review to the projects config-repo and Zuul reported a config error: some/untrusted/project/zuul.d/main.yaml: "Job base-job-X not defined". | 12:42 |
| jangutter | But, base-job-X which inherits from the default base job, is defined in the base-job config repo.... | 12:43 |
| avass | jangutter: it sounds like zuul hasn't loaded base-job-X for some reason | 12:43 |
| avass | jangutter: you can check what jobs are available in: https://zuul.opendev.org/t/zuul/jobs for example | 12:44 |
| avass | jangutter: and config errors at: https://zuul.opendev.org/t/zuul/config-errors maybe there's some error in the config for base-job-X | 12:44 |
| jangutter | avass: that https://zuul-ui/job/base-job-X is showing up nicely (it's just been merged) | 12:45 |
| avass | jangutter: was the change dependent on that job submitted shortly after? what happens if you recheck it, still config error? | 12:46 |
| *** nhicher_ has joined #zuul | 12:48 | |
| jangutter | avass: still syntax error. and the t/tenantname/config-errors is just blank. | 12:51 |
| avass | jangutter: weird then I can only suspect a mispelled parent name or something like that | 12:53 |
| jangutter | avass: we're using load-branch in our tenant config to use the "main" branch.... would that interfere? | 12:55 |
| jangutter | avass: the fun thing is that the job is _running correctly_ in the untrusted project. But an entirely unrelated change in the config project causes troubles for Zuul... | 12:56 |
| jangutter | avass: bwahaha, I think we hit this one: https://review.opendev.org/c/zuul/zuul/+/762354 | 12:58 |
| avass | jangutter: ah yep that sounds like that could be it :) | 12:59 |
| jangutter | avass: I manually merged the change and it's working as intended, so it only affects the config-repo checks. | 13:00 |
| *** sanjayu__ has joined #zuul | 13:06 | |
| *** sanjayu_ has quit IRC | 13:09 | |
| zbr|rover | fungi: ok drop use of bionic for zuul? related to lack of yarnpkg on deps. | 13:38 |
| zbr|rover | apparently installing yarn using bindep is not quite so portable | 13:39 |
| fungi | not quite so portable how? are we testing doc builds on bionic? | 13:39 |
| fungi | it might be a reason to do doc builds on focal instead | 13:40 |
| zbr|rover | that is what i did, but with an undesired side effect, look at https://review.opendev.org/c/zuul/zuul/+/766460/ | 13:40 |
| zbr|rover | fixed the docs, bug broke something else :D | 13:40 |
| fungi | probably the addition of compile test there | 13:41 |
| zbr|rover | probably fixable by removing test/compile, but it would not be correct, as they are needed for that. | 13:41 |
| fungi | the alternative is to specify that it's only on versions of distros which provide it (everyone else is on their own to work out how to get yarn installed anyway, like before) | 13:42 |
| zbr|rover | my hopes were to avoid using versions as I hate wasting time updating these files every ~6mo | 13:44 |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul master: Document local testing https://review.opendev.org/c/zuul/zuul/+/766460 | 13:44 |
| zbr|rover | at least I got it working, it also involved a small fix on program-output, got a release yesterday. finally now it displays the failure reason when it does. | 13:45 |
| zbr|rover | used to be very cryptic | 13:45 |
| fungi | one way to avoid updating distro versions is to say something like platform:dpkg !platform:ubuntu-bionic | 13:46 |
| fungi | that strikes a bit of a balance, in that the zuul jobs won't try to install the yarnpkg package on ubuntu-bionic, if someone tries running bindep locally on a debian-stretch system it will perpetually report they need yarnpkg, but that's not the end of the world (at least serves as a reminder they need to go find yarn from somewhere, they can even satisfy it from stretch-backports) | 13:48 |
| zbr|rover | fungi: there is a tox-bindep plugin I wrote which could tell people what they miss before running the job, but i am not sure you will want me to enable it by default. | 13:54 |
| fungi | well, if people choose to install tox-bindep it'll do the thing they install it to do, right? | 14:05 |
| *** ajitha has quit IRC | 14:24 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: ensure-kubernetes: remove dns resolvers hack https://review.opendev.org/c/zuul/zuul-jobs/+/784427 | 14:37 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Remove command args override https://review.opendev.org/c/zuul/zuul-operator/+/784181 | 14:38 |
| *** jangutter_ has joined #zuul | 14:40 | |
| *** jangutte_ has joined #zuul | 14:41 | |
| *** jangutter has quit IRC | 14:42 | |
| *** jangutter_ has quit IRC | 14:45 | |
| *** sanjayu__ has quit IRC | 14:51 | |
| zbr|rover | fungi: yes. that plugin runs bindep before tox commands. it works even if you install it side tox, no need to touch tox.ini file. | 14:58 |
| zbr|rover | it may prove annoying if bindep.txt files were not written for the platform you are running on. | 14:59 |
| *** zbr|rover is now known as zbr | 15:29 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Document algorithm var for remove-build-sshkey https://review.opendev.org/c/zuul/zuul-jobs/+/783988 | 15:30 |
| *** jfoufas1 has quit IRC | 15:30 | |
| Shrews | fungi, et. al. that change ^ reminds me that remind you all that a new ansible 2.11 feature will allow validating role arguments at run time (https://docs.ansible.com/ansible-core/devel/user_guide/playbooks_reuse_roles.html#role-argument-validation). So the things you have in README now for documentation might be better served by moving them to `meta/main.yml` at some point. That would also allow `ansible-doc` to display the role docs. | 15:34 |
| Shrews | just fyi. totally not required | 15:34 |
| fungi | Shrews: oh neat! | 15:35 |
| corvus | structured docs are great :) | 15:35 |
| zbr | finally! i was dreaming about that for a very long time. | 15:36 |
| zbr | it should not be very hard to migrate the docs from zuul-jobs to use the newer spec | 15:37 |
| corvus | probably automatable | 15:41 |
| corvus | i think the first step would be to get the new-style docs to render in sphinx so that https://zuul-ci.org/docs/zuul-jobs/ works | 15:42 |
| *** ykarel is now known as ykarel|away | 15:43 | |
| *** ykarel|away has quit IRC | 15:53 | |
| *** hashar has quit IRC | 15:58 | |
| zbr | ansible community team will need do the same for collections | 16:12 |
| *** sshnaidm is now known as sshnaidm|afk | 16:20 | |
| *** jcapitao is now known as jcapitao_afk | 16:21 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 16:37 |
| *** hamalq has joined #zuul | 16:40 | |
| tristanC | it seems like `use-buildset-registry` somehow disables the etcd service started by `ensure-kubernetes`, resulting in a broken cluster ( Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused ) | 16:50 |
| openstackgerrit | Merged zuul/zuul master: Use load-branch with trusted repository https://review.opendev.org/c/zuul/zuul/+/762354 | 16:51 |
| tristanC | or perhaps it is something else, the first visible issue happens when the role is being applied, it cause `pkg/osutil: received terminated signal, shutting down...` | 17:00 |
| avass | tristanC: use-buildset-registry restarts the docker daemon so maybe that messes with it somehow | 17:07 |
| *** jangutter has joined #zuul | 17:09 | |
| *** jangutte_ has quit IRC | 17:12 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Remove command args override https://review.opendev.org/c/zuul/zuul-operator/+/784181 | 17:21 |
| tristanC | avass: maybe, it's odd though because the api server does work for some request after the docker restart.. i've added a retry loop in ^ | 17:22 |
| *** nils has quit IRC | 17:43 | |
| *** jcapitao_afk has quit IRC | 17:46 | |
| *** jcapitao_afk has joined #zuul | 17:48 | |
| corvus | clarkb, fungi, tobiash, swest: erm, i think i caught the memleak: https://i.imgur.com/obJIexi.png -- it is related to semaphores, but it's not because of the layout reference. if we removed the layout reference it's true we wouldn't leak layouts anymore (and wouldn't leak as much memory), but the real issue is that we're leaking semaphorehandlers because they inherit from ZooKeeperBase and therefore the | 17:53 |
| corvus | zk client keeps references to them. so we need to clean those up. if we do that, the layouts should get cleaned up too. | 17:53 |
| fungi | aha! | 17:53 |
| corvus | it's actually such a well-behaved leak, i got that from the quick-start setup where i was prototyping the next data collection. so should be really easy to verify the fix. | 17:54 |
| *** jcapitao_afk has quit IRC | 17:55 | |
| clarkb | corvus: its the onconnect and ondisconnect method registration that keeps the refs alive? | 17:55 |
| corvus | ya | 17:55 |
| clarkb | cool, I'm glad that got tracked down. Any idea why that didn't show up in the investigation yesterday? | 17:56 |
| corvus | i think unlike other zk base objects, we discard the semaphorehandler each time we reconfigure, so it's not really long-lived | 17:56 |
| corvus | clarkb: we just never completed the objgraph search because the search space was too big | 17:56 |
| clarkb | ah | 17:56 |
| clarkb | if we are going to do a big zuul restart to land that fix getting in https://review.opendev.org/c/zuul/zuul/+/784142 as part of it is likely a good idea too (assuming people agree with my analysis on that change) | 17:57 |
| corvus | clarkb: i will look at that more in depth tonight/tomorrow and make sure to approve if it hasn't been, but just from reading the 2 lines of code added, lgtm :) | 17:58 |
| corvus | i've got to task switch away for the rest of the day | 17:58 |
| clarkb | ok | 17:59 |
| clarkb | would it be helpful if someone looked at fixing the leak too? | 17:59 |
| fungi | there's no huge hurry for opendev's sake at least... looking at our current memory consumption i doubt we'll get in trouble in again until monday at the earliest | 18:00 |
| corvus | i'm not gonna say no, but i'm planning on doing it tonight/tomorrow :) | 18:01 |
| clarkb | ok | 18:01 |
| clarkb | I'm probably nto going to jump right into that as from memory it isn't quite clear to me how one would signal that a semaphorehandler was done being used. Maybe a call at the old code that moved the old semaphore forward to unregister the current semaphore from the zk client? | 18:02 |
| clarkb | actually ya that might be a simple thign to write I'll give it a quick go and others can feel free to replace it or update it with something ebtter | 18:11 |
| *** ykarel|away has joined #zuul | 18:13 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update operator-framwork to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784457 | 18:20 |
| corvus | clarkb: i was going to look into whether it needs to inherit from zk base at all | 18:20 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Fix SemaphoreHandler leak https://review.opendev.org/c/zuul/zuul/+/784458 | 18:22 |
| clarkb | corvus: oh that is a good point, it does seem like it doesn't use the on connect and on disconnect stuff (yet?) | 18:23 |
| clarkb | anyway feel free to replace that change or push new patchsets to it. Won't bother me at all | 18:23 |
| * clarkb lunches | 18:24 | |
| *** ykarel|away has quit IRC | 18:25 | |
| clarkb | ate a quick lunch and confirmed that SemaphoreBase seems to only use self.kazoo_client. Let me see about changing this up a little | 18:43 |
| fungi | https://zuul-ci.org/docs/zuul/reference/developer/specs/circular-dependencies.html is implemented now, right? | 18:45 |
| fungi | also is https://zuul-ci.org/docs/zuul/reference/developer/specs/kubernetes-operator.html done? seems like we have people using it, or trying to | 18:46 |
| fungi | also i suppose a point of process... do we remove completed specs after the implementations merge, or after they appear in a tagged release? | 18:46 |
| fungi | i can't remember if there are still bits of https://zuul-ci.org/docs/zuul/reference/developer/specs/tenant-scoped-admin-web-API.html which haven't merged yet | 18:49 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Fix SemaphoreHandler leak https://review.opendev.org/c/zuul/zuul/+/784458 | 19:01 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 19:07 |
| *** jangutter_ has joined #zuul | 19:32 | |
| *** jangutter has quit IRC | 19:35 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 20:03 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 20:20 |
| *** nhicher_ has quit IRC | 20:21 | |
| *** holser has joined #zuul | 20:34 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 20:39 |
| mordred | pabelanger: ^^ left a drive-by comment | 20:52 |
| *** decimuscorvinus has quit IRC | 21:42 | |
| *** decimuscorvinus has joined #zuul | 21:42 | |
| clarkb | https://review.opendev.org/c/zuul/zuul/+/784458 did pass testing | 21:44 |
| clarkb | harder to say it absolutely fixes the leaks but at least that is a good sign | 21:44 |
| corvus | clarkb: ++ i'll test it locally tonight | 21:50 |
| *** pabelanger has joined #zuul | 21:50 | |
| pabelanger | so, I finally got https://review.opendev.org/c/zuul/zuul-jobs/+/765177 to pass. I think we have enough testing in place to ensure no breaks with ensure-podman | 21:51 |
| pabelanger | we've been hitting some random podman issues in zuul.a.c, that's mostly why I started working on it again | 21:51 |
| clarkb | guillaumec: I'm trying to catch up on https://review.opendev.org/c/opendev/gear/+/784083 and I guess the issue is you cannot use select/poll/etc on ssl sockets unless they are non blocking? | 23:20 |
| clarkb | and that chagnes switches to a non blocking setup. I think that will work due to how the blocking variant was already only handling a packet at a time | 23:20 |
| clarkb | (basically lbocking vs non blocking was pretty transparent to end users as they just got gearman jobs/events | 23:21 |
| clarkb | guillaumec: I wonder if we even need Connection anymore | 23:26 |
| clarkb | seems the server side was already nonblocking and now the client side is being switched to non blocking too | 23:26 |
| clarkb | and worker uses baseclient too so it is converted as well | 23:26 |
| clarkb | if we are worried about it causing user visible behavior differences we could switch on whether or not ssl is used | 23:32 |
| *** jangutter_ has quit IRC | 23:38 | |
| *** tosky has quit IRC | 23:40 | |
| corvus | clarkb: https://review.opendev.org/784458 fixes the leak in my simple local test; i think we should get it merged in prep for the next restart. <-- zuul-maint | 23:41 |
| *** rlandy has quit IRC | 23:42 | |
| corvus | clarkb: on 784142 did you eventually get that test to fail before your fix? | 23:43 |
| clarkb | https://review.opendev.org/c/zuul/zuul/+/784165 and https://review.opendev.org/c/zuul/zuul/+/784142 are the other two fixes that might be good to get in too | 23:43 |
| clarkb | corvus: yes the key was the content of the event | 23:44 |
| clarkb | or repo_state is what it calls it I think | 23:44 |
| clarkb | in particular I was passing in a refs/changes/xy/abxy: $rev entry in the repo state but the executors never fetch the changes in that way | 23:44 |
| clarkb | they do a fetch_head merged into target ref of refs/zuul/uuid | 23:45 |
| clarkb | anyway because they never have that ref it tripped the first condition always in testing. When I dropped that after understanding repo_state better the test did what I expected | 23:45 |
| clarkb | one thing that recently occurred to me is how periodic jobs would work if they just check out master too | 23:48 |
| clarkb | it is certainly odd how we haven't seemed to notice before but all the behavior we observed directly on the executors seemed to point to this | 23:48 |
| corvus | we almost never ff? :) | 23:50 |
| clarkb | could be | 23:50 |
| corvus | clarkb: lgtm; i'd like tobiash to take a look at it; do you think it can wait until next week, or should we merge now and ask for a retro-review? | 23:50 |
| clarkb | it can probably wait | 23:51 |
| clarkb | considering we only just noticed the behavior that seems to have beenthere all along | 23:51 |
| corvus | clarkb: i'm pretty weakly -1 on 784165 | 23:56 |
| clarkb | oh I didn't realize that we were also passing exception objects outside of the except: context before | 23:58 |
| corvus | i believe we thread-shifted them | 23:58 |
| clarkb | I agree that unless something is fundamentally odd this should be fine as is | 23:58 |
| clarkb | ah | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
