| *** jamesmcarthur has joined #zuul | 00:08 | |
| *** tosky has quit IRC | 00:15 | |
| *** rlandy has quit IRC | 00:21 | |
| *** hamalq has quit IRC | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #zuul | 01:35 | |
| *** zenkuro has quit IRC | 01:59 | |
| *** jamesmcarthur has quit IRC | 02:24 | |
| *** jamesmcarthur has joined #zuul | 02:37 | |
| *** wuchunyang has joined #zuul | 02:45 | |
| *** jamesmcarthur has quit IRC | 03:41 | |
| *** jamesmcarthur has joined #zuul | 03:43 | |
| *** wuchunyang has quit IRC | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** jamesmcarthur has joined #zuul | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** saneax has joined #zuul | 05:09 | |
| *** ykarel has joined #zuul | 05:23 | |
| *** ykarel_ has joined #zuul | 05:26 | |
| *** ykarel has quit IRC | 05:29 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** wuchunyang has joined #zuul | 05:35 | |
| *** vishalmanchanda has joined #zuul | 05:35 | |
| *** jamesmcarthur has joined #zuul | 06:00 | |
| *** ykarel_ has quit IRC | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:05 | |
| *** jfoufas1 has joined #zuul | 06:10 | |
| *** jamesmcarthur has joined #zuul | 07:09 | |
| *** jcapitao has joined #zuul | 07:09 | |
| *** jamesmcarthur has quit IRC | 07:15 | |
| *** jamesmcarthur has joined #zuul | 07:30 | |
| *** jamesmcarthur has quit IRC | 07:34 | |
| *** hashar has joined #zuul | 07:37 | |
| *** saneax has quit IRC | 07:37 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** ykarel_ has joined #zuul | 07:56 | |
| *** ykarel__ has joined #zuul | 07:58 | |
| *** ykarel__ is now known as ykarel | 08:01 | |
| *** ykarel_ has quit IRC | 08:01 | |
| *** rpittau|afk is now known as rpittau | 08:08 | |
| *** tosky has joined #zuul | 08:22 | |
| *** jamesmcarthur has joined #zuul | 08:51 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** piotrowskim has joined #zuul | 09:04 | |
| *** nils has joined #zuul | 09:21 | |
| avass | we're seeing zuul-web disconnecting from gearman because of a timeout now and then, I'm not sure why yet, but this doesn't seem to crash zuul-web and it also doesn't try to reconnect to gearman from what I can see | 09:24 |
|---|---|---|
| avass | so that makes us stuck with a pod in a bad state, adding a liveness probe probably mitigates that but is there any fix for this yet? | 09:26 |
| *** mgoddard has joined #zuul | 09:32 | |
| tobiash | avass: we've seen this as well and I think this might be thread starvation since zuul-web waits until every request is processed by the scheduler and there the processing is single threaded | 09:43 |
| tobiash | we're using a liveness probe to circumvent this | 09:44 |
| tobiash | and https://review.opendev.org/c/zuul/zuul/+/708377 | 09:44 |
| avass | tobiash: yeah I suspected something like that | 09:47 |
| avass | but I think it would be a good idea if the pod could crash or try to reconnect if it doesn't have a gearman connection | 09:49 |
| tobiash | it does have a gearman connection, it's just that all threads wait for a task to finish in the scheduler (at least that's my assumption so far) | 09:52 |
| avass | I'm getting "No connected Gearman servers" after the first timeout so my assumption was that it disconnects :) | 09:55 |
| avass | but I haven't spent a lot of time debugging this yet | 09:56 |
| tobiash | hrm, then your issue seems to be different than ours | 09:58 |
| tobiash | but gear should actually reconnect automatically | 09:58 |
| tobiash | that's weird | 09:58 |
| avass | oh however I think it just throws that when it times out so you could be right. | 10:03 |
| *** jamesmcarthur has joined #zuul | 10:15 | |
| *** wuchunyang has quit IRC | 10:17 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Example Docker compose: keycloak integration https://review.opendev.org/c/zuul/zuul/+/769943 | 10:21 |
| *** saneax has joined #zuul | 10:30 | |
| tobiash | corvus: do you want to have a look at 765873 (bump openshift dep to solve urllib3 conflict in nodepool)? clarkb and I are authors that's why it has no +3 yet I guess | 10:38 |
| avass | okay so the problem seems to be that gear either isn't able to reconnect or that it never tries, but I only have error logs from gear so I can't see what happens | 10:50 |
| tobiash | is the problem just zuul-web or the executor as well? | 10:51 |
| avass | haven't experienced any executor problems so I don't think so | 10:52 |
| avass | can't see any gear errors in the executor logs | 10:55 |
| *** rpittau has quit IRC | 11:06 | |
| *** rpittau has joined #zuul | 11:07 | |
| *** jcapitao is now known as jcapitao_lunch | 11:22 | |
| avass | also just noticed our executors seem to be leaking processes | 11:36 |
| *** jamesmcarthur has joined #zuul | 11:44 | |
| *** jamesmcarthur has quit IRC | 11:49 | |
| *** zenkuro has joined #zuul | 12:01 | |
| tobiash | zombies? | 12:16 |
| avass | zyup | 12:17 |
| avass | Foufas, Johannes | 12:17 |
| avass | 8:58 AM | 12:17 |
| avass |  | 12:17 |
| avass | well, we will now if all build dll fails... | 12:17 |
| tobiash | we run the executor under tini that reaps zombies | 12:17 |
| avass | oh oops, accidental paste | 12:17 |
| tobiash | need to check if the upstream images do something similar | 12:18 |
| avass | we're overriding that with a "zuul-executor -f -d" maybe not a good idea then :) | 12:19 |
| *** dry has joined #zuul | 12:19 | |
| tobiash | upstream doesn't do that as long python-base doesn't use it as entrypoint | 12:19 |
| tobiash | checking | 12:20 |
| avass | however the executor image seem to be using /usr/local/bin/zuul-executor -f so it should be practically the same | 12:20 |
| *** msuszko has quit IRC | 12:21 | |
| tobiash | it's in python-base: https://opendev.org/opendev/system-config/src/branch/master/docker/python-base/Dockerfile#L33 | 12:21 |
| tobiash | so that should be reaping | 12:21 |
| tobiash | avass: do you see dumb-init in ps | 12:21 |
| avass | no | 12:23 |
| tobiash | do you build your own images? | 12:23 |
| avass | no but like I said, we're setting the command to "zuul-executor -f -d" to get debug logging instead | 12:23 |
| tobiash | command just appends to entrypoint, so if you overwrite cmd and not entrypoint that should be fine | 12:24 |
| avass | oh right | 12:25 |
| *** jpena is now known as jpena|lunch | 12:25 | |
| *** jcapitao_lunch is now known as jcapitao | 12:41 | |
| *** rlandy has joined #zuul | 12:50 | |
| avass | tobiash: kubernetes command actually overrides docker entrypoint, I think I've seen that before and explains my confusion :) | 13:08 |
| *** jamesmcarthur has joined #zuul | 13:08 | |
| tobiash | oh | 13:08 |
| avass | entrypoint -> command, command -> args | 13:08 |
| tobiash | way of least surprise... | 13:08 |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** jamesmcarthur has joined #zuul | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:19 | |
| *** jpena|lunch is now known as jpena | 13:21 | |
| *** rfolco has joined #zuul | 13:41 | |
| avass | tobiash: yeah that fixed it | 13:42 |
| *** jamesmcarthur has joined #zuul | 14:38 | |
| *** ikhan has joined #zuul | 15:00 | |
| mordred | yeah - you want to keep dumb-init in your command line if you're overriding things | 16:00 |
| mordred | oh - nevermind - I was scrolled back a smidge | 16:00 |
| *** zenkuro has quit IRC | 16:12 | |
| *** zenkuro has joined #zuul | 16:13 | |
| *** jfoufas1 has quit IRC | 16:34 | |
| *** jcapitao has quit IRC | 16:39 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:47 | |
| *** openstackgerrit has quit IRC | 16:55 | |
| *** jpena is now known as jpena|off | 17:01 | |
| *** rpittau is now known as rpittau|afk | 17:32 | |
| *** piotrowskim has quit IRC | 17:33 | |
| *** nils has quit IRC | 17:48 | |
| *** hamalq has joined #zuul | 17:58 | |
| *** ykarel has quit IRC | 18:00 | |
| tristanC | corvus: could you please reconsider your -2 on this zuul-runner change https://review.opendev.org/c/zuul/zuul/607078 , if we can merge it, then it would be possible to test the other zuul-runner change with real job from an existing zuul | 18:15 |
| tristanC | (this change, and the other one to support depends-on are the only one affecting the rest api used by the zuul-runner cli) | 18:16 |
| fungi | looks like it's lifted now | 18:27 |
| corvus | tristanC: i've removed the -2 because we merged the spec, so we have consensus there. though i was and still am concerned about merging changes to the api without reviewing the whole stack | 18:27 |
| corvus | tristanC: i agree it might be nice for folks to test it out against a real zuul, but i don't think that's necessary | 18:28 |
| corvus | tristanC: i'm not opposed, just 'concerned' :) | 18:29 |
| corvus | (i don't want to make it harder to fix the api if we find a problem with it several changes down the stack, that's all) | 18:29 |
| corvus | tristanC: maybe the way to address that is to get at least some review of the whole stack before merging that one, even if the whole stack isn't 2x+2 yet | 18:30 |
| *** jamesmcarthur has quit IRC | 18:32 | |
| tristanC | corvus: alright, thank you, i'm working on rebasing the stack again. | 18:32 |
| *** jamesmcarthur has joined #zuul | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:39 | |
| *** ianychoi_ has quit IRC | 18:46 | |
| *** ianw_pto is now known as ianw | 18:58 | |
| *** jamesmcarthur has joined #zuul | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** hashar has quit IRC | 19:30 | |
| *** holser has quit IRC | 19:31 | |
| *** holser has joined #zuul | 19:35 | |
| *** holser has quit IRC | 19:39 | |
| *** holser has joined #zuul | 19:40 | |
| *** zenkuro has quit IRC | 19:42 | |
| *** holser has quit IRC | 19:43 | |
| *** zenkuro has joined #zuul | 19:43 | |
| *** holser has joined #zuul | 19:43 | |
| *** arxcruz has quit IRC | 20:14 | |
| *** zenkuro has quit IRC | 20:15 | |
| *** zenkuro has joined #zuul | 20:15 | |
| *** jamesmcarthur has joined #zuul | 20:36 | |
| *** jamesmcarthur has quit IRC | 20:40 | |
| *** jamesmcarthur has joined #zuul | 20:40 | |
| *** vishalmanchanda has quit IRC | 20:49 | |
| corvus | tristanC, clarkb, tobiash: question on https://review.opendev.org/765873 | 21:11 |
| clarkb | corvus: I'll have to differ to tobiash on that question as the change in question was one of tobiash's edits | 21:13 |
| *** rfolco has quit IRC | 21:25 | |
| *** hashar has joined #zuul | 21:31 | |
| tobiash | responded, it was necessary to make it work. I don't know if it works with v4 though | 21:47 |
| *** openstackgerrit has joined #zuul | 21:48 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Separate out executor concerns from AnsibleJob https://review.opendev.org/c/zuul/zuul/+/648643 | 21:48 |
| corvus | tobiash, tristanC: i'm confused; we should have been using openshift3 for a while in the func test; i would not expect the client library to force a change like that | 21:53 |
| tobiash | the old library might have done some implicit conversion. All I can tell is that it only worked lime this. | 21:56 |
| tobiash | *like | 21:56 |
| corvus | ok, that wfm; i think the only remaining q i have is whether we think anyone is using v4 and if we need to test this with v4? | 21:56 |
| corvus | my guess is that if neither tobiash nor tristanC are using v4, the number of v4 users is probably pretty close to 0 | 21:57 |
| corvus | tristanC: i upgraded my vote on 765873 to +2; i'll let you +W based on your feelings about whether we need to test v4. | 21:59 |
| tristanC | corvus: it seems quite difficult to run func test with v4 without an existing deployment | 22:01 |
| corvus | tristanC: yeah, i was thinking testing 'by hand' | 22:01 |
| corvus | but if you aren't then i don't know of anyone using v4 yet, so it's probably not important | 22:02 |
| tristanC | corvus: we do use v4 internally, but through the simpler openshiftpods driver | 22:02 |
| tristanC | (which doesn't create service account or rolebinding) | 22:03 |
| corvus | tristanC: ok. i leave it at your discretion :) | 22:03 |
| tobiash | I dont't have a v4 testenv yet, but I think if its not working (I'd assume it will work since the want a hich degree of backwards compatibility) fixing it would be easy by adding a fallback | 22:03 |
| corvus | tobiash: i agree, backwards compat on this seems likely | 22:04 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: runner: implement prep-workspace https://review.opendev.org/c/zuul/zuul/+/607082 | 22:11 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: runner: add configuration schema https://review.opendev.org/c/zuul/zuul/+/640672 | 22:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: runner: add prepare-workspace command line interface https://review.opendev.org/c/zuul/zuul/+/644770 | 22:16 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: runner: add execute sub-command https://review.opendev.org/c/zuul/zuul/+/630944 | 22:23 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Adding Get Started page https://review.opendev.org/c/zuul/zuul-website/+/766520 | 22:37 |
| *** arxcruz has joined #zuul | 23:09 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Adjust title on Get Started page https://review.opendev.org/c/zuul/zuul-website/+/770490 | 23:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Add "Get Started" link to header https://review.opendev.org/c/zuul/zuul-website/+/770491 | 23:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Add vexxhost and opendev logos to get-started https://review.opendev.org/c/zuul/zuul-website/+/770492 | 23:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Link to opendev.org on get-started page https://review.opendev.org/c/zuul/zuul-website/+/770493 | 23:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Add Acme Gating to get-started page https://review.opendev.org/c/zuul/zuul-website/+/770494 | 23:12 |
| corvus | jamesmcarthur, fungi: ^ | 23:12 |
| *** hashar has quit IRC | 23:15 | |
| corvus | mnaser: can you ack 770492 in particular? I took the liberty of proposing we add your logo in that change. | 23:16 |
| *** jamesmcarthur has quit IRC | 23:16 | |
| fungi | mnaser: but also 770491 since we're mentioning vexxhost's service offering there | 23:16 |
| fungi | er, i mean 766520 | 23:17 |
| corvus | oh yep, that too :) | 23:17 |
| fungi | i would have approved it already otherwise | 23:18 |
| *** rlandy has quit IRC | 23:19 | |
| corvus | the logos are two circles and a triangle. that amuses me for some reason. | 23:20 |
| corvus | notably missing: squares and octogons ("stop" merging broken code!) | 23:21 |
| fungi | the retentive topologist in me wants to redo one of the circles as a spindle so we have one, two and three segment forms | 23:22 |
| openstackgerrit | Merged zuul/nodepool master: Bump openshift dep https://review.opendev.org/c/zuul/nodepool/+/765873 | 23:25 |
| *** jamesmcarthur has joined #zuul | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:35 | |
| *** jamesmcarthur has joined #zuul | 23:35 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!