Tuesday, 2021-06-01

opendevreviewIan Wienand proposed zuul/zuul master: zuul-stream: drop ARA bindep install  https://review.opendev.org/c/zuul/zuul/+/79386800:24
opendevreviewIan Wienand proposed zuul/zuul master: zuul-stream: drop ARA bindep install  https://review.opendev.org/c/zuul/zuul/+/79386800:27
opendevreviewMerged zuul/zuul-jobs master: Bump golang version  https://review.opendev.org/c/zuul/zuul-jobs/+/79358600:48
opendevreviewMerged zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7  https://review.opendev.org/c/zuul/zuul-jobs/+/78489401:01
opendevreviewMerged zuul/zuul-jobs master: Add ensure-skopeo role  https://review.opendev.org/c/zuul/zuul-jobs/+/79290001:01
*** zbr has quit IRC02:22
*** zbr has joined #zuul02:34
*** zbr is now known as Guest41302:49
*** zbr has joined #zuul02:49
*** zbr is now known as Guest41402:53
*** zbr has joined #zuul02:53
*** Guest413 has quit IRC02:53
*** Guest414 has quit IRC02:57
*** zbr has quit IRC03:01
*** zbr has joined #zuul03:02
*** timburke has joined #zuul03:52
*** ricolin_ has joined #zuul04:32
*** ricolin has quit IRC04:36
*** ricolin_ is now known as ricolin04:36
*** y2kenny has quit IRC04:47
*** marios has joined #zuul05:14
opendevreviewSimon Westphahl proposed zuul/zuul master: Show pipeline event queue lengths in Zuul web  https://review.opendev.org/c/zuul/zuul/+/79378105:28
*** timburke has quit IRC05:47
*** msuszko has joined #zuul06:49
*** msuszko has quit IRC06:53
*** hashar has joined #zuul07:19
*** hashar has joined #zuul07:19
*** jpena|off is now known as jpena07:26
*** rpittau|afk is now known as rpittau07:44
*** tosky has joined #zuul07:48
*** tosky has quit IRC07:48
*** msuszko[m] has joined #zuul07:56
*** msuszko[m] is now known as msuszko07:56
*** tosky has joined #zuul08:08
opendevreviewSimon Westphahl proposed zuul/zuul master: Show pipeline event queue lengths in Zuul web  https://review.opendev.org/c/zuul/zuul/+/79378108:28
opendevreviewAlbin Vass proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390108:32
opendevreviewSimon Westphahl proposed zuul/zuul master: Show pipeline event queue lengths in Zuul web  https://review.opendev.org/c/zuul/zuul/+/79378108:33
opendevreviewMatthieu Huin proposed zuul/zuul-client master: Add support for XDG-compliant config file  https://review.opendev.org/c/zuul/zuul-client/+/79379608:34
swestmhu: updated 793781 according to your comments08:34
mhuswest, nice, I'll check it when the CI completes08:36
swestmhu: k, thanks!08:36
avass[m]apparently gitlabs protected_branches api doesn't return all branches that are protected. instead it returns rules over which branches are protected08:36
avass[m]the documentation doesn't really mention it: https://docs.gitlab.com/ee/api/protected_branches.html#list-protected-branches :/08:37
mhuswest, that makes me think, you may also want to move the global queue length info at the top of the status page, set it by the zuul version and reconfiguration time info08:46
swestmhu: not sure if that's a good change from our users' perspective as a lot of them are using that info as an indicator e.g. that there is a reconfig going on08:49
mhuah, makes sense08:50
mhuswest, I think that's better now, thanks!09:05
opendevreviewMatthieu Huin proposed zuul/zuul-client master: Add search filters when relevant  https://review.opendev.org/c/zuul/zuul-client/+/78884709:15
*** sshnaidm|off is now known as sshnaidm09:57
*** sshnaidm has joined #zuul09:58
*** jangutter has quit IRC10:08
*** jpena is now known as jpena|lunch11:32
*** sshnaidm has quit IRC11:51
*** sshnaidm has joined #zuul11:59
opendevreviewLida Liu proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390112:14
*** jpena|lunch is now known as jpena12:29
opendevreviewAlbin Vass proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390112:31
opendevreviewAlbin Vass proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390112:32
opendevreviewSimon Westphahl proposed zuul/zuul master: Remove layout attribute from queue items  https://review.opendev.org/c/zuul/zuul/+/79262212:54
opendevreviewSimon Westphahl proposed zuul/zuul master: Optimize layout re-calculation after re-enqueue  https://review.opendev.org/c/zuul/zuul/+/79262312:54
mhuhello zuul-maint, please find below a suggestion for the next patches to merge on zuul-client before releases 0.0.5 and 0.0.6 https://etherpad.opendev.org/p/zuulclient-mini-roadmap12:55
mhuIt's been about 15 patches and 4 months since the last release + the builds subcommand is already merged, we should have buildsets, build-info and buildset-info in there as well12:56
*** rpittau is now known as rpittau|afk12:57
opendevreviewAlbin Vass proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390113:00
*** dani has joined #zuul13:13
*** dani has quit IRC13:15
corvusswest: can you take a look at my comments on https://review.opendev.org/792622 ? i think that might be ready with one more tiny revision13:28
corvuszuul-maint: https://review.opendev.org/793700 is a small change that would be great to merge asap i think; it should fix a test race13:29
*** mordred has quit IRC13:35
*** mordred has joined #zuul13:35
opendevreviewLida Liu proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390113:36
*** eliadcohen_ is now known as eliadcohen13:53
opendevreviewTristan Cacqueray proposed zuul/zuul master: gerrit: delay event queue attempts  https://review.opendev.org/c/zuul/zuul/+/79397714:26
opendevreviewTristan Cacqueray proposed zuul/zuul master: gerrit: delay event queue attempts  https://review.opendev.org/c/zuul/zuul/+/79397714:30
tristanCzuul-maint: when restarting zookeeper, the schedulers logs are flooded with `ERROR gerrit.GerritPoller: Exception on Gerrit poll with gerrit` because of `kazoo.exceptions.ConnectionClosedError`. Could you have a look at the proposed fix ^14:33
*** gouthamr_ has joined #zuul14:35
corvustristanC: lgtm; tobias.henkel ^14:42
tobiash[m]Lgtm14:43
*** opendevreview has quit IRC14:44
tobiash[m]However I think technically the message of the first exception is no longer correct14:44
tristanCavass[m]: corvus: tobiash[m]: thank you for the prompt review!14:44
corvustobias.henkel: oh yeah, it's probably worth looking at that as a followup14:45
*** gouthamr has joined #zuul14:46
*** Open10K8S has joined #zuul14:56
*** timburke has joined #zuul14:58
*** Open10K8S has quit IRC15:00
*** Open10K8S has joined #zuul15:01
*** Open10K8S has quit IRC15:01
*** Open10K8S has joined #zuul15:03
*** opendevreview has joined #zuul15:04
opendevreviewMerged zuul/zuul master: Move "create|delete reference ..." messages to own sub logger  https://review.opendev.org/c/zuul/zuul/+/79163815:04
*** Open10K8S has left #zuul15:06
opendevreviewLida Liu proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390115:07
corvusmhu: you taught me a python in 793796;  mordred, do you want to look at that again?  or i could take your "great patch" comment as an implicit +2 on the new rev15:08
corvusmhu: (i did leave a +2 comment on that if you want to take a look)15:08
*** opendevreview has quit IRC15:09
mordredcorvus: +A15:10
mordredalso - I hadn't seen that python either!15:10
mhucorvus, ah right the pathlib.Path syntax ... I was very puzzled the first time I saw it (thanks to tristanC), but it is indeed neat!15:11
corvuslooks like it's Path implementing the / operator15:11
corvusi was like "strings can do that?.... no... oh... it's a path object" :)15:11
mordredyah - I'm fairly certain that's what it is15:11
mordredpathlib came out in one of the 3.x releases - I keep forgetting about it15:11
*** gouthamr_ has quit IRC15:12
corvuswhat would you do to use it for a url path?15:13
*** gouthamr has quit IRC15:13
*** gouthamr has joined #zuul15:14
corvusPurePosixPath maybe?15:14
corvusalso maybe 'urlpath' from pypi15:15
corvuszuul-maint: how does this look for a zuul release?  commit bd1a669cc8e4eb143ecc96b67031574968d51d1e (HEAD -> master, tag: 4.4.0, gerrit/master)15:15
corvusthat's what opendev is running; one commit back from current head15:16
fungiso it's still basically like os.path.join() under the hood i guess?15:16
*** open10k8s has joined #zuul15:16
corvusfungi: i assume; i haven't looked at the impl.15:16
fungijust in the constructor for the object looks like15:16
fungidefinitely terse compared to the old way15:16
avass[m]fungi: yeah15:16
tristanCfungi: corvus: the pathlib also features .name, .parent and read_text()15:17
fungicool, so like the os.path functions just more oo15:17
*** gouthamr is now known as identify15:18
fungi.read_text() looks awesome, that reduces a ton of common boilerplate15:18
*** opendevreview has joined #zuul15:19
opendevreviewMerged zuul/zuul master: Fix test race with executor stats  https://review.opendev.org/c/zuul/zuul/+/79370015:19
fungiand was added in python 3.5, so usable basically anywhere these days15:19
*** gouthamr has joined #zuul15:19
*** identify has quit IRC15:20
opendevreviewMerged zuul/zuul-client master: Clarify ~/.zuul.conf and --use-config option spelling  https://review.opendev.org/c/zuul/zuul-client/+/78900715:23
*** open10k8s has quit IRC15:29
corvuszuul-maint: re-ping regarding that release tag a few lines back (maybe got lost in the pathlib excitement)15:32
opendevreviewMerged zuul/zuul-client master: Add support for XDG-compliant config file  https://review.opendev.org/c/zuul/zuul-client/+/79379615:34
fungioh, yeah checking once my current meeting wraps up15:35
fungilooks like the only newer changes to merge since bd1a669 are 793700 (Fix test race with executor stats) and 791638 (Move "create|delete reference ..." messages to own sub logger) which probably aren't urgent15:38
fungidefinitely some feature changes in the list between 4.3.0 and bd1a669, so 4.4.0 makes sense15:38
tristanCcorvus: the tag lgtm, but we will have to cherry-pick 79397715:39
tristanC(we as in, software factory)15:39
corvustristanC: i was just about to mention that...15:39
corvustristanC: btw, are you using a single node or multi node zk?15:40
tristanCcorvus: this happen with single node zk15:40
corvusi would guess if you used multi-node, you might see zero or one exceptions....15:41
corvustristanC, fungi: i think it would be safe to put the tag on 793977 after it marges; only it and the ref logger changes affect run-time code, and they seem pretty low risk.15:44
opendevreviewLida Liu proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390115:44
fungii'm checking right now to see whether pbr renders bd1a669 to the same dev version as is reported at the bottom of https://zuul.opendev.org/t/zuul/status (noting that the merge commit in the gate which ended up in the image we're running in opendev won't match that commit id), so to see if it probably corresponds to what we're currently running since the restart over the weekend15:45
fungibut building a zuul sdist is not quick, owing to pip install of all its deps i think15:46
fungithere it is. 4.3.1.dev53 yep, so that matches15:46
fungibut also yes i'd be okay with the current master branch state plus 793977, all of those seem unlikely to cause adverse impact15:49
clarkbI'll check as soon as I get this meeting agenda out15:56
clarkblooks like the major changes are the pyyaml fixup for executor side decryption, multiple semaphores, and improved stats/logging16:02
*** marios has quit IRC16:02
clarkbdo we know if multiple sempahores have been used in production by anyone yet? otherwise no concerns from me16:02
avass[m]clarkb: I can check16:02
avass[m]we upgraded yesterday but I'm not sure if we're actually using it yet16:03
corvusclarkb: what would the concern be?16:03
corvusclarkb: oh, wanting to know if they've been production-tested?16:03
clarkbcorvus: just that we might have to do a 4.4.1 release if there is unexpected contention behavior or similar16:04
clarkbya16:04
clarkbI think doing a 4.4.1 is also fine, but if we have feedback now is a good time to incorporate it :)16:04
corvusclarkb: yeah, as far as a release goes, i'd be more concerned with regressions, since after all, if the new feature doesn't work as expected, it's not a big deal to .1 fix16:04
opendevreviewMerged zuul/zuul master: gerrit: delay event queue attempts  https://review.opendev.org/c/zuul/zuul/+/79397716:05
avass[m]clarkb, corvus I found a change using it in review with  +1 Verified from zuul.16:05
fungii'm not super concerned if there's a new bug in a new feature, that's normal. more on the lookout for potential regressions but haven't seen any today16:05
clarkbavass[m]: excellent16:05
fungior what corvus also just said16:05
corvuseven better16:06
corvusokay, 977 merged; should i A) retag that sha assuming we're happy with only review, B) restart opendev with it for a quick sanity check then retag, or C) just push up the existing tag without the new changes?16:07
fungi140 pipeline queue items in opendev's openstack tenant, so busy but not overwhelmingly so that a restart would be problematic16:09
fungibut i'm also not all that worried about those three additional changes, after skimming them16:09
clarkbconsidering the change that merged is "just" a log cleanup change I'd probably be happy with tagging the original proposal16:10
corvusclarkb: thoughts? ^16:10
clarkbI may undervalue the impact of that logging cleanup though16:10
corvusclarkb: https://review.opendev.org/793977 adds a sleep16:10
clarkboh hrm in that case its probably fine to tag it as is16:11
corvusthere could be second order effects, but honestly, we probably wouldn't see them after a simple restart anyway; we'd have to have some kind of zk issue come up.16:11
corvusclarkb: so is that a vote for (A) tag current master?16:12
clarkbcorvus: yes16:12
fungiand that way softwarefactory doesn't need to cherry-pick that last change16:13
clarkbthere are three changes since the opendev restart. One is a test fix, another is a logging update, and the other adds a sleep to improve logging16:13
clarkbthose all seem pretty safe to me16:13
fungithat was my conclusion as well16:13
corvuszuul-maint: updated tag proposal: commit 7b623a99ba96d3dbe10e29e1c37fbdaae6d2c863 (HEAD -> master, tag: 4.4.0, origin/master)16:13
clarkbcorvus: ++16:14
*** vexxhost_ has joined #zuul16:14
fungiyep, lgtm16:15
fungithat's my current master branch tip16:15
fungiand git log shows the expected three changes now after the previously proposed commit16:16
*** vexxhost_ has quit IRC16:16
corvusokay, i'll go ahead and push it, then send out an announcement later today16:17
fungithanks!16:19
opendevreviewSimon Westphahl proposed zuul/zuul master: Remove layout attribute from queue items  https://review.opendev.org/c/zuul/zuul/+/79262216:21
opendevreviewSimon Westphahl proposed zuul/zuul master: Optimize layout re-calculation after re-enqueue  https://review.opendev.org/c/zuul/zuul/+/79262316:21
swestcorvus: updated ^16:22
corvusswest: awesome!  thanks for working through that with me, i think the additional comments are a good foundation for understanding; makes a lot of sense now.  :)16:24
swestcorvus: thanks for your patience :D sometimes when you have all the details in your head it's hard to see the non-obvious things.16:26
corvusswest: yep!  i think one of the best benefits of code review is improving shared understanding among contributors16:28
corvusswest: oh, and that change makes the next one so simple :)16:28
corvusclarkb: if you still have some of the layout cache optimization work in your head (you may recall that we reverted that); i think the unrevert at https://review.opendev.org/792622 and https://review.opendev.org/792623 is ready16:30
corvusbasically, the first change makes it structurally harder to have the bug that necessitated the revert (by removing references to layout objects)16:31
clarkbcorvus: I can give it a try16:34
corvusit's not urgent16:35
clarkbI'm trying to get through some local system updates (its like 3k pacakges * 3 machines) and some cleanup from yesterday's garage fun then I can do reviews and provision that cert and prep for the infra meeting16:40
clarkband I think I have a meeting in there somewhere too16:40
opendevreviewLida Liu proposed zuul/zuul master: Gitlab: filter repository branches with protected branch rules  https://review.opendev.org/c/zuul/zuul/+/79390116:46
*** jpena is now known as jpena|off16:47
clarkbswest: corvus question on https://review.opendev.org/c/zuul/zuul/+/79262217:06
clarkbswest: corvus  I think I may have answered my own question(s) looking at the child change. I have updated my comments. Would be good if you can double hceck17:14
corvusclarkb: replied18:27
tristanCis it expected that `zuul tenant-conf-check` requires a zookeeper connection?18:52
tristanCthe command is now failing with `Failed connecting to Zookeeper within the connection retry policy.`18:53
avass[m]tristanC: I think that schedules work to mergers to load all configuration and check for config errors, so yep18:53
avass[m]i could be wrong about that one though18:55
tristanCavass[m]: i thought that was zuul-scheduler --validate-tenants18:57
avass[m]tristanC: right, I was thinking of that one18:57
tristanCwe are having a bunch of integration issues since the 4.2, for example changing the hostname seems to prevent the scheduler from processing trigger event until the previous lock expires.19:00
*** hashar has quit IRC19:01
opendevreviewTristan Cacqueray proposed zuul/zuul master: zuul tenant-conf-check: disable scheduler creation  https://review.opendev.org/c/zuul/zuul/+/79402719:11
corvustristanC: you mean changing the hostname while the scheduler process is running?19:19
tristanCcorvus: and restarting it19:20
tristanCit has to be restarted for sql access19:20
corvustristanC: it sounds like the scheduler may not be shutting down gracefully19:26
corvuswhich, if it hasn't progressed past the sql connection stage, sounds pretty likely19:26
tristanCcorvus: i guess this does not happen by default with systemd? i'm retrying with `ExecStop=/usr/bin/zuul-scheduler stop`19:36
corvustristanC: this behavior change may be relevant too: https://review.opendev.org/79033919:36
corvusit will wait for sql at startup now -- perhaps you don't need to externally restart with that in place?19:37
corvustristanC: but if you do, then i could see that potentially being a problem19:38
corvustristanC: i would expect 'stop' to shut everything down correctly19:38
corvusso exactly what's happening is still a mystery, but there are some places to look19:39
tristanCsupporting hostname change is a bit tricky with lots of corner case, so we'd rather force a full restart in that situation19:42
tristanCfrom merger git email to sql access, it has been safer to just reload19:43
corvusyeah, that's certainly unexpected and hard to defensively code for :)19:44
tristanCi think what is happening is that the gerrit connection leak an election lock with the previous hostname, and when the scheduler restart, the new gerrit connection does not become the leader19:45
corvustristanC: so i think the area to focus on is why isn't the lock (i assume it's an event queue election) being released on shutdown  (and i assume it's not being released and we're waiting on the zookeeper ephemeral node timeout)19:45
corvustristanC: i thought the election recipe only used the hostname for informational purposes (ie, it's actually the fact that the connection created the znode that gives it the lock)19:46
corvustristanC: that's why i'm leaning toward unclean shutdown.19:46
corvustristanC: i could be wrong about that, but that's where i'd start looking19:46
tristanCcorvus: the cmd.scheduler exit_handler does not seem to call scheduler.stop19:47
corvusthat sure sounds like it could be relevant.19:47
opendevreviewTristan Cacqueray proposed zuul/zuul master: scheduler: call stop on SIGTERM  https://review.opendev.org/c/zuul/zuul/+/79403519:49
tristanCthe hostname change seems unrelated indeed19:51
tristanCcorvus: thank you for the help19:52
corvustristanC: np, thanks for finding that :)19:55
tristanCin another attempt i'm running `zkCli.sh deleteall /zuul/events` before restarting the scheduler, and that seems to help19:55
corvustristanC: good idea (temporarily) -- the election nodes are under /zuul/events/connnection/$name/election i think19:57
*** Ganneff has left #zuul19:57
tristanCcalling zuul-scheduler stop is not enough, i'm now retrying with 79403520:40
*** hashar has joined #zuul20:46
*** hashar has quit IRC20:47
clarkbcorvus: I think I'm getting confused by lines 851-852 of https://review.opendev.org/c/zuul/zuul/+/792622/5/zuul/manager/__init__.py Why does that logging need to be in an if block? Are we only interested in logging that we are calculating a layout if we have a cache miss that isn't an initial population of the cache?21:35
clarkbcorvus: I agree about your comment on defense against the None key21:36
corvusclarkb: re comment -- yeah that's my understanding.  the way i'm reading it is basically "this logs a cache miss that results in a recalculation".  the calculation itself will have a whole bunch of logging anyway, so i think this is just an extra "here's why we're about to do all this stuf..."21:38
corvusclarkb: and i think it's interesting because i don't think it should happen until we have another scheduler21:39
corvusso it's a potentially interesting bit of debug info21:39
clarkbgot it, that helps21:39
clarkbas for whether or not we should be defensive there I'm not sure. We definitely expect None as a valid uuid value in some places, and if that leaked into the cache we coudl have shared layouts ?21:40
corvusclarkb: yeah, that's my understanding.  obvs it shouldn't happen, but if it did, only something like your suggestion of an explicit test would catch it21:40
corvus(i mean a layout object should never have None for a uuid; an item can have None as it's layout_uuid, indicating it's not yet set)21:42
clarkbright and we only update the cache using the layout object's uuid (by way of the newly updated item object)21:42
corvusthe layout class accepts a layout param in the initializer, and None means 'generate a new one'; that's in like one line of python, so unlikely to get wrong, but concievable it could happen in a later refactor or something21:43
clarkbmaybe the cache should be set using only the layout object and that is good enough21:43
corvusclarkb: yep (re updating the cache)21:43
corvusi think line 857 is the only place the cache is written to21:43
corvusyeah (other than the delete); so that's the only insertion21:44
clarkbya so this is probably ok as is, unless we change the management around uuids21:48
clarkband that seems unlikely since we are using rnaodm uuids21:48
corvus++21:49
corvusclarkb: cool, i'll let that sit overnight in case any of that discussion sparks a concern with swest; if not i'll +w tomorrow21:55
clarkbsounds good. I'm looking at the second change now21:56
clarkbcorvus: question on https://review.opendev.org/c/zuul/zuul/+/79262322:13
corvusclarkb: yeah, i think that one has had me scratching my head before too; but i think it's all actually correct; replied22:20
clarkbcorvus: aha that makes sense22:22
corvusclarkb: thanks for reviewing; i'm looking forward to speeding up reconfiguration times :)22:26
corvusmaybe this will make it so it's less important to have those queue length items on the status page ;)22:27
clarkbthat would be cool22:27
corvusavass: if you have time for a quick re-review of https://review.opendev.org/793829 i'd appreciate it22:34
*** tosky has quit IRC23:11
*** eliadcohen_ has joined #zuul23:47
*** eliadcohen has quit IRC23:51

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!