Friday, 2021-02-12

*** jamesmcarthur has joined #zuul00:03
*** jamesmcarthur has quit IRC00:08
openstackgerritMerged zuul/zuul-storage-proxy master: Use opendev base docker image and add jobs  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77499800:08
corvushttps://hub.docker.com/r/zuul/zuul-storage-proxy/tags exists00:20
corvusi'll approve the rest00:20
openstackgerritMerged zuul/zuul-storage-proxy master: Make container name prefix more generic  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77524300:32
openstackgerritMerged zuul/zuul-storage-proxy master: Log at info level  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77524700:32
openstackgerritMerged zuul/zuul-storage-proxy master: Just use CLOUD_NAMES as env variable  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77524800:32
openstackgerritMerged zuul/zuul-storage-proxy master: Switch to uwsgi  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77529400:35
*** tosky has quit IRC00:36
*** jamesmcarthur has joined #zuul01:05
*** jamesmcarthur has quit IRC01:37
*** jamesmcarthur has joined #zuul01:39
*** ikhan has joined #zuul02:45
*** ikhan has quit IRC02:49
*** jamesmcarthur has quit IRC03:37
*** jamesmcarthur has joined #zuul03:45
*** jamesmcarthur has quit IRC03:49
*** jamesmcarthur has joined #zuul03:50
*** jamesmcarthur has quit IRC04:18
*** jamesmcarthur has joined #zuul04:22
*** jamesmcarthur has quit IRC04:27
*** jamesmcarthur has joined #zuul04:28
*** jamesmcarthur has quit IRC04:33
*** ykarel_ has joined #zuul04:51
*** jamesmcarthur has joined #zuul04:55
*** jamesmcarthur has quit IRC04:59
*** jamesmcarthur has joined #zuul05:07
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** jamesmcarthur has quit IRC05:34
*** jamesmcarthur has joined #zuul05:36
*** jamesmcarthur has quit IRC05:38
*** jamesmcarthur has joined #zuul05:42
*** jamesmcarthur has quit IRC05:45
*** jfoufas1 has joined #zuul05:50
*** jamesmcarthur has joined #zuul05:52
*** ykarel_ is now known as ykarel05:59
*** jamesmcarthur has quit IRC05:59
*** jamesmcarthur has joined #zuul06:01
*** jamesmcarthur has quit IRC06:13
openstackgerritRon Izraeli proposed zuul/zuul master: Linkify BuildOutput  https://review.opendev.org/c/zuul/zuul/+/77510907:13
*** ykarel_ has joined #zuul07:47
iceyam I correct that I can backup the /var/lib/zuul/keys/secrets/project/ directory on a zuul-scheduler to backup the secrets used for encrypting project secrets?07:48
*** ykarel has quit IRC07:50
iceylooks like it, found https://docs.opendev.org/opendev/system-config/latest/zuul.html#secrets :)07:52
*** hashar has joined #zuul07:54
*** rpittau|afk is now known as rpittau08:02
*** jamesmcarthur has joined #zuul08:14
*** jamesmcarthur has quit IRC08:18
*** tosky has joined #zuul08:24
*** harrymichal has joined #zuul08:32
fungithat path will depend on how you've deployed things, so at least double-check there's something in there08:34
openstackgerritTobias Henkel proposed zuul/zuul master: Gracefully handle non-existent label on unlabel  https://review.opendev.org/c/zuul/zuul/+/77532908:41
avassoh a merge conflict in a .gitmodules file causes problems with executors because it can't run 'git fetch', I guess the executors doesn't reset the repos before working on the next job?08:42
avassI geuss they can't easily do that because they're in a detached head08:46
*** saneax has joined #zuul08:47
*** jpena|off is now known as jpena08:56
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previouos head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533409:17
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previouos head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533409:19
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previous head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533409:21
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previous head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533409:30
*** nils has joined #zuul09:44
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previous head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533410:08
avass775334 should fix errors from committing broken .gitmodules file10:10
*** andy-ladjadj has joined #zuul10:13
andy-ladjadjHello Zuul community, i have a review with +2 but in merge conflict for the second time, what is the process to resolve the conflict and start the gate in the "same time" (https://review.opendev.org/c/zuul/zuul/+/755929)? Thx10:20
avassandy-ladjadj: you can't10:23
*** ykarel_ is now known as ykarel10:37
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previous head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533410:52
andy-ladjadjAvass: I wanted to say humain process to avoid to reproduce "merge conflict" before workflow +110:55
*** jamesmcarthur has joined #zuul11:20
*** jamesmcarthur has quit IRC11:20
*** sshnaidm|afk has quit IRC11:41
*** sshnaidm|afk has joined #zuul11:49
*** sshnaidm|afk is now known as sshnaidm|off11:50
*** andy-ladjadj has quit IRC11:51
*** hashar has quit IRC12:03
*** andy-ladjadj has joined #zuul12:22
*** andy-ladjadj has quit IRC12:24
openstackgerritAlbin Vass proposed zuul/nodepool master: WIP: Add shell-type config  https://review.opendev.org/c/zuul/nodepool/+/77537112:29
avasstobiash: before I continue on this, you don't happen to already have a patch available for that ^ ? :)12:30
*** jpena is now known as jpena|lunch12:31
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: update-json-file: avoid failure when destination does not exists  https://review.opendev.org/c/zuul/zuul-jobs/+/77537312:33
avasszbr: does that need to run as sudo?12:36
zbravass: valid question, or should it be   become: '{{ update_json_file_become }}' ?12:38
*** rlandy has joined #zuul12:38
zbrif I put owner/group for the folder too, will it mess /etc too or will apply them only when needing to create folder?12:39
avassoh, I don't know12:40
*** hashar has joined #zuul12:42
* zbr goes back to testing12:42
openstackgerritAlbin Vass proposed zuul/zuul master: Reset repo to previous head on .gitmodules error  https://review.opendev.org/c/zuul/zuul/+/77533412:48
*** andy-ladjadj has joined #zuul12:49
*** andy-ladjadj has quit IRC12:51
*** jamesmcarthur has joined #zuul13:21
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: update-json-file: avoid failure when destination does not exists  https://review.opendev.org/c/zuul/zuul-jobs/+/77537313:22
*** jamesmcarthur has quit IRC13:25
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: update-json-file: avoid failure when destination does not exists  https://review.opendev.org/c/zuul/zuul-jobs/+/77537313:26
*** jpena|lunch is now known as jpena13:30
openstackgerritAlbin Vass proposed zuul/zuul master: Use shell-type config from nodepool  https://review.opendev.org/c/zuul/zuul/+/77538213:32
avasszbr: I guess that stat task is there so ansible doesn't change change owner/mode if it already exists?13:45
zbravass: exactly13:46
*** rlandy has quit IRC14:07
*** rlandy has joined #zuul14:07
*** rpittau is now known as rpittau|afk15:03
openstackgerritJames E. Blair proposed zuul/zuul master: Prepare Zookeeper for scale-out scheduler  https://review.opendev.org/c/zuul/zuul/+/71726915:14
corvusswest: ^ just updated the commit msg to remove the part about the test time15:15
swest corvus: oh, ok. I think felixedel is still working on it15:15
corvusswest, felixedel: oh?  i thought everything was addressed in that one?15:16
swesthe wanted to cleanup the connection creation AFAIK15:16
corvushrm; and here i am wanting to merge it...15:17
felixedelcorvus: Yes, I wanted to clean up the connection creation and this seems to be scattered across multiple changes15:17
corvuswhat needs cleaning up?15:17
felixedelIf you are fine with that change, we could also merge it and I move my cleanup parts into https://review.opendev.org/c/zuul/zuul/+/754360/2615:19
felixedelWhile I was doing that I discovered that I could also directly channge it in the initial change of the stack, but I'm also fine to do it in the "Merge ZooKeeper connection methods" change15:20
corvusfelixedel: i think i would like to do the minimum required in order to "require zk connections" so we can get 4.0 released :)15:21
corvusfelixedel: so let's merge it as-is and refactor later?15:23
felixedelok, so I will implement my cleanups in https://review.opendev.org/c/zuul/zuul/+/754360/26 and we could go ahead and merge https://review.opendev.org/c/zuul/zuul/+/717269/61 and https://review.opendev.org/c/zuul/zuul/+/721254/5515:24
felixedel(the first two patches of the stack)15:24
felixedelcorvus: Would that be ok for you?15:28
corvuswell, i'm looking at 360 now since we're all here....15:29
felixedelThe cleanup is mainly about merging the ZooKeeperClient and the ZooKeeperConnection (which was introduced in that change), so we don't have to use one or the other in certain conditions, but only have one kind of "entry point" to ZooKeeper.15:33
felixedelAs I did this cleanup originally further up in the stack and rebase dit down this would now also affect other patches (like the connect xyz to ZooKeeper) which it didn't before.15:34
corvusfelixedel: i left a comment on 36015:34
corvusfelixedel: sorry i misspoke on that comment; i should have said nothing uses it as a connection manager15:38
felixedelcorvus: I've replied15:40
*** ykarel has quit IRC15:42
*** ykarel has joined #zuul15:42
corvusfelixedel: okay, i'm having trouble understanding what you want the end product to look like; it feels like the same thing is being refactored in three sequential changes.  is it possible for us to just look at 721254, 716221, 716575, 716262?  and defer the refactoring until later?15:46
corvusfelixedel: either that, or i could wait a day or 2 for you to straighten out the refactor idea?15:54
corvusbut based on what i see, i don't think we should merge anything until we have a plan to get to 71626215:54
*** harrymichal has quit IRC15:57
felixedelWe could also do the refactoring later, I just thought we don't need to introduce e.g. the ZooKeeperConnection and throw it away X patches afterwards.15:58
felixedelMaybe we can discuss this on Monday.15:59
corvusfelixedel: i agree, that's why i was suggesting we drop 360 from the stack too (basically, i'm looking at dropping 360 and 442)15:59
corvusfelixedel: sure.  i'll mark 717269 as WIP.16:01
felixedelHmm, I will check on Monday if the refactoring might not even be necessary when we don't introduce the connection in the first place. Like I said, originally I did that later on where it was necessary16:02
corvusokay.  i may look at that a little bit today.  i'll let you know what i find.16:02
felixedelok16:02
corvusfelixedel: have a good weekend :)16:02
avasscorvus: if you got time later could you take a look at: https://review.opendev.org/c/zuul/zuul/+/775334 ? It's pretty straight forward.16:07
*** EmilienM has quit IRC16:07
avassBut I guess there could be a better way to get out of that situation than a 'git reset --hard HEAD^'16:07
*** EmilienM has joined #zuul16:16
zbrcorvus: any chance to help with https://review.opendev.org/c/zuul/zuul-jobs/+/775373 ?16:18
*** rlandy is now known as rlandy|mtg16:22
*** ykarel is now known as ykarel|away16:54
*** jamesmcarthur has joined #zuul16:58
*** ykarel|away has quit IRC17:06
*** rlandy|mtg is now known as rlandy17:12
*** jpena is now known as jpena|brb17:13
*** hashar has quit IRC17:18
corvusavass: if someone commits 2 broken .gitmodules in a row, would that fix fail (by essentially doing what we're doing today?)17:26
avasscorvus: that would most probably happen yep17:27
avassI wish there was a 'git --ignore-submodules <command>' since zuul doesn't care about them anyway really17:28
corvusavass: hrm.  let me think about this a bit more.17:28
avasswould there be a problem if the executor just deletes the .gitmodules and commits that so it then can continue working as normal?17:31
avassI guess that would be gc'd eventually anyway17:31
corvusavass: another option might be to just delete the repo17:31
avassyeah but that's a bit too much when the repo is a couple of GB17:32
corvusavass: this doesn't happen often, right? :)17:32
avassno :)17:32
avassit happened when we moved a gerrit instance to azure and had to update the remote url17:32
corvusavass: i think maybe we can just delete the .gitmodules file and not commit it17:33
avasssomeone accidentally commited a .gitmodule with a merge conflict and that broke that repo on executor completely17:33
corvusi think git uses the copy in the working tree?17:33
avassI think I tried that and still got some error. but I was trying to get it working again so I didn't spend much time investigating why17:34
avasswait I might have saved a copy of the repo, let me check17:35
avassoh, it throws an error but actually does a fetch17:36
corvuswhat's the error?  i don't see one in my local testing17:37
avass"error: bad config line 9 in blob :.gitmodules"17:37
avassbut errorcode is 0 so I guess that should work?17:37
corvushrm.  i dunno;  i don't like that error message :/17:38
corvusavass: was the problematic commit actually merged into master (or a development branch)?  like -- it wasn't just a change?17:39
avassit was just a change17:39
avassdeleting the file and committing that works afaik however17:39
*** gmann is now known as gmann_afk17:40
avassand that wouldn't be any problem just in case there isn't a HEAD^ or HEAD^ is also faulty17:40
*** nils has quit IRC17:51
*** jamesmcarthur has quit IRC17:59
*** jamesmcarthur has joined #zuul18:00
corvusavass: i don't like that 'bad config in blob' error because it seems like git is reading something other than the current working tree... i don't know what commits it's looking at18:00
avassthat only appears when the file is only deleted and not committed18:01
corvusright; i can't reproduce that18:01
avassoh weird18:01
corvusin my test, git is only looking at the current working tree; in yours it seems to be looking at blobs.  and i don't know what the difference is.18:02
avassare you doing that locally? the executors are always working in a detached head state so that might have something to do with it18:02
*** jpena|brb is now known as jpena18:03
corvusgood idea; i checkout out a commit sha and still see the same behavior18:03
*** jamesmcarthur_ has joined #zuul18:03
*** jamesmcarthur has quit IRC18:04
corvushrm, i think i just found a code path where we could miss a repo reset18:14
tobiashcorvus: do you happen to know if having many threads waiting for a signal is as bad as having many active threads or do you think that's less of a problem18:17
corvustobiash: you mean waiting for a mutex (not a signal -- only one thread can wait for signals)18:17
corvusright?18:17
corvus(signals like sigint, etc)18:18
tobiashcorvus: I mean the thread events18:18
tobiashthe transition could be easier by some sort of future the thread waits for and there is one thread that does the server list processing and signals the waiting-for-deletion thread18:19
tobiashthe NodeDeleter thread calls manager.waitForNodeCleanup(node.external_id) which currently queries the server list18:20
corvustobiash: i don't remember the particulars right now, but i want to say that some mutex constructs in python can end up doing a bit of busy-waiting so it could be bad.  it's worth looking into either the implementation or running a test program with strace.18:21
tobiashwhat if we change that in the provider to register the thread in the manager and then just sleep until the manager notifies the thread18:21
corvusavass: i think i'd prefer to delete the repo in this case; i know that's not great, but it's such an edge case and i'd like this code to be really robust18:23
corvusavass: maybe if we get a second config error after the reset, delete.  so we keep the special case handling for when the cleanup is easy, and then delete after that.18:23
corvusavass: i'm open to the "commit a fix" method, but i'd like to convince myself we'd never see that commit again, and i'm uncomfortable with the fact that we would leave that method with the fixup commit as HEAD.18:24
avasscorvus: we could also commit a fix, reset and delete the fix-commit18:26
corvusreset to what?18:26
avassI mean, fetch18:26
avassbut then we'd need to keep track of a commit that should be deleted later18:26
corvuscommit fix, fetch, then reset to the known broken commit18:27
*** harrymichal_ has joined #zuul18:27
avassdoesn't the 'reset' method technically checkout a specific commit? might have gotten that wrong18:27
avassprobably got that confused earlier18:32
avasscorvus: we could _technically_ do a check here https://opendev.org/zuul/zuul/src/branch/master/zuul/merger/merger.py#L343 to make sure we reset to a ref the executor can work from.18:34
avassbut I don't like it18:34
corvusavass: honestly, i'm not sure.  there are a lot of code paths in there; it's always been complex, and now there's approximately 2x the options since the executors and mergers now behave differently.  i guess what i'm saying is that deleting the repo is something i'm comfortable with as it's not a significant change.  maybe the fixup commit idea is better (as it would handle this and the existing special18:36
corvuscase), but i think we need to understand the full call stack for all of the different ways it's used.  that's worthwhile work, but it's not straightforward.18:36
*** hamalq has joined #zuul18:36
corvusto be clear, i'd love to page all of that stuff back in and improve it; i'm just trying to balance that with the idea that you wanted a quick fix :)18:37
avasstbh the best fix here would be to get git to just not read the .gitmodules file18:37
avasscorvus: yeah :)18:37
*** jfoufas1 has quit IRC18:38
corvusavass: and i dunno, maybe it's okay to come out of the _git_fetch method on a random commit, as long as it's valid18:39
corvusavass: if so, it's probably pretty straightforward to do the fixup commit; we just need to make sure that's the case18:39
corvus(because it's certainly a behavior change)18:39
avasscorvus: what would happen that if instead of deleting the repo it's cloned locally, deleted and replaced?18:40
corvusavass: it'll clone with HEAD as a bad commit18:40
avassoh we have no heads in those repos either18:44
corvusavass: if you really don't want to delete the repo, then i think pursuing the fixup approach is worthwhile, just make sure that every code path that calls fetch does something that will get the repo to a known state without the fixup commit, since we may be changing an assumption about the side effects of that method.18:45
avassyeah I'm just concerned about someone commiting a broken .gitmodules file in a big repo, trying to recheck that multiple times, breaks the same repo on multiple executors so it gets stuck cloning that repo18:47
avasswHowever we've not seen this happen until now so it might just be good enough18:50
corvusthat's a fair concern.  it may be worth the time to do the fixup approach.18:52
*** gmann_afk is now known as gmann18:52
*** jpena is now known as jpena|off18:58
*** harrymichal_ has quit IRC19:03
corvustobiash: it looks like a threading.Event should be mostly outside of the GIL while waiting -- https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L9719:04
tobiashso should be ok?19:04
tobiashI'll throw it into my testenv and see how it performs19:05
tobiashif that works, waiting for active instances could be also optimized relatively easy19:05
corvustobiash: i think so; it's probably still worth a quick test program to see if there's any load with 1k threads waiting19:05
tobiash++19:05
corvusbut to me, that looks like it's as good as it gets19:06
openstackgerritTobias Henkel proposed zuul/nodepool master: WIP: Optimize node deletion  https://review.opendev.org/c/zuul/nodepool/+/77543819:13
tobiashthat at least works with a local unit test ^19:13
tobiashtrying now in test env19:13
tobiashif that improves things it probably could be adapted to handle most of wait-for-server as well19:14
tobiashat least the part of waiting for the instance to be active19:14
*** jamesmcarthur_ has quit IRC19:24
*** jamesmcarthur has joined #zuul19:25
*** jamesmcarthur has quit IRC19:30
tobiashcorvus: confirmed under mac: 1000 threads waiting for an event cause 0 load19:41
tobiashlinux as well19:45
*** jamesmcarthur has joined #zuul19:51
*** jamesmcarthur has quit IRC19:58
*** jamesmcarthur has joined #zuul20:01
*** jamesmcarthur has quit IRC20:01
*** jamesmcarthur has joined #zuul20:02
*** jamesmcarthur has quit IRC20:03
*** hashar has joined #zuul20:19
*** jamesmcarthur has joined #zuul20:22
*** jamesmcarthur has quit IRC20:24
*** jamesmcarthur has joined #zuul20:24
*** harrymichal_ has joined #zuul20:25
mnasercorvus, tobiash: thoughts on https://review.opendev.org/c/zuul/zuul-jobs/+/774650 ?20:37
corvusmnaser: yep, it's on my list for this afternoon now that we've got the server side nailed down20:37
mnasercorvus: awesome, we're pretty excited about driving that and getting that landed so we can have fully multi-tenanted environments for the vexxhost ci :)20:38
*** jamesmcarthur has quit IRC20:43
*** jamesmcarthur has joined #zuul20:44
*** hashar has quit IRC20:46
*** jamesmcarthur has quit IRC20:49
tobiashmnaser, corvus: I've posted a comment and question on that20:59
*** jamesmcarthur has joined #zuul21:14
corvusmnaser: not sure if you saw, but with tobiash's help, mordred, fungi, and i wrote and merged about 5 changes yesterday on the server side to make that hopefully production ready for the general case; so one of the things on my checklist is to make sure the job side matches any changes there.21:18
*** jamesmcarthur has quit IRC21:18
*** jamesmcarthur has joined #zuul21:18
*** andy-ladjadj has joined #zuul21:29
*** jamesmcarthur has quit IRC21:36
*** andy-ladjadj has quit IRC21:36
*** jamesmcarthur has joined #zuul21:38
*** jamesmcarthur has quit IRC21:43
*** jamesmcarthur has joined #zuul21:44
openstackgerritAdam Richter proposed zuul/zuul master: Linkify BuildOutput in the Task Summary view  https://review.opendev.org/c/zuul/zuul/+/77510921:47
*** rlandy has quit IRC22:05
*** sduthil has quit IRC22:07
*** jamesmcarthur has quit IRC22:12
*** jamesmcarthur has joined #zuul22:17
*** jamesmcarthur has quit IRC22:33
fungizuul-maint: (and anyone else interested) 775109 is a simple change to make any urls which appear in the build summary content clickable, i was hesitant to approve without a bit of consensus on whether others think doing that is a good idea, but it does seem to be working at least23:17
fungialso i'm not super comfortable with my level of js/jsx knowledge23:19
fungiand i expect if it is something we want, we'll also want to be consistent with it across other tabs like the log viewer and console23:22
fungibut those could presumably be implemented in separate changes23:22
corvusfungi: just like the ansi thing it needs to be performance tested on large data23:41
fungicorvus: this is an excellent point. i suppose with the current implementation only affecting the summary view, getting large data in there may be a challenge?23:44
corvusfungi: yes (though only because we truncate to a small number of lines by default);  but if we want to be consistent in other views it would be an issue23:45
fungiyep, i wonder if there's a good way to test that short of implementing it for the other tabs23:46
tristanCit seems like react comes with a profiler, it should be possible to measure the render performance in a test23:49
corvuswell, i'd start by just adding it to the log veiewer, and loading up a 10mb file.  if it doesn't bomb, then do something more sophisticated.  i expect it to bomb.23:51
corvusso i would not put a lot of effort into it before doing that smoke test.23:51
*** hamalq has quit IRC23:51
corvusbut i dunno -- we have the severity regexes in there, maybe it'd be ok23:52
openstackgerritJames E. Blair proposed zuul/zuul master: Mandatory Zookeeper connection for ZuulWeb in tests  https://review.opendev.org/c/zuul/zuul/+/77545923:57
openstackgerritJames E. Blair proposed zuul/zuul master: Connect executor to Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77546023:57
openstackgerritJames E. Blair proposed zuul/zuul master: Connect merger to Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77546123:57
openstackgerritJames E. Blair proposed zuul/zuul master: Update component doc re ZooKeeper  https://review.opendev.org/c/zuul/zuul/+/77546223:57
corvusthat's an alternative stack where i *think* i've sorted out all the conflicts/issues to just get that merged; need to see the test results from each of those23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!