Thursday, 2021-01-21

*** jamesmcarthur has joined #zuul00:11
*** harrymichal has quit IRC00:13
*** harrymichal has joined #zuul00:14
*** jamesmcarthur has quit IRC00:40
*** harrymichal has quit IRC01:26
*** hamalq has quit IRC01:40
*** maxamillion_ has joined #zuul01:52
*** maxamillion_ has quit IRC02:01
*** bhavikdbavishi has joined #zuul03:33
*** bhavikdbavishi1 has joined #zuul03:36
*** bhavikdbavishi has quit IRC03:38
*** bhavikdbavishi1 is now known as bhavikdbavishi03:38
*** zenkuro has quit IRC04:01
*** ajitha has joined #zuul04:24
*** ykarel has joined #zuul04:52
*** lyr has quit IRC04:52
*** tflink has quit IRC04:53
*** lyr has joined #zuul04:53
*** tflink_ has joined #zuul04:53
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** bhavikdbavishi has quit IRC05:47
*** bhavikdbavishi has joined #zuul05:49
*** vishalmanchanda has joined #zuul05:58
*** ykarel_ has joined #zuul06:16
*** ykarel has quit IRC06:19
*** bhavikdbavishi has quit IRC06:30
*** bhavikdbavishi has joined #zuul06:32
*** saneax has joined #zuul06:37
*** reiterative has quit IRC06:56
*** reiterative has joined #zuul06:56
*** harrymichal has joined #zuul06:58
*** bhavikdbavishi1 has joined #zuul07:27
*** ykarel_ is now known as ykarel07:27
*** bhavikdbavishi has quit IRC07:29
*** bhavikdbavishi1 is now known as bhavikdbavishi07:29
*** jfoufas1 has joined #zuul07:38
*** jcapitao has joined #zuul07:49
*** jpena|off is now known as jpena07:52
*** bhavikdbavishi has quit IRC08:02
*** rpittau|afk is now known as rpittau08:11
*** tosky has joined #zuul08:16
*** hashar has joined #zuul08:23
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Rename config repos file config for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77081508:33
*** bhavikdbavishi has joined #zuul08:45
*** bhavikdbavishi1 has joined #zuul08:50
*** bhavikdbavishi has quit IRC08:51
*** bhavikdbavishi1 is now known as bhavikdbavishi08:51
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Rename config repos file config for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77081509:00
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenants in unparsed abide as dict  https://review.opendev.org/c/zuul/zuul/+/77145609:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Refactor config/tenant (re-)loading  https://review.opendev.org/c/zuul/zuul/+/77145709:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Tenant read/write lock in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77145809:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock pipelines during processing  https://review.opendev.org/c/zuul/zuul/+/77145909:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock global event queues during processing  https://review.opendev.org/c/zuul/zuul/+/77146009:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenant layout state in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77146109:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Configure unique command socket path per scheduler  https://review.opendev.org/c/zuul/zuul/+/77146209:09
openstackgerritSimon Westphahl proposed zuul/zuul master: Support cross scheduler config loading  https://review.opendev.org/c/zuul/zuul/+/77146309:09
*** harrymichal has quit IRC09:29
*** nils has joined #zuul09:31
*** ykarel_ has joined #zuul09:34
*** bhavikdbavishi1 has joined #zuul09:36
*** bhavikdbavishi has quit IRC09:36
*** tosky has quit IRC09:36
*** bhavikdbavishi1 is now known as bhavikdbavishi09:36
*** tosky_ has joined #zuul09:36
*** ykarel has quit IRC09:37
*** ykarel_ is now known as ykarel09:39
*** tosky_ is now known as tosky09:43
*** harrymichal has joined #zuul10:04
*** harrymichal has quit IRC10:26
*** hashar is now known as hasharAway10:53
*** harrymichal has joined #zuul11:11
*** jcapitao is now known as jcapitao_lunch11:26
*** jpena is now known as jpena|lunch12:30
*** bhavikdbavishi has quit IRC12:36
*** jcapitao_lunch is now known as jcapitao12:39
*** rlandy has joined #zuul12:46
*** tosky has quit IRC13:06
*** tosky has joined #zuul13:06
openstackgerritSimon Westphahl proposed zuul/zuul master: Switch to Zookeeper backed trigger event queues  https://review.opendev.org/c/zuul/zuul/+/76117213:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Switch to Zookeeper backed management event queues  https://review.opendev.org/c/zuul/zuul/+/76173813:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Use logical timestamp to detect outdated changes  https://review.opendev.org/c/zuul/zuul/+/76375513:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Make buildset mandatory on build  https://review.opendev.org/c/zuul/zuul/+/77090013:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Implement ZooKeeper builds API  https://review.opendev.org/c/zuul/zuul/+/77090113:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Switch to ZooKeeper backed job execution and result events  https://review.opendev.org/c/zuul/zuul/+/77090213:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Merge ZooKeeper connection and client classes  https://review.opendev.org/c/zuul/zuul/+/77144213:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Refactor pipeline processing in run handler  https://review.opendev.org/c/zuul/zuul/+/77145213:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Dequeue superceded items via management event  https://review.opendev.org/c/zuul/zuul/+/77145313:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Text stream API for sharded Zookeeper data  https://review.opendev.org/c/zuul/zuul/+/77145413:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Cache unparsed config files in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77145513:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenants in unparsed abide as dict  https://review.opendev.org/c/zuul/zuul/+/77145613:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Refactor config/tenant (re-)loading  https://review.opendev.org/c/zuul/zuul/+/77145713:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Tenant read/write lock in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77145813:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock pipelines during processing  https://review.opendev.org/c/zuul/zuul/+/77145913:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock global event queues during processing  https://review.opendev.org/c/zuul/zuul/+/77146013:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenant layout state in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77146113:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Configure unique command socket path per scheduler  https://review.opendev.org/c/zuul/zuul/+/77146213:20
openstackgerritSimon Westphahl proposed zuul/zuul master: Support cross scheduler config loading  https://review.opendev.org/c/zuul/zuul/+/77146313:20
*** jpena|lunch is now known as jpena13:29
openstackgerritSimon Westphahl proposed zuul/zuul master: Switch to ZooKeeper backed job execution and result events  https://review.opendev.org/c/zuul/zuul/+/77090213:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Merge ZooKeeper connection and client classes  https://review.opendev.org/c/zuul/zuul/+/77144213:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Refactor pipeline processing in run handler  https://review.opendev.org/c/zuul/zuul/+/77145213:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Dequeue superceded items via management event  https://review.opendev.org/c/zuul/zuul/+/77145313:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Text stream API for sharded Zookeeper data  https://review.opendev.org/c/zuul/zuul/+/77145413:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Cache unparsed config files in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77145513:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenants in unparsed abide as dict  https://review.opendev.org/c/zuul/zuul/+/77145613:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Refactor config/tenant (re-)loading  https://review.opendev.org/c/zuul/zuul/+/77145713:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Tenant read/write lock in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77145813:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock pipelines during processing  https://review.opendev.org/c/zuul/zuul/+/77145913:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Lock global event queues during processing  https://review.opendev.org/c/zuul/zuul/+/77146013:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Store tenant layout state in Zookeeper  https://review.opendev.org/c/zuul/zuul/+/77146113:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Configure unique command socket path per scheduler  https://review.opendev.org/c/zuul/zuul/+/77146213:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Support cross scheduler config loading  https://review.opendev.org/c/zuul/zuul/+/77146313:45
*** zenkuro has joined #zuul14:03
*** ykarel is now known as ykarel|mtg14:10
*** systemd is now known as systemc14:10
*** vishalmanchanda has quit IRC14:38
*** sshnaidm|ruck is now known as sshnaidm|afk14:49
*** ykarel|mtg is now known as ykarel15:08
*** hasharAway has quit IRC15:18
*** Eighth_Doctor has quit IRC15:19
*** maxamillion[m] has quit IRC15:19
*** mordred has quit IRC15:20
*** irclogbot_3 has quit IRC15:21
*** irclogbot_2 has joined #zuul15:23
fungidoes anyone have opinions on dropping gentoo testing for base roles in zuul-jobs until opendev has working gentoo images again? https://review.opendev.org/77110515:49
fungiit's currently blocking other changes15:50
*** masterpe has joined #zuul15:50
*** Eighth_Doctor has joined #zuul15:50
*** mordred has joined #zuul15:50
*** nils has quit IRC15:53
clarkb++ to dropping until we can test it properly15:53
avassfungi: that's probably a good idea, or make them non-voting. I'm not sure who uses gentoo however15:54
clarkbshould be simple to comment it out in the pipeline config for now15:54
corvus+2 from me15:54
fungithere's already a revert of it proposed we can recheck to confirm it's working again15:55
avassI'll let someone else +3 that15:55
*** sshnaidm|afk is now known as sshnaidm|ruck15:56
fungithanks!15:58
fungiit was at least keeping back changes needed for centos stream support, which has become a popular topic of discussion lately15:59
*** hashar has joined #zuul15:59
fungier, a change needed for centos stream support16:01
tobiashcorvus: I'm currently investigating a potential bottleneck in nodepool. Profiling indicates that we might spend a significant amount of time in the NodeDeleter threads.16:04
tobiashwe have one thread for each node that is in deleting state and each one checks every two seconds if the node is gone16:05
tobiashI could think about two approaches to improve this, the easy one would be to increase that interval to like 10-15s, another approach could be to streamline this into one thread that has a list of nodes to delete and periodically checks them all16:06
openstackgerritMerged zuul/zuul-jobs master: Temporarily stop running Gentoo base role tests  https://review.opendev.org/c/zuul/zuul-jobs/+/77110516:11
*** jfoufas1 has quit IRC16:19
corvustobiash: when booting a server, there's an optimized code path inside of openstacksdk that gets the entire server list once and then notifies an individual thread when its ready.  that code started in nodepool, then moved into sdk.  but it doesn't look like there's similar code for deleting (because it's usually "fast enough" not to matter?)  anyway, a third approach would be to see if we can somehow use the16:32
corvusfull server list cache to make that more efficient.16:32
corvustobiash: we're not using this method; i wonder if it would behave differently?  https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/resource.py#L210816:36
corvusthat looks like it might be the delete counterpart to the wait_for_server we use on creation16:36
clarkbcorvus: yes I think we expect server deletes to take only a few seconds16:36
clarkbmake the request, poll a small number of times then mark it done16:36
clarkbwhereas creates take 45-300 seconds or so16:37
corvusclarkb: yeah, and generally that's beneficial because it lets us turn over servers quickly which helps avoid backlogs.  so i think we should try to avoid the 'increase interval' approach -- it may trade one kind of backlog for another.  (<-- tobiash)16:38
corvusi haven't fully paged in all of the sdk code though, so i'm not certain about exactly where the caching is happening these days16:39
corvusjust mostly saying that's another avenue we should look at16:39
tobiashI was always wondering if that caching actually works16:39
tobiashis there a way to verify that?16:39
corvustobiash: maybe run with sdk request logging?16:40
tobiashyeah, probably16:40
tobiashthe getServer we use atm seems to already use list_servers in the background16:41
corvusi'm assuming that's where the benefit is; so if wait_for_delete does the same thing, then we might get a benefit16:42
tobiashthat's the profile I got with sigusr2: http://paste.openstack.org/show/801839/16:45
tobiashlooking at the sdk code I believe that we're already using the optimized variant16:47
clarkbianw: thinking about the leaked /opt/dib_tmp contents from opendev's nodepool-builders I wonder if we should update nodepool to clear out that dir on startup similar to how zuul executors clean up their build dirs16:47
clarkbianw: in particular I think many of them are leaking when we restart nodepool-builder during the middle of builds as dib doesn't manage to clean up after itself in that situation. And we do that whenever nodepool docker image updates16:48
clarkbmaybe only clean it up if it is explicitly set in the nodepool.yaml config?16:48
*** jamesmcarthur has joined #zuul16:51
openstackgerritMatthieu Huin proposed zuul/zuul-client master: Add show running-jobs subcommand  https://review.opendev.org/c/zuul/zuul-client/+/77185316:53
*** jpena is now known as jpena|off16:59
*** ykarel has quit IRC17:09
*** ykarel has joined #zuul17:11
*** ykarel has quit IRC17:17
*** jcapitao has quit IRC17:21
corvustobiash: but from that profile, can you tell that the delete path is using list_servers?  that could be from launching servers, right?17:27
clarkbguillaumec: I'm trying to catch up on the zuul + gerrit 3.3.x situation. I think in a previous converstaion I saw that you mentioned the proper fix hasn't landed yet? Do you have a link to that fix?17:31
clarkbthe gerrit fix I mean17:31
zbryou may want to send feedback on upcoming patternfly native log viewer: https://github.com/patternfly/patternfly-design/issues/955#issuecomment-76479645417:32
*** hamalq has joined #zuul17:33
guillaumecclarkb, monorail ticket https://bugs.chromium.org/p/gerrit/issues/detail?id=13800  , intermediate fix which make patchset level comment as old style comment in stream-events: https://gerrit-review.googlesource.com/c/gerrit/+/291144 (merged)17:34
guillaumecclarkb, patch from Luca (not yet merged) https://gerrit-review.googlesource.com/c/gerrit/+/291225 which exposes patchset level comment as new data field in stream-events17:35
clarkbguillaumec: Zuul is currently compatible with the intermediate fix, but needs to add support for the unmrged change?17:35
clarkb(also I've starred that issue now, thanks)17:35
guillaumecclarkb, yes,  support for  https://gerrit-review.googlesource.com/c/gerrit/+/291225 was added in zuul by https://review.opendev.org/c/zuul/zuul/+/76674817:36
clarkbexcellent, thank you17:36
*** rpittau is now known as rpittau|afk17:42
*** rlandy is now known as rlandy|biab17:43
zbris it possible to tell zuul to load roles from a subfolder of a repository? how?17:45
avasszbr: I don't think so but I could be wrong17:49
avassI'd expect that to be an option in the tenant config otherwise17:50
avassor job config17:51
*** bhavikdbavishi has joined #zuul17:58
corvuszbr: see paragraphs 4 and 5 of https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.roles18:09
*** rlandy|biab is now known as rlandy18:41
zbrand what happens if a project looks like both standalone role and also has a roles folder? i will findout soon :p19:02
zbrbtw: we really need to do something about sphinx search, is pretty much useless now; just try to search for “role” or “roles”... results are...19:03
*** bhavikdbavishi has quit IRC19:15
*** Shrews has quit IRC19:27
*** Shrews has joined #zuul19:28
*** jamesmcarthur has quit IRC19:52
*** hashar has quit IRC20:00
*** jamesmcarthur has joined #zuul20:03
*** irclogbot_2 has quit IRC20:14
*** tosky has quit IRC20:14
*** tosky has joined #zuul20:15
*** irclogbot_3 has joined #zuul20:17
*** ajitha has quit IRC20:56
*** rlandy is now known as rlandy|biab21:17
*** GomathiselviS has joined #zuul21:29
*** jamesmcarthur_ has joined #zuul21:45
*** jamesmcarthur has quit IRC21:49
*** jamesmcarthur has joined #zuul22:02
*** jamesmcarthur_ has quit IRC22:05
clarkbianw: did you see my question/thought about having nodepool-builder clean up dib_tmp/ if it is configured in nodepool.yaml when the builder starts up?22:13
clarkbianw: I believe that if the builder (and dib) have stopped and are started again weshouldn't need any of the content there any longer22:13
ianwclarkb: umm no but i agree it's probably helpful; although my worry would be about lost mounts22:14
clarkboh hrm22:15
clarkbwe'd be deleting things out from under mounts?22:15
clarkbpotentially anyway22:15
ianwi mean dib should not be leaving mounts behind ... but that's not to say it wouldn't happen in practicality22:16
clarkbya, and I think the major source of these leaks is when we get a new nodepool-builder image and stop the processes and restart on the new image22:16
clarkbdib doesn't seem to very gracefully handle that (though maybe we could tell docker-compose to do a more gentle shutdown, I think it uses sigint then sigkill if sigint doesn't work)22:17
clarkbdo you know if dib sets up its exit routines as signall handlers too? I wonder if that would go a long way to addressing this22:18
*** rlandy|biab is now known as rlandy22:20
*** GomathiselviS has quit IRC22:22
corvusclarkb: i'm pretty sure it does and n-b relies on that22:31
corvusso it *might* be the case that we "only" need to make the docker-compose -> nodepool-builder kill more graceful22:32
clarkbaha22:33
clarkblet me read some docker compose docs22:33
clarkb"by default stop using sigterm"22:33
corvusclarkb: just speculating -- it might even be the case that n-b is getting a signal that it's handling, and it kills dib gracefully, but that takes too long and compose kills it harder.22:34
clarkbya the wait before escalating from sigterm to sigkill is 10s22:35
clarkbwhich is maybe not long enough to do file deletions and unmounts22:35
clarkbI know the rm's I was doing yseterday took a while (its lots of small files and consequently stat'ing)22:36
openstackgerritAndy Ladjadj proposed zuul/zuul master: [web][config] move timezone component to preferences  https://review.opendev.org/c/zuul/zuul/+/75592922:47
*** jamesmcarthur_ has joined #zuul22:51
*** jamesmcarthur has quit IRC22:55
*** jamesmcarthur has joined #zuul23:02
*** jamesmcarthur_ has quit IRC23:05
mordredcorvus, clarkb: makes me want a OS level thing similar to the tempdir fixture23:07
mordred"this is a tempdir for this process and when the process dies please OS delete it kthxbai"23:08
corvusmordred: heh, when you say it that way, it makes me realize that there sort-of already is: if you unlink a dir while still using it in posix, you get that behavior.  however, i dunno about mounts under it....23:09
*** jamesmcarthur has quit IRC23:12
*** jamesmcarthur has joined #zuul23:16
*** jamesmcarthur has quit IRC23:16
*** jamesmcarthur has joined #zuul23:16
*** holser has quit IRC23:17
*** holser has joined #zuul23:19
*** harrymichal has quit IRC23:23
*** rlandy has quit IRC23:26
*** holser has quit IRC23:35
mordredyah - I also don't know - I'm guessing somethign would be weird23:39
*** jamesmcarthur has quit IRC23:55
*** jamesmcarthur has joined #zuul23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!