Tuesday, 2020-02-18

*** lennyb has quit IRC00:02
*** lennyb has joined #zuul00:03
clarkbtobiash: ok I think there is another issue. select.poll objects only update their fd lists on subsequent poll calls00:34
clarkbthis doesn't seem to be an issue for the epoll method, but it does mean that the poll tests are unhappy without a timeout00:34
clarkbtobiash: I pushed up a new patchset to address these issues. Feel free to update if you find other problems00:54
*** rlandy has quit IRC01:10
*** Goneri has quit IRC01:25
*** bhavikdbavishi has joined #zuul02:12
*** igordc has quit IRC02:17
*** bhavikdbavishi1 has joined #zuul02:28
*** bhavikdbavishi has quit IRC02:30
*** bhavikdbavishi1 is now known as bhavikdbavishi02:30
*** rfolco has quit IRC02:33
*** rfolco has joined #zuul02:33
*** rfolco has quit IRC02:38
*** jamesmcarthur has joined #zuul02:55
*** jamesmcarthur has quit IRC04:14
*** jamesmcarthur has joined #zuul04:19
*** bhavikdbavishi has quit IRC04:55
*** bolg has joined #zuul05:16
*** jamesmcarthur has quit IRC05:21
*** evrardjp has quit IRC05:34
*** evrardjp has joined #zuul05:34
*** raukadah is now known as chandankumar06:02
openstackgerritMerged zuul/zuul master: Cleanup handlers defined in OS_LOG_DEFAULTS  https://review.opendev.org/70794206:05
openstackgerritMerged zuul/zuul master: Don't set OS_LOG_DEFAULTS if unset  https://review.opendev.org/70778306:21
*** felixedel has joined #zuul06:40
*** saneax has joined #zuul07:01
openstackgerritMerged zuul/zuul master: cleanup: install docker-compose via role  https://review.opendev.org/70790307:24
*** Defolos has joined #zuul07:37
*** dpawlik has joined #zuul07:38
*** bhavikdbavishi has joined #zuul07:39
*** bhavikdbavishi1 has joined #zuul07:57
*** bhavikdbavishi has quit IRC08:00
*** bhavikdbavishi1 is now known as bhavikdbavishi08:00
*** avass has joined #zuul08:06
*** felixedel has quit IRC08:08
swestcorvus: could you have a look at https://review.opendev.org/#/c/643309/ again?08:23
*** johanssone has quit IRC08:28
*** johanssone has joined #zuul08:32
*** tosky has joined #zuul08:32
*** jpena|off is now known as jpena08:52
*** mhu has joined #zuul09:27
*** felixedel has joined #zuul09:33
*** sshnaidm|afk is now known as sshnaidm09:59
openstackgerritMerged zuul/zuul master: Fix github app authentication to work with checks API endpoints (2)  https://review.opendev.org/70810910:20
*** fdegir has quit IRC10:34
*** fdegir has joined #zuul10:35
*** felixedel has quit IRC11:00
*** felixedel has joined #zuul11:29
*** rfolco has joined #zuul11:55
*** felixedel has quit IRC12:19
*** fdegir has quit IRC12:29
*** fdegir has joined #zuul12:31
*** felixedel has joined #zuul12:37
*** jpena is now known as jpena|lunch12:39
openstackgerritTobias Henkel proposed zuul/zuul master: Increase cherrypy thread pool  https://review.opendev.org/70837712:43
*** rlandy has joined #zuul12:56
openstackgerritFabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver  https://review.opendev.org/67271213:08
openstackgerritFabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch  https://review.opendev.org/64492713:12
openstackgerritFabien Boucher proposed zuul/zuul master: URLTrigger driver time based  https://review.opendev.org/63556713:14
openstackgerritJan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper  https://review.opendev.org/70571613:40
*** jpena|lunch is now known as jpena13:42
*** Goneri has joined #zuul13:46
openstackgerritTobias Henkel proposed zuul/zuul master: Increase cherrypy thread pool size  https://review.opendev.org/70837713:51
*** gothicmindfood has joined #zuul13:57
*** jamesmcarthur has joined #zuul14:03
*** igordc has joined #zuul14:04
Shrewstobiash: that's quite a jump from 10 to 50 threads. are all 50 pre-created, or is that just a max value?14:08
*** bolg has quit IRC14:09
tobiashShrews: the thread pool in cherrypy is static14:14
tobiashso pre-created14:14
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch  https://review.opendev.org/70840214:36
openstackgerritTobias Henkel proposed zuul/zuul master: Log duration of tenant reconfigurations  https://review.opendev.org/70840314:43
openstackgerritSimon Westphahl proposed zuul/zuul master: wip: Proof of concept for matrix jobs  https://review.opendev.org/70840614:53
*** sgw has joined #zuul14:55
*** jamesmcarthur has quit IRC14:56
*** felixedel has quit IRC14:56
*** jamesmcarthur has joined #zuul14:57
*** michael-beaver has joined #zuul14:59
*** igordc has quit IRC15:00
*** johanssone has quit IRC15:02
*** Goneri has quit IRC15:03
*** jamesmcarthur has quit IRC15:04
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch  https://review.opendev.org/70840215:05
*** johanssone has joined #zuul15:08
*** Goneri has joined #zuul15:11
mordredShrews: how was not being at work?15:16
Shrewswet15:16
mordredShrews: just they way you like it15:16
mordredShrews: I'm now picturing you just sitting in a chair outside in the rain glaring the whole time15:17
*** NBorg has joined #zuul15:21
clarkbI'd suggest an umbrella but I think it would blow away15:21
mordredclarkb: but you could maybe use the umbrella as a boat if the water got too high15:24
*** jamesmcarthur has joined #zuul15:24
*** bhavikdbavishi has quit IRC15:40
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: [DNM] Use centos8 deps repo  for RHEL8 nodes  https://review.opendev.org/70843015:53
*** saneax has quit IRC15:59
*** chandankumar is now known as raukadah16:00
mordredcorvus: I was just telling ssnaidm about using a logging config because he's having issues with verbose logging16:07
mordredcorvus: he asked "I suppose I can find the explanations about keys in zuul docs somewhere?"16:07
mordredcorvus: but I don't see anywhere we have the keys documented ... do we and I'm just missing it?16:08
corvusmordred: no docs on that.  it's not intended to be a user-configurable feature16:08
corvuswhat's the issue?16:08
sshnaidmcorvus, so maybe I'll look at code, which part can help here?16:09
corvus(let's fix the underlying issue)16:09
mordredsshnaidm: it's also possible the answer is just that you should stop running with debug16:09
sshnaidmcorvus, the issue is too much noise with debug on, what should be fine actually, but I'm looking for ways to reducer it with more flexible logs config16:09
corvussshnaidm: are you running with latest master?16:10
sshnaidmcorvus, almost16:10
corvuswe just merged a logging change lemme check16:10
corvussshnaidm: check if you're running with 1e04bdb4aa2e76d5c0217185071ba123e41704c916:10
sshnaidmmordred, yeah, and I can live with that, but if there are ways, why not..16:10
mordredsshnaidm: the overall intent is that running logging at INFO level should provide everythign an operator needs without extra noise16:10
mordredso we've been working to make the default logging solid16:11
corvussshnaidm: 1e04bdb4aa2e76d5c0217185071ba123e41704c9 gets rid of a bunch of the http/ssh/git traffic that was being logged at debug.  so if that was what was flooding you, you should be all set :)16:12
fungiprobably already something you've considered, but could it be gear 0.15.0? i haven't heard what's going on with the excessive logging exactly16:12
sshnaidmcorvus, cool, I think I can update to this version, thanks16:12
corvussshnaidm: if what's left is too much, then i'd suggest switching to info.  and if info is too little, let's figure out what we need to move from debug to info to make it useful.16:12
corvusfungi: good point.  we did merge a pin to avoid 0.15.0, but if that slipped into sshnaidm's deployment, that could be an issue16:13
sshnaidmcorvus, I think per-project config settins would help me, but not sure it's possible..16:13
sshnaidmcorvus, ok, I think I'll update to latest zuul and see if it helps firstly16:14
corvussshnaidm: yeah, per-project would be prohibitively difficult.  just let us know what you want or don't want in logging and we'll try to adjust it.  i really want a good logging config without folks needing to write a logging config file.16:14
sshnaidmcorvus, actually I think per-connection setting are fine too16:16
sshnaidmlike for github.com, opendev.org, etc16:16
corvusthat's probably more do-able16:17
mordredcorvus: the use case for the zuul in question is one that they're using to reproduce upstream gate issues - so it has a set of opendev projects configured so that it can get the jobs to re-use - but it's really just tracking those and not running jobs on them - right sshnaidm?16:19
sshnaidmmordred, correct16:19
corvusmakes sense16:20
sshnaidmmordred, the only way we use them is required-projects and depends-on in gerrit..16:20
mordredcorvus: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/templates/main.yaml.j2 is their zuul config, fwiw16:20
*** jamesmcarthur_ has joined #zuul16:32
tobiashcorvus: I think we also should do a gear release with the revert from yesterday to fix older zuul releases?16:34
corvustobiash: yeah16:36
*** jamesmcarthur has quit IRC16:36
corvuscommit 9933f068213c9da6c4b0ea59659c13bcf2ebe104 (HEAD -> master, tag: 0.15.1, origin/master, origin/HEAD, refs/changes/61/708261/1)16:36
corvustobiash: look good? ^16:36
tobiash++16:37
clarkbwfm though I think the revert revert is working now (at least as much as I can test it locally, no OSX)16:38
tristanCzuul-maint: could you please add the zuul-runner spec to your review queue : https://review.opendev.org/68127716:42
*** jpena is now known as jpena|brb16:46
*** Defolos has quit IRC16:53
*** igordc has joined #zuul17:09
*** rishabhhpe has joined #zuul17:12
rishabhhpeHi All , can anyone please help me to get through this error17:12
rishabhhpehttp://paste.openstack.org/show/789713/17:12
corvusrishabhhpe: it looks like zuul can't connect to the gerrit server.  you might try running ssh with zuul's private key to see if you can connect manually17:15
rishabhhpeyes i tried it is working17:16
corvusrishabhhpe: can you tell us more about the error?  is it able to connect and then the error happens later on?  or is it not ever able to connect?17:17
*** jpena|brb is now known as jpena17:23
corvustristanC: on 702106 i see 2 comments from clarkb on ps22 that are unreplied; do you want to reply to those?17:32
corvustobiash: i think tristanC addressed your comments on 702106, can you confirm?17:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #zuul17:34
tristanCcorvus: i replied to clarkb questions as a global comment, e.g. ansible command stdin can be flaky, and uses custom module would let switch to a dhall python binding later17:35
corvustristanC: ah, yep, sorry i didn't see that; was reviewing in diff mode17:36
corvusclarkb: in which case -- want to give 702106 a once-over?17:36
openstackgerritMerged zuul/zuul-operator master: Update sdk to v0.13.0 and install dhall-to-json  https://review.opendev.org/70210317:39
*** felixedel has joined #zuul17:41
corvusmordred: do you want to give https://review.opendev.org/643309 another look?17:43
tristanCoh wow, that's exciting, can't wait to use docker.io/zuul/zuul-operator to run zuul :) Thanks corvus for pushing this!17:44
corvustristanC: thank you!  i think everyone's had a chance to look at the dhall stuff and register their interest, so i think we can start merging those as we get final code reviews17:45
clarkbcorvus: 702106 lgtm17:49
clarkbI +2'd but did not approve17:49
tristanChttps://review.opendev.org/#/c/704187/ needs a config-core to encrypt the docker hub secret and setup zuul-operator-promote-image job17:50
clarkbtristanC: gerrit reports that change cannot merge due to a path conflict and needs a rebase17:51
clarkb(did .zuul.yaml get renamed or put into .zuul.d/ ?17:51
tristanCclarkb: iirc the job parent changed, i can rebase but the change really needs a secret17:52
corvustristanC: i'll do the rebase and add the secret17:52
tobiashcorvus: 702106 lgtm17:53
*** rishabhhpe has quit IRC17:54
*** rishabhhpe has joined #zuul17:54
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add zuul-operator-promote-image job  https://review.opendev.org/70418717:56
rishabhhpecorvus: i am not able to do ssh also .. for process i had transferred my public to the gerrit repo but still i am getting connection time out error18:05
*** Defolos has joined #zuul18:08
AJaegertristanC: FYI, config-core has no +2 on zuul-operator - I guess you meant infra-root18:14
tristanCAJaeger: yep, corvus took care of it18:14
AJaegergreat18:15
corvusreally i think zuul-maint is entitled to that credential, but we don't have a great way to share it with new zuul-maint folks right now.18:17
*** rishabhhpe has quit IRC18:21
*** rishabhhpe has joined #zuul18:21
*** rishabhhpe has quit IRC18:25
*** jpena is now known as jpena|off18:33
*** tosky has quit IRC18:35
mordredcorvus: https://review.opendev.org/#/c/707668/ - a patch you might not have noticed - would allow https://review.opendev.org/#/c/707250/ - although now that I look at the second one, it looks ike we're not set up for speculative images with python-builder18:52
mordredoh - that's because tenants18:52
mordredright?18:52
tobiashcorvus: when you have time (not urgent) I'd like to discuss a topic related to the repo state. tldr is how we could ensure a repo state on the whole buildset with all repos that are used, not just those that are in the dependency chain19:04
corvusmordred: yeah, probably tenants19:08
corvustobiash: i thought we did do setRefs on all required-projects?19:09
tobiashcorvus: we generate the repo state during initial merge when not all required projects are known (the jobs are not frozen at that point)19:09
tobiashso I think for generating a buildset-repo-state we'd need to update the repo state after job freezing (which would cause more load on the mergers)19:10
corvustobiash: so, roughly speaking, you're saying that once we freeze the jobs for a buildset, we should also getRefs on all required projects and send them along as well19:10
corvusthat sounds reasonable19:11
tobiashyes, that makes that correct, the question however is do we want to make that effort for every buildset or make it optional due to the increased load on mergers?19:11
corvusi think always do it -- so it's more correct.  and that's why mergers are scalable :)19:12
tobiash++ :)19:12
tobiashthat was an efficient discussion :)19:13
corvustobiash: ++19:16
*** felixedel has joined #zuul19:43
*** felixedel has quit IRC19:43
openstackgerritJeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly  https://review.opendev.org/70439119:58
*** Goneri has quit IRC19:58
corvusclarkb: can you look at https://review.opendev.org/704184 ?20:43
clarkbdone20:47
*** michael-beaver has quit IRC20:49
*** jamesmcarthur_ has quit IRC20:52
*** jamesmcarthur has joined #zuul20:53
*** jamesmcarthur has quit IRC20:58
*** Goneri has joined #zuul21:08
*** jamesmcarthur has joined #zuul21:12
*** lseki has joined #zuul21:12
*** jamesmcarthur has quit IRC21:13
*** jamesmcarthur has joined #zuul21:13
lsekihello everyone o/21:19
lsekiI'm using zuul-cloner from zuul 2.6.0, and I'm facing a GitPython issue [0], whose fix was merged in v2.1.8 [1].21:19
lsekiThough zuul 2.6.0 is requiring GitPython<2.1.2 [2].21:19
lseki[0] https://github.com/gitpython-developers/GitPython/issues/68721:19
lseki[1] https://github.com/gitpython-developers/GitPython/issues/70421:19
lseki[2] https://opendev.org/zuul/zuul/src/tag/2.6.0/requirements.txt#L721:19
corvuslseki: we don't support zuul v2 anymore, so we won't be issuing a new release, but you should be able to update the requirements file in your local build/installation and install the new gitpython21:20
*** tflink_ has joined #zuul21:21
*** tflink has quit IRC21:22
lsekicorvus: oh, ok... I will do that until I migrate to zuul v3. Thanks!21:22
*** tflink_ is now known as tflink21:23
fungioof, http://zuul.opendev.org/t/zuul/status is showing a whole lot of broken... do we know the cause yet? if not, i'll dig21:24
fungiahh, it's zuul-upload-image timing out21:24
funginodepool-upload-image seems to be taking a while too21:25
corvushrm.  i'm assuming it's the build portion that's taking a long time...21:26
fungithat's not immediately apparent as i skim the timestamps in the console log21:29
funginot seeing any loooong gaps between timestamps21:29
corvuslooks like the job starts at 20:45, docker build starts at 20:53 and times out at 21:1521:29
corvusyeah, it seems like it's proceeding, just slowly.21:29
funginodepool-upload-image for 705497,2 did complete at least21:31
fungiahh, and the 5 failing changes at the top of the zuul/zuul queue are because they're dependent on the first one which is failing21:32
openstackgerritMerged zuul/nodepool master: Dockerfile: create a nodepool user with uid 10001  https://review.opendev.org/70549721:32
fungiso really we just have one build timing out, but maybe an indication we need to look at slimming down the image build jobs or bump out the timeout21:32
fungialso looking at the status dashboard, it appears that timed-out builds show up as blue in the summary line rather than red, i wonder whether that's intentional21:34
corvusit looks like a build might usually take 12 minutes21:34
corvusso this might just be a catastrophically slow node21:35
fungiyeah, if so no need to optimize for a pathological failure we rarely see21:35
clarkbwhich cloud did it run on?21:35
corvusfungi: i doubt the color is intentional; i think it's from zuulv2 which didn't have a timeout state21:35
corvusclarkb: ord21:35
clarkbok not one I would've expected to have slowness21:36
corvusme neither21:36
corvusit could also be our lack of build mirrors in docker builds?21:36
tobiashI have a change up to use mirrors but that's not working cimpletely yet21:49
tobiash(649448)21:53
tobiashIt fails with post failure but I didn't understand yet why21:53
corvustobiash: that's a really strange error that does not seem related to the change21:55
corvusbut it happened 2x21:55
tobiashcorvus: yes, so seems reproducible21:56
tobiashEven on a debug change on top21:56
corvusoooooh21:58
corvustobiash: it's evaluating "docker_images" which now has "{{ zuul_site_mirror_fqdn }}" in the variable, and that variable has "{{nodepool.region}}" in it.  but "nodepool" is only defined for the hosts in the inventory, not the executor.21:59
corvusand the push role runs on the executor21:59
tobiashhrm, that's unfortunate22:00
corvustobiash: maybe in .zuul.yaml we can say: PACKAGE_MIRROR={{ zuul_site_mirror_fqdn | default('') }}22:00
clarkbtobiash: did you have a chance to rerevie wthe geard fix?22:01
clarkbcorvus: fwiw I think https://review.opendev.org/#/c/708267/ is probably good as far as addressing tobiash geard needs22:01
tobiashclarkb: lgtm, but I hadn't time today for in depth testing22:02
corvustobiash: maybe you can let us know when you do more testing?22:03
tobiashsure, I think I'd try sibling testing against zuul (both versions and compare them)22:04
tobiashwill depends-on work or do I also need to add gear as required project?22:04
corvusi'm happy to merge the change, but i need to abdicate responsibility for it -- i wrote the original to support epoll only, and micro-optimized it for basically one use case -- supporting both always sounded too daunting to me :)22:05
clarkbtobiash: I think depends on won't work because gear is installed via pypi22:06
tobiashand I'm not keen on breaking it again so I'll do further testing before we should merge it :)22:06
clarkbif we updated the job to install from source it would work22:06
clarkb(which we can probably do in the test change)22:06
corvusclarkb: doesn't tox-siblings take care of rewriting that for us?22:07
tobiashthat was my thought22:07
clarkboh hrm actually probably22:07
clarkbI always forget that exists now22:07
clarkbso ya ignore me22:07
tobiashI'll just try it hopefully tomorrow22:07
fungiif it's a required-projects entry22:07
fungithen yeah, t-s ought to do that22:08
corvustobiash: so yeah ^ what fungi says; you'll need to add it to required-projects22:08
tobiashk, thx22:08
fungithough that means the job will always test from source, whether there's any change dependency or not22:08
fungiso if we also want to test with releases, we need two jobs22:08
corvusyeah, i think this would be a DNM change22:08
corvusa one off22:08
corvusbecause i'm not interested in gear changing :)22:09
fungithat should be fine then22:09
tobiashre mirror: I think that could work22:09
corvusbut gear is not in the zuul tenant22:09
corvusso you'd have to add that first22:09
tobiashHrm, ok22:10
corvus(even the universe does not want gear to change :)22:10
corvusbut i'm happy to approve adding it to the tenant for this if we want22:10
*** avass has quit IRC22:11
tobiashok, I'll upload a change for this tomorrow22:11
*** felixedel has joined #zuul22:21
openstackgerritMerged zuul/zuul master: Handle draft pull requests in canMerge  https://review.opendev.org/70415022:24
openstackgerritMerged zuul/zuul master: Disable pip cache when managing ansible  https://review.opendev.org/70418422:25
*** rfolco has quit IRC22:26
*** klindgren_ has quit IRC22:35
*** klindgren has joined #zuul22:36
*** jamesmcarthur has quit IRC22:49
openstackgerritMerged zuul/zuul master: Flesh out the glossary significantly  https://review.opendev.org/70439122:53
funginow that's merged, i should probably get cracking on linking up all the glossary terms throughout our docs22:53
*** Goneri has quit IRC22:54
*** jamesmcarthur has joined #zuul23:01
openstackgerritMerged zuul/zuul master: Log duration of AnsibleJob.execute()  https://review.opendev.org/70458923:19
*** jamesmcarthur has quit IRC23:19
*** armstrongs has joined #zuul23:24
*** armstrongs has quit IRC23:28
*** armstrongs has joined #zuul23:30
*** armstrongs has quit IRC23:39
*** rlandy has quit IRC23:41
openstackgerritMerged zuul/zuul master: ansible manager: only failed if last ansible failed  https://review.opendev.org/70469923:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!