Monday, 2019-03-18

*** wolverineav has quit IRC00:04
*** wolverineav has joined #openstack-infra00:05
*** slaweq_ has joined #openstack-infra00:11
*** wolverineav has quit IRC00:14
*** wolverineav has joined #openstack-infra00:22
*** slaweq_ has quit IRC00:24
*** dave-mccowan has joined #openstack-infra00:29
*** dave-mccowan has quit IRC00:31
*** wolverineav has quit IRC00:40
*** wolverineav has joined #openstack-infra00:40
*** wolverineav has quit IRC00:52
*** jamesmcarthur has joined #openstack-infra00:58
*** slaweq_ has joined #openstack-infra01:11
*** wolverineav has joined #openstack-infra01:13
openstackgerritIan Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org  https://review.openstack.org/64115501:16
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675901:16
*** wolverineav has quit IRC01:20
*** wolverineav has joined #openstack-infra01:21
*** slaweq_ has quit IRC01:24
*** wolverineav has quit IRC01:26
*** jamesmcarthur has quit IRC01:45
*** jamesmcarthur has joined #openstack-infra01:55
*** wolverineav has joined #openstack-infra02:04
*** jamesmcarthur has quit IRC02:06
*** jamesmcarthur_ has joined #openstack-infra02:06
*** slaweq_ has joined #openstack-infra02:11
*** wolverineav has quit IRC02:15
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Get executor job params  https://review.openstack.org/60707802:22
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner  https://review.openstack.org/60707902:22
*** irclogbot_1 has quit IRC02:23
*** slaweq_ has quit IRC02:24
*** edmondsw has quit IRC02:26
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915502:30
*** ricolin has joined #openstack-infra02:35
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915502:40
*** ykarel has joined #openstack-infra02:52
*** jamesmcarthur_ has quit IRC02:59
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915503:09
*** slaweq_ has joined #openstack-infra03:11
*** hongbin has joined #openstack-infra03:12
*** hongbin has quit IRC03:21
*** slaweq_ has quit IRC03:25
*** wolverineav has joined #openstack-infra03:29
*** wolverineav has quit IRC03:33
*** whoami-rajat has joined #openstack-infra03:46
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915503:48
*** bgmccollum has quit IRC03:56
*** bgmccollum has joined #openstack-infra04:00
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915504:01
*** janki has joined #openstack-infra04:05
*** udesale has joined #openstack-infra04:08
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: config: use yaml.safe_load instead of load  https://review.openstack.org/64384904:14
*** slaweq_ has joined #openstack-infra04:14
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915504:18
*** Lucas_Gray has joined #openstack-infra04:20
*** Lucas_Gray has quit IRC04:21
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915504:24
*** slaweq_ has quit IRC04:24
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915504:35
*** ramishra has joined #openstack-infra04:46
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915504:48
*** rodrigods has quit IRC05:01
*** slaweq_ has joined #openstack-infra05:16
*** slaweq_ has quit IRC05:24
*** pcrews has joined #openstack-infra05:28
*** bgmccollum has quit IRC05:28
*** bgmccollum has joined #openstack-infra05:31
*** ykarel_ has joined #openstack-infra05:43
*** rcernin has quit IRC05:43
*** ykarel has quit IRC05:45
*** jaosorior has joined #openstack-infra05:45
*** ykarel_ is now known as ykarel05:51
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: rewrite refreshable and filters with higher order component  https://review.openstack.org/64385605:56
*** rcernin has joined #openstack-infra05:59
*** jbadiapa has joined #openstack-infra06:06
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915506:06
*** slaweq_ has joined #openstack-infra06:11
*** slaweq_ has quit IRC06:24
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915506:25
*** e0ne has joined #openstack-infra06:40
*** kjackal has joined #openstack-infra06:42
*** wolverineav has joined #openstack-infra06:45
*** wolverineav has quit IRC06:50
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: rewrite refreshable and filters with higher order component  https://review.openstack.org/64385606:54
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915506:56
*** jaosorior has quit IRC06:58
*** e0ne has quit IRC07:02
*** AJaeger has quit IRC07:03
*** kopecmartin|off is now known as kopecmartin07:03
*** AJaeger has joined #openstack-infra07:07
*** tkajinam_ has joined #openstack-infra07:11
*** jaosorior has joined #openstack-infra07:11
*** tkajinam has quit IRC07:13
*** jtomasek has joined #openstack-infra07:13
*** slaweq_ has joined #openstack-infra07:13
*** gfidente has joined #openstack-infra07:15
openstackgerritMerged openstack-infra/zuul master: Manage ansible installations within zuul  https://review.openstack.org/63193007:16
openstackgerritMerged openstack-infra/zuul master: Validate ansible installations on startup  https://review.openstack.org/63741807:16
openstackgerritMerged openstack-infra/zuul master: Make ansible version configurable  https://review.openstack.org/63742207:16
openstackgerritMerged openstack-infra/zuul master: Install ansible during executor startup if needed  https://review.openstack.org/64064407:16
openstackgerritMerged openstack-infra/zuul master: Run tox remote concurrent  https://review.openstack.org/64065407:16
openstackgerritMerged openstack-infra/zuul master: Support ansible 2.6  https://review.openstack.org/63193107:16
openstackgerritMerged openstack-infra/zuul master: Support ansible 2.7  https://review.openstack.org/63193207:16
openstackgerritMerged openstack-infra/zuul master: Switch default ansible version to 2.7  https://review.openstack.org/63742407:16
*** pcaruana has joined #openstack-infra07:19
*** kjackal has quit IRC07:23
*** dpawlik has joined #openstack-infra07:23
*** pgaxatte has joined #openstack-infra07:27
*** jaosorior has quit IRC07:35
*** kjackal has joined #openstack-infra07:37
*** apetrich has joined #openstack-infra07:38
*** jaosorior has joined #openstack-infra07:44
*** jaosorior has quit IRC07:44
*** xek has joined #openstack-infra07:45
*** iurygregory has joined #openstack-infra07:52
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: manifest: add ansible-config.conf  https://review.openstack.org/64387207:56
*** yamamoto has quit IRC08:04
*** tosky has joined #openstack-infra08:05
*** yamamoto has joined #openstack-infra08:05
*** yamamoto has quit IRC08:06
*** tosky has quit IRC08:07
*** ginopc has joined #openstack-infra08:08
*** kjackal has quit IRC08:08
*** kjackal has joined #openstack-infra08:11
*** zhangfei has joined #openstack-infra08:14
*** ccamacho has joined #openstack-infra08:19
*** rpittau|afk is now known as rpittau08:19
openstackgerritMerged openstack-infra/zuul master: Support common ansible dependencies  https://review.openstack.org/64353408:20
*** ykarel is now known as ykarel|lunch08:21
openstackgerritMerged openstack-infra/zuul master: Add missing docs for zuul-manage-ansible  https://review.openstack.org/64354408:22
openstackgerritMerged openstack-infra/zuul master: ansible-config: add common extra dependencies  https://review.openstack.org/64353708:23
*** tosky has joined #openstack-infra08:24
*** tkajinam_ has quit IRC08:24
*** priteau has joined #openstack-infra08:26
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: manifest: add zuul/ansible and ansible-config.conf  https://review.openstack.org/64387208:27
*** jpena|off is now known as jpena08:29
*** lennyb has joined #openstack-infra08:30
*** ramishra has quit IRC08:35
*** ramishra has joined #openstack-infra08:35
*** sshnaidm|afk is now known as sshnaidm|rover08:42
*** kjackal_v2 has joined #openstack-infra08:43
*** kjackal has quit IRC08:44
openstackgerritMerged openstack-infra/zuul master: Install virtualenv from source  https://review.openstack.org/64365508:55
*** rossella_s has quit IRC09:04
*** rossella_s has joined #openstack-infra09:04
*** dtantsur|afk is now known as dtantsur09:11
openstackgerritrenminmin proposed openstack/diskimage-builder master: Add DIB_LOCAL_IMAGE to use ubuntu local cloud image  https://review.openstack.org/64388509:11
*** rascasoft has joined #openstack-infra09:12
*** zhangfei has quit IRC09:15
*** kjackal has joined #openstack-infra09:18
*** kjackal_v2 has quit IRC09:19
*** zhangfei has joined #openstack-infra09:27
*** ykarel|lunch is now known as ykarel09:27
*** rfolco has joined #openstack-infra09:34
*** roman_g has joined #openstack-infra09:34
*** kjackal has quit IRC09:50
*** kjackal has joined #openstack-infra09:51
*** wolverineav has joined #openstack-infra09:54
*** wolverineav has quit IRC09:58
*** Lucas_Gray has joined #openstack-infra10:06
*** kjackal has quit IRC10:21
*** yamamoto has joined #openstack-infra10:22
*** kjackal has joined #openstack-infra10:22
*** yamamoto has quit IRC10:22
dtantsurhi folks! anything work with review.o.o? takes ages to connect, gertty goes to offline mode10:27
*** udesale has quit IRC10:28
*** e0ne has joined #openstack-infra10:28
*** yamamoto has joined #openstack-infra10:28
*** Lucas_Gray has quit IRC10:32
*** Lucas_Gray has joined #openstack-infra10:33
fricklerdtantsur: looks normal from my pov. you might possibly want to check for issues in your ipv6 connectivity10:38
dtantsurfrickler: it's back now (was not only me before), probably a temporary glitch10:39
rpittaufrickler, had also some issues during the morning, looks good now since 10 minutes10:40
fricklerhmm, o.k., I had no issues earlier either. but if things are back to working now, then let's hope it stays that way10:42
*** jbadiapa has quit IRC10:44
*** kjackal has quit IRC10:55
*** kjackal has joined #openstack-infra10:56
*** zhangfei has quit IRC11:04
*** kjackal has quit IRC11:05
*** slaweq_ is now known as slaweq11:14
*** kjackal has joined #openstack-infra11:15
openstackgerritIan Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org  https://review.openstack.org/64115511:17
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675911:17
*** sajauddin has joined #openstack-infra11:36
*** kaiokmo has joined #openstack-infra11:39
*** kaiokmo has quit IRC11:39
*** kaiokmo has joined #openstack-infra11:40
*** udesale has joined #openstack-infra11:45
*** rcernin has quit IRC11:46
*** edmondsw has joined #openstack-infra11:50
*** jaosorior has joined #openstack-infra11:51
*** rh-jelabarre has joined #openstack-infra11:57
*** rlandy has joined #openstack-infra12:01
*** udesale has quit IRC12:01
*** udesale has joined #openstack-infra12:02
*** trown|outtypewww is now known as trown12:09
*** sajauddin has quit IRC12:13
*** markvoelker has quit IRC12:22
*** udesale has quit IRC12:27
*** udesale has joined #openstack-infra12:28
*** yamamoto has quit IRC12:30
*** whoami-rajat has quit IRC12:35
*** eharney has joined #openstack-infra12:39
*** kgiusti has joined #openstack-infra12:42
*** Lucas_Gray has quit IRC12:50
*** e0ne has quit IRC12:51
*** Lucas_Gray has joined #openstack-infra12:51
*** zul has joined #openstack-infra12:51
*** e0ne has joined #openstack-infra12:52
*** Lucas_Gray has quit IRC12:53
*** Lucas_Gray has joined #openstack-infra12:53
*** pcaruana|afk| has joined #openstack-infra12:54
*** dpawlik has quit IRC12:55
*** yamamoto has joined #openstack-infra12:55
*** eharney has quit IRC12:56
*** pcaruana has quit IRC12:56
*** jpena is now known as jpena|lunch12:57
*** mriedem has joined #openstack-infra12:58
openstackgerritMerged openstack-infra/zuul master: manifest: add zuul/ansible and ansible-config.conf  https://review.openstack.org/64387213:02
*** jtomasek_ has joined #openstack-infra13:02
*** jtomasek has quit IRC13:03
*** jtomasek_ has quit IRC13:05
*** jtomasek has joined #openstack-infra13:06
*** whoami-rajat has joined #openstack-infra13:08
*** dave-mccowan has joined #openstack-infra13:09
*** dave-mccowan has quit IRC13:15
*** needssleep is now known as TheJulia13:18
*** dpawlik has joined #openstack-infra13:24
*** lseki has joined #openstack-infra13:27
*** hwoarang has quit IRC13:37
*** hwoarang has joined #openstack-infra13:38
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Whitelist ansible_python_interpreter for add_host  https://review.openstack.org/64372613:48
*** pcaruana|afk| has quit IRC13:48
*** kjackal has quit IRC13:55
*** kjackal_v2 has joined #openstack-infra13:55
*** e0ne has quit IRC13:58
*** Qiming has quit IRC13:59
*** Qiming has joined #openstack-infra14:00
*** jpena|lunch is now known as jpena14:04
*** eharney has joined #openstack-infra14:04
*** e0ne has joined #openstack-infra14:05
openstackgerritMerged openstack-infra/zone-opendev.org master: Add acme.opendev.org  https://review.openstack.org/64115414:07
*** verdurin has joined #openstack-infra14:12
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul master: Set fingergw user to zuul  https://review.openstack.org/64395414:14
*** Lucas_Gray has quit IRC14:22
*** Lucas_Gray has joined #openstack-infra14:23
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul master: Update executor ansible on installation  https://review.openstack.org/64395714:24
openstackgerritMonty Taylor proposed openstack-infra/git-review master: Use remote_url instead of remote for download  https://review.openstack.org/64396614:31
mordredzbr, corvus, fungi: ^^14:32
mordredI discovered that usepushurl does not work with opendev.org - and then I was sad because I use usepushurl14:32
mordredluckily there is a four character ix14:32
mordredfix14:32
*** pcaruana has joined #openstack-infra14:33
*** yamamoto has quit IRC14:34
zbrmordred: nice, testing it now.14:35
mordredzbr: it works for me locally - I haven't run the test suite, so I'm not sure yet if it breaks something there14:35
*** yamamoto has joined #openstack-infra14:37
openstackgerritMonty Taylor proposed openstack-infra/git-review master: Use remote_url instead of remote for download  https://review.openstack.org/64396614:37
mordredzbr: ^^ added releasenote14:37
*** armax has joined #openstack-infra14:39
openstackgerritSorin Sbarnea proposed openstack-dev/pbr master: DNM: add boo  https://review.openstack.org/64399114:39
openstackgerritAdam Coldrick proposed openstack-infra/storyboard master: Use one database per test class rather than per test  https://review.openstack.org/64400014:40
zbrmordred: i suppose that the move to opendev would trigger a set of fixes in tools like git-review, gertty,....14:40
fricklerianw: hmm, nice error, I think ansible should also be a bit more intelligent trying to expand this: http://logs.openstack.org/55/641155/14/check/system-config-run-base/a75e854/job-output.txt.gz#_2019-03-18_11_29_05_81405914:41
openstackgerritJens Harbott (frickler) proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org  https://review.openstack.org/64115514:41
*** e0ne has quit IRC14:41
*** e0ne has joined #openstack-infra14:42
*** yamamoto has quit IRC14:42
fungizbr: yes, i expect changing the domain name we've been using for these services for upwards of 8 years will shake out some unexpected bugs, though hopefully they're all fairly minor14:44
fungi(not to mention we're trying to keep things backward-compatible as much as possible to keep the transition mostly seamless)14:45
clarkbpeople do you these tools against other domains though. Hopefully not many corner cases as a result14:46
clarkb*do use these toole14:46
clarkbbah typing is hard14:46
fungiright, i expect the issues to be corner cases in infrequently-used features, like the one mordred just identified14:47
roman_gWow, wow, what I've found: I'm not able to remove a reviewer, if he has left negative review for a previous patch set. I just didn't want to spam reviewer's e-mail while I continue to work on the PS.14:48
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler  https://review.openstack.org/54216014:49
roman_gBut I think I can remove reviewer who has left positive vote.14:49
fungiroman_g: yes, that's correct14:50
roman_g*thumbs_up*14:50
fungiroman_g: see https://review.openstack.org/Documentation/access-control.html#category_remove_reviewer for documentation of that behavior14:50
roman_gThanks :)14:51
*** e0ne has quit IRC14:53
mordredzbr, fungi, clarkb: yah- although at some point we also need to file a bug with gitea so we can fix our ability to push refs/changes/* to it14:53
*** woojay has joined #openstack-infra14:54
*** e0ne has joined #openstack-infra14:55
fungimordred: as in gitea is allowing that when we want it rejected?14:56
*** josephrsandoval has joined #openstack-infra15:02
*** jamesmcarthur has joined #openstack-infra15:02
*** e0ne has quit IRC15:05
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Tolerate missing project  https://review.openstack.org/57987215:10
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line  https://review.openstack.org/64424115:11
*** armax has quit IRC15:18
mordredfungi: no - the other way around - we aren't doing it because gitea gets confused and unhappy if we do15:19
*** e0ne has joined #openstack-infra15:20
*** josephrsandoval has quit IRC15:23
*** janki has quit IRC15:26
clarkbmordred: corvus: so the thing I was trying to debug late friday is the docker image job collection doesn't seem to be 100%reliable and skopeo is the current suspect actor15:27
clarkbin the post image build sync between the buildset registry and the intermediate registry I/we were seeing skopeo go out to lunch until the job timeout hit15:27
clarkblooking at the registry logs on both sides it seemed that the data was transfered and skopeo was just failing to exit properly (the logs showed a buch of happy blob transfers and there were no open tcp connections between nodes at the time)15:28
mordredclarkb: AWESOME15:29
clarkbone thing I noticed is that there is a newer version of skopeo available on the atomic ppa but we don't auto update to it with our unattended upgrades policy15:29
clarkbI figured maybe step 0 in further debugging is to upgrade that?15:29
clarkbfwiw this is why we haven't updated our gitea images with the logos or newer gitea yet15:30
clarkbalso for some reason the gitea version bump didn't run image build jobs in check but did run them in gate (oh ya this failure isn't 100%)15:30
corvusclarkb: i saw your debugging in backlog thanks... i also think may we should do this: https://review.openstack.org/644241  (cc mordred)15:32
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing  https://review.openstack.org/64366615:33
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424715:33
clarkb++ that should help with debugging15:33
*** yamamoto has joined #openstack-infra15:34
*** dpawlik has quit IRC15:35
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul master: Update executor ansible on installation  https://review.openstack.org/64395715:35
corvusconfig-core: sometime today we're going to upgrade to a version of zuul that supports multiple versions of ansible.  should we set the global default to 2.5 (the current version) before we do that, or should we "upgrade" all our jobs to run on ansible 2.7 at the same time?15:36
clarkbcorvus: can you check the comments on https://review.openstack.org/#/c/644241/1 ?15:37
mordredcorvus: I thnik I'm ok with upgrading - since we can always set the value if things break - and we can set the value on targetted jobs if they do15:37
clarkbThe openstack release process is starting to pick up. I think as long as we are planning to be around to debug jobs with them and/or set the ansible verison for them as necessary bumping should be fine15:38
openstackgerritAdam Coldrick proposed openstack-infra/storyboard master: Use one database per test class rather than per test  https://review.openstack.org/64400015:38
fricklercorvus: system-config-run-base-ansible-devel seems to be hard failing, so I'm not sure how good we'd be at running other stuff with 2.7, so I'd vote to stick to 2.5 until we get that fixed15:40
clarkbfrickler: I think the devel job failures are related to the testinfra api incompat with running ansible directly15:40
clarkbfrickler: I expect that is mostly non representative since that is also for ansible 2.8 development15:40
corvuswell, if we wanted to be more cautious, we can set the default to 2.5 and then set "unittests" to use 2.7 or something...15:41
*** ykarel is now known as ykarel|away15:41
*** yamamoto has quit IRC15:42
clarkbfungi: do you know if anyone has debugged the mis applied retired project acls in gerrit?15:42
corvusbut if we want to roll the dice, then we could do so, and if things go wrong, we can just force-merge a change to 'base' which sets it to 2.515:42
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Properly handle missing ansible version in executor  https://review.openstack.org/64425215:43
corvusclarkb: see reply on https://review.openstack.org/64424115:43
clarkbcorvus: thanks /me drinks more tea15:44
fungiclarkb: i do not know, no15:44
clarkbfungi: k, its getting near the top of my list so likely to look at it today if others haven't15:45
fungii'm currently juggling a bunch of things, so happy for you to grab it if you can15:45
clarkbcorvus: mordred: re skopeo I think we probably want to rerun with the no_logs removed then can decide if we want to upgrade skopeo when we have that info15:49
corvusclarkb: or we could go ahead and upgrade skopeo too, then if it happens again, we're 1 step ahead in debugging.... i think we can change 2 variables this time and be okay15:53
mordredclarkb: yah. although I'm also pretty sure we want to upgrade skopeo15:53
mordredwhat corvus said15:53
clarkbmordred: corvus: ok `apt-get update && apt-get install -y skopeo` look right?15:54
mordredclarkb: ++15:54
corvus++15:54
clarkbI'll ask ansible to run that on the executors momentarily15:54
mordredclarkb: you could also ask it to do -m pkg -a name=skopeo,update=yes if you wanted to get fancy15:55
clarkbI used ze01 as my test node and it also installed containers-common containers-golang15:56
clarkbwhich is a depends on skopeo15:57
corvusinteresting...  looks like that's some seccomp rules15:57
corvushopefully it doesn't have a bad interaction with bwrap15:58
corvuser "that" == containers-golang15:58
corvuscontainers-common is also interesting15:59
corvushttp://paste.openstack.org/show/747976/15:59
mordredif there is an adverse reaction, I bet we could go talk to our friends over in podman land15:59
corvusthat seems to be a lot of configuration....16:00
clarkbalso odd considering skopeo is an http transfer tool basically16:00
corvusi... i kinda liked the fact that skopeo was just a stand-alone tool16:00
clarkb(it doesn't run containers)(16:00
corvusi'm not thrilled that it's starting to grow tentacles into system configuration16:00
corvusclarkb: i'm regretting my suggestion that we upgrade skopeo16:01
*** rpioso is now known as rpioso|afk16:01
clarkbso far I've only done it on ze0116:01
corvusclarkb, mordred: the thing i'm now worried about is whether the new version will cause new problems16:02
*** weshay|rover is now known as weshay16:02
*** josephrsandoval has joined #openstack-infra16:02
clarkbbut also if we build new executors tehy would get newer skopeo anyway16:02
clarkb(so this may be a case where dealing with new problems is better problems)16:02
corvuslike, are we going to need to upgrade registries.conf or something to make it work?16:02
corvusclarkb: i agree, i just wasn't expecting such wide-ranging changes16:03
corvusand i'm not sure if we want to deal with them at the same time as our other mysterious problem16:03
clarkb++16:03
*** dpawlik has joined #openstack-infra16:04
mordredcorvus: k. new registries.conf is basically file with default16:04
corvusi mean, i like what skopeo is doing -- it lets you opt-out of the dockerhub-centric registry system16:04
corvusmordred: you're thinking that the new files are configuring it to behave in approximately the same way it was?16:04
mordredyeah16:04
corvusokay.  let's yolo and upgrade them and deal with everything at once :)16:05
mordredthe main change seems to be the default.yaml is configure signatures to go into /var/lib/atomic/sigstore16:05
mordredwhich is a weird location on ubuntu - but the package does create it, so fine16:05
corvusmordred: yeah, we don't do anything explicitly with signatures, so i don't know what that's going to mean for us16:05
corvus(also, how does that interact with perms for an unprivileged tool.  maybe it has nothing to do with skopeo, that's just an intersection with another project atomic tool)16:06
clarkbya this seems to be the common stuff for libcontainers/runc16:06
mordredcorvus: I know that a while back I had an issue with buildah not working out of the box because it was missing a containers.conf file16:06
clarkbunfortunately maybe a bit too common when it comes to skopeo which isn't running containers16:06
corvusclarkb: i'm +1 for upgrading all the ze's when/if you are16:06
clarkbcorvus: ok I'll run the command on the other executors now16:07
clarkbalso bwrap doesn't do runc/libcontainer right? so should be unaffected16:07
clarkbskopeo is updated on all executors now16:08
*** dpawlik has quit IRC16:08
corvusftr, bwrap appears to still work on ze01 :)16:09
clarkbthis is actually a nice side effect of using a properly standaloen tool like bwrap16:10
corvushow did my change to the push/pull roles fail on an ansible-lint failure in fetch-javascript-output?  http://logs.openstack.org/41/644241/1/check/openstack-zuul-jobs-linters/d0cc006/job-output.txt.gz#_2019-03-18_15_51_03_41837316:10
clarkbwith all the container world changing around it we get consistent behavior16:10
corvusyes i would like unix container tools please :)16:10
corvusohhhhh16:11
clarkbcorvus: I think that means we are missing a required-project and/or role in the path? mordred possible fallout of your changes to fixup js tooling after the bionic switch?16:11
corvusis that fallout from multi-ansible?16:11
clarkboh maybe?16:11
corvusit's about zuul_return, and we add zuul to the tox-linters job to supply that16:11
corvusbut it may have moved16:11
mordredOH!16:11
mordredcorvus: yeah16:12
corvusi'll work on a patch16:12
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64426616:13
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line  https://review.openstack.org/64424116:13
corvuswe're going to need to do that everywhere we run linters on zuul jobs16:14
*** e0ne has quit IRC16:14
*** e0ne has joined #openstack-infra16:19
*** udesale has quit IRC16:21
openstackgerritJames E. Blair proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64427216:23
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Update zuul ansible location  https://review.openstack.org/64427316:24
openstackgerritJames E. Blair proposed opendev/base-jobs master: Update zuul ansible location  https://review.openstack.org/64427416:24
openstackgerritClark Boylan proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64427516:24
corvusclarkb: ^ we're dueling16:25
* fungi has duelling banjos stuck in his head now, thanks16:25
*** mpjetta has quit IRC16:25
openstackgerritClark Boylan proposed opendev/base-jobs master: Update zuul ansible location  https://review.openstack.org/64427616:25
clarkbcorvus: oh derp I'll abandon mine16:25
openstackgerritJames E. Blair proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location  https://review.openstack.org/64427716:26
corvusthat's all that come to mind off the top of my head16:26
clarkbI checked system config and we don't do anything like that there16:27
clarkbso ya that lost looks good to me16:27
AJaegercorvus: yeah, I think you hit them all... Thanks16:27
corvus(those all share a change id, btw, so they're asy to find)16:27
*** mpjetta has joined #openstack-infra16:27
corvuss/asy/easy/16:27
*** diablo_rojo has joined #openstack-infra16:33
*** ginopc has quit IRC16:33
openstackgerritMerged openstack-infra/zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64426616:34
*** ginopc has joined #openstack-infra16:34
*** wolverineav has joined #openstack-infra16:35
*** dpawlik has joined #openstack-infra16:36
clarkbinfra-root if we can get a second reviewer on https://review.openstack.org/#/c/643713/1 and its parent I'm happy to watch those go in today and debug/fix any problems that might come up16:37
clarkbthis is more puppet4 upgrades16:37
AJaegercorvus: Just got two -1 for the linters jobs ;(16:38
corvusi guess that's what i get for pushing them all before one of them finishes :)16:38
clarkbwell the zuul-jobs fix did fix it16:39
openstackgerritMerged openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line  https://review.openstack.org/64424116:39
corvusclarkb, AJaeger: do we install zuul from source in zuul-jobs, but from pypi in the other repos?16:41
clarkbI'll recheck the logos change now to get data on ^16:41
AJaegercorvus: we should install from zuul - if not, let's fix...16:42
*** dpawlik has quit IRC16:42
AJaegersorry, no time right now to dig into this, corvus16:42
openstackgerritJohn Studarus proposed openstack-infra/irc-meetings master: moving time up with US daylight savings time change  https://review.openstack.org/64428016:43
corvusi don't think that's it... openstack-zuul-jobs run openstack-zuul-jobs-linters, same as zuul-jobs16:43
corvusso they're both running the same job which installs zuul from source16:43
*** gfidente has quit IRC16:44
*** Lucas_Gray has quit IRC16:47
corvusoh16:50
corvushttp://logs.openstack.org/74/644274/1/check/openstack-zuul-jobs-linters/4926b66/job-output.txt.gz#_2019-03-18_16_33_59_55080516:50
corvusmordred, clarkb: ^ that's the problem16:50
corvusthere's no setup.cfg in the other repos, so we're not installing zuul as a tox sibling16:50
*** armax has joined #openstack-infra16:51
mordredclarkb: fire at will on those puppet changes16:51
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424716:51
mordredcorvus: oh. hrm.16:51
corvuser... well, there *is* a setup.cfgi n openstack-zuul-jobs16:51
*** wolverineav has quit IRC16:52
corvusand that did install zuul as a sibling16:52
corvusso let's just focus on that one16:52
corvushttp://logs.openstack.org/72/644272/1/check/openstack-zuul-jobs-linters/7674bc8/job-output.txt.gz16:53
*** rossella_s has quit IRC16:53
mordredcorvus: yeah - so we just need a setup.cfg in opendev/base-jobs16:54
corvusmordred: right, but i assume if we did that, we'd just hit the error in openstack-zuul-jobs ^16:54
*** gfidente has joined #openstack-infra16:55
mordredcorvus: zuul_return isn't in zuul/ansible/base16:56
mordredoh - there it is16:56
corvus./zuul/ansible/base/actiongeneral/zuul_return.py16:56
mordrednevermind16:56
mordredyeah16:56
*** josephrsandoval has quit IRC16:57
*** rossella_s has joined #openstack-infra16:57
*** wolverineav has joined #openstack-infra16:59
*** pgaxatte has quit IRC17:00
*** mgoddard has quit IRC17:02
corvusi'm out of ideas17:03
*** trown is now known as trown|lunch17:03
* clarkb catches up after eating some breakfast17:06
*** mgoddard has joined #openstack-infra17:08
clarkbansible==2.3.3.0,ansible-lint==3.5.1 is it possible we want a newer version of ansible in that env?17:09
corvuswow.  presumably we want 2.7 now.  but we did have 2.5.15 in the change that worked17:10
clarkbpreviously we would've installed ansible 2.5 via the zuul requirements but now with multi ansible that is deferred17:10
AJaegertest-requirements says "ansible < 2.4"17:10
AJaegerthat needs manual update when zuul changes ansible version17:10
corvusit's different in zuul-jobs: ansible>=2.5.1,<2.617:11
*** dpawlik has joined #openstack-infra17:11
corvusit's difficult for me to see how that would affect this, but let's go ahead and change openstack-zuul-jobs to match17:12
clarkb++17:12
clarkbthere appears to be a bug in the skopeo changes http://logs.openstack.org/22/643622/1/check/opendev-buildset-registry/1e76abc/job-output.txt.gz#_2019-03-18_17_08_19_80404417:12
clarkbI'm looking at ^ now17:13
openstackgerritJames E. Blair proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64427217:13
corvusclarkb: doh.  that's my mistake.  will fix17:13
clarkbcorvus: ok thanks17:13
*** e0ne has quit IRC17:14
*** armax has quit IRC17:15
*** dpawlik has quit IRC17:16
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: fix proxy_port error in push/pull roles  https://review.openstack.org/64432217:16
corvusclarkb: ^17:16
clarkbcorvus: I went ahead and approved it since its a straightfoward config change via removal17:17
corvus++17:17
*** ykarel|away has quit IRC17:18
*** snapiri has quit IRC17:19
clarkbas a general reminder to the channel I'll be driving to and from seattle tomorrow. I expect to run the meeting as I should be there well before 1900UTC. But I may be afk for good chunks of the early and late day17:19
fungithanks for the heads up!17:21
*** mpjetta has quit IRC17:22
*** kopecmartin is now known as kopecmartin|off17:22
* clarkb puts together a meeting agenda and remembers to send it an hour later today :)17:24
clarkbI think that was why ianw missed the cutoff. I was operating on local time and sent it at 1800UTC instead of 1900UTC. Sorry17:24
*** rpioso|afk is now known as rpioso17:25
* fungi shakes fist at arbitrary time changes17:25
mordredas a similar reminder to the channel - I'll be basically out tues-thurs. I will probably also work, but I have no idea which hours it might be17:26
*** rossella_s has quit IRC17:26
*** mattw4 has joined #openstack-infra17:29
*** mpjetta has joined #openstack-infra17:30
*** ramishra has quit IRC17:35
*** rossella_s has joined #openstack-infra17:35
openstackgerritMerged openstack-infra/zuul-jobs master: buildset registry: fix proxy_port error in push/pull roles  https://review.openstack.org/64432217:35
*** jamesmcarthur has quit IRC17:38
*** jamesmcarthur has joined #openstack-infra17:39
*** ricolin has quit IRC17:41
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Increase wait time of test_plugins  https://review.openstack.org/64437517:42
*** rpittau is now known as rpittau|afk17:43
openstackgerritMerged openstack-infra/zuul master: Don't call the merger for non-live items  https://review.openstack.org/64370317:44
*** rossella_s has quit IRC17:44
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Update zuul ansible location  https://review.openstack.org/64427217:45
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Add foreground option  https://review.openstack.org/63564917:46
*** bgmccollum has quit IRC17:47
*** rossella_s has joined #openstack-infra17:47
*** rossella_s has quit IRC17:48
*** chandankumar is now known as raukadah17:51
*** bgmccollum has joined #openstack-infra17:52
*** yamamoto has joined #openstack-infra17:53
corvusclarkb: oh hey, i guess it was the ansible version?17:57
corvusclarkb, mordred, AJaeger: i'll update the other patches to (a) have a setup.cfg and/or (b) update ansible version as required17:58
clarkbthe skopeo run on ovh-bhs1 just succeeded17:58
clarkbs/on/again/st17:58
*** gyee has joined #openstack-infra17:58
corvusclarkb: cool, i saw failures from mtl01 and gra117:59
clarkbit wasn't a 100% failure so not sure if fixed but that is a good sign17:59
corvusso it would be good to get a pass from one of those17:59
corvusor even both :)17:59
clarkb++17:59
corvusclarkb: i rechecked 640901 and 94090417:59
corvusthose push to intermediate, and they only run a build job18:00
*** trown|lunch is now known as trown18:00
corvusso should be less wasteful / faster to repeatedly recheck those18:00
clarkbk18:00
corvusinfra-root: would you please review https://review.openstack.org/643954 and https://review.openstack.org/643957 which are needed for the zuul restart with multi-ansible ?18:01
corvusi've verified the lest failure on the second is unrelated puppet stuff (elasticsearch)18:01
corvuss/lest/test/18:01
*** armax has joined #openstack-infra18:02
corvuslol one of the ansible fixes was actually an irc-access failure18:03
clarkbcorvus: left a note about explicit ordering on the second change. I don't think that is strictly required though18:04
corvusclarkb: yeah, good point.  i think as long as we aren't auto-restarting it, it doesn't matter?18:05
clarkbcorvus: ya18:05
clarkbit would be more for humans understanding how the pieces go together probably18:05
mordredcorvus: done18:06
*** sthussey has joined #openstack-infra18:06
*** jpena is now known as jpena|off18:06
*** dustinc is now known as dustinc|lunch18:13
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Update zuul ansible location  https://review.openstack.org/64427318:16
*** iurygregory has quit IRC18:17
corvusconfig-core: ^ that's an omnibus fix for 2 things broken in p-c right now18:17
*** jtomasek has quit IRC18:18
openstackgerritJames E. Blair proposed opendev/base-jobs master: Update zuul ansible location  https://review.openstack.org/64427418:18
*** jtomasek has joined #openstack-infra18:19
openstackgerritJames E. Blair proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location  https://review.openstack.org/64427718:22
corvuswow.  i think *every* one of those 5 failures was for a different reason.18:22
corvusclarkb: http://logs.openstack.org/01/640901/2/check/zuul-preview-build-image/5b1e40c/zuul-info/inventory.yaml is a pass on mtl0118:22
*** dtantsur is now known as dtantsur|afk18:23
*** eharney_ has joined #openstack-infra18:23
corvusclarkb, mordred: it's worth considering that the gitea image layers may be larger than the zuul-preview layers18:23
clarkbhttp://logs.openstack.org/01/640901/2/check/zuul-preview-build-image/5b1e40c/job-output.txt.gz#_2019-03-18_18_08_08_142865 is the step that was breaking before18:23
clarkbcorvus: when I looked on friday it seemed like we were talking 10s of megabytes so no huge but not tiny either18:23
corvuswould be cool if skopeo told us how big each layer was18:24
*** bgmccollum has quit IRC18:25
corvusa full gitea image is 451MB, but i don't know the layer breakdown18:25
openstackgerritMerged openstack-infra/zuul master: Whitelist ansible_python_interpreter for add_host  https://review.openstack.org/64372618:25
*** eharney has quit IRC18:25
openstackgerritJames E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image  https://review.openstack.org/64438718:26
corvusclarkb: ^ for rechecks18:27
clarkbcorvus: related to this any idea why https://review.openstack.org/#/c/643623/1 skipped the image builds on check?18:28
clarkbI'm looking at the job config and files is set to docker/gitea/.* which should match taht file. Then the build-image jobs provides: docker-image via the opendev jobs and buildset-registry requires: docker-image so it shoudl'ev run I think18:28
corvusclarkb: are you using the gerrit webui with hideci enabled?18:29
clarkbcorvus: no, if you expand the jobs you'll see they were all skippec18:29
clarkb*skipped18:29
corvusclarkb: well, zuul says: Warning: Requirements ['docker-image'] not met by build 32fae011811144ef9491fc16e54c5cf718:29
openstackgerritMerged openstack-infra/puppet-zuul master: Set fingergw user to zuul  https://review.openstack.org/64395418:29
*** wolverineav has quit IRC18:31
corvusso that's the proximate cause; we'll have to dig deeper to find out why the requirement wasn't met18:31
clarkbbuild there is a zuul build uuid?18:31
corvusya18:31
corvusah, because it was a post failure of the parent change18:32
corvushttp://zuul.openstack.org/builds?uuid=32fae011811144ef9491fc16e54c5cf718:32
clarkbah ok so it was caused by the thing we are trying to fix with the skopeo upgrade18:32
corvusperhaps that should be a failure, not a skip?18:33
*** gmann is now known as gmann_afk18:34
*** rossella_s has joined #openstack-infra18:36
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: Fail jobs when requirements not met  https://review.openstack.org/64439218:37
mordredcorvus: ++ yes - I think that's the right choice18:38
openstackgerritMerged opendev/base-jobs master: Update zuul ansible location  https://review.openstack.org/64427418:38
openstackgerritJames E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image  https://review.openstack.org/64438718:38
*** armax has quit IRC18:40
*** jamesmcarthur has quit IRC18:41
*** gfidente has quit IRC18:42
clarkbhrm jobs still failing on some of those multi ansible file location fixes /me looks18:43
corvusi'm looking at the irc one18:44
corvusproject-config18:44
clarkbok I'll look at https://review.openstack.org/#/c/644277/ (zuul-base-jobs)18:44
clarkblooks like actual linting failures. I can push a new patchset to fix18:44
*** mriedem has quit IRC18:45
corvusclarkb: i tried to fix that by pinning18:45
corvusclarkb: it looks like the pin didn't work?18:45
tobiashso there is already fallout without even restarting an executor?18:46
clarkbcorvus: oh ya it still installed 4.1.018:46
corvustobiash: the ansible lint jobs are not robunt18:46
corvusrobust18:46
tobiashah18:46
corvusi expect that we will remove it entirely soon18:46
clarkbcorvus: I see the issue I'll push a new patchset18:47
corvuswe've spent something like 100 person hours doing nothing more than trying to keep it working the same way it used to over the past week alone.18:47
corvusand we can never upgrade it because it's getting far too opinionated about how to construct roles (so that they can have a high "score" on galaxy)18:48
*** mriedem has joined #openstack-infra18:48
openstackgerritClark Boylan proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location  https://review.openstack.org/64427718:49
*** wolverineav has joined #openstack-infra18:49
clarkbcorvus: ^ we defined the deps in multiple places so I reduced it to a single location18:49
fungiwhat legitimate bugs does ansible lint catch which the zuul parser wouldn't?18:49
fungii feel like not many18:49
fungiand we don't purport to follow any particular stylistic conventions with zuul job playbooks/roles18:50
corvusi can't reproduce the irc error locally: http://logs.openstack.org/73/644273/2/check/project-config-irc-access/12afdda/job-output.txt.gz#_2019-03-18_18_23_45_93068918:51
openstackgerritAnkita Bansal proposed openstack-infra/storyboard master: Project Group UI should permit removing project group even if it is not empty.  https://review.openstack.org/64226618:51
openstackgerritMerged openstack-infra/zuul master: Properly handle missing ansible version in executor  https://review.openstack.org/64425218:53
fungithat does seem like a very bizarre sort order18:53
corvusinput=output is what i get when i run it locally18:54
corvusso even if i move 'branches' to the end, it still passes18:54
fungii can't even imagine something like LC_COLLATE would influence ordering b before p18:55
* mordred has confusion18:55
corvusfungi: well, it's not sorted18:55
*** jamesmcarthur has joined #openstack-infra18:56
*** gmann_afk is now known as gmann18:56
mordredcorvus: when you run it locally, are you running it in tox / with python2.7?18:56
fungipython dict ordering wouldn't have changes between xenial and bionic's python2.7 would it?18:57
corvusmordred: no, in a py3.6 venv18:57
fungiyeah, the job there is running on 2.718:57
corvusoh?18:57
mordrednod. it seems to be running in 2.7 in the gate, which uses the old unordered dict implementation18:57
corvuswhy?18:57
fungithis is a great question18:58
mordredit doesn't make any sense to me looking at the tox.ini file18:59
mordred but http://logs.openstack.org/73/644273/2/check/project-config-irc-access/12afdda/job-output.txt.gz#_2019-03-18_18_20_20_96276718:59
clarkbis there a hard coded shebang?19:00
fungitox itself is running with 2.7 according to the log19:00
mordredwell - if tox was installed with python2 and basepython is not set to the contrary19:00
mordredit'll run under python219:00
*** jamesmcarthur has quit IRC19:00
mordredand basepython is not set in [testenv] in that tox.ini19:00
mordredso maybe we should do that19:01
corvusso probably pyyaml <5.1 used an ordereddict, and pyyaml 5.1 does not, but dicts themselves are ordered19:01
fungiseems to be the case, yes. not set in [testenv:irc] either, so it's going to be 2.719:01
corvusi'll update the patch19:01
mordredalso - why are we installing grafyaml from source in teh grafyaml env?19:02
mordredthat's probably for siblings support from before we had siblings?19:02
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Update zuul ansible location  https://review.openstack.org/64427319:02
corvusmordred: unsure; i defered that but otherwise ^ does that look right?19:03
mordred++19:03
openstackgerritMerged openstack-infra/system-config master: Use OpenDev icon in gitea  https://review.openstack.org/64362219:03
*** dave-mccowan has joined #openstack-infra19:05
mordredwell - it turns out it only passes with master of grafyaml19:05
*** dpawlik has joined #openstack-infra19:05
mordredso if we want to use it from release, we'd need to cut a grafyaml release19:05
*** dpawlik has quit IRC19:10
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424719:11
clarkbgrr now hacking/flake8 fail on zuul-base-jobs looks like an issue with older hacking against newer flake8. I'm testing if newer hacking works locally19:12
*** sshnaidm|rover is now known as sshnaidm|afk19:13
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424719:14
openstackgerritClark Boylan proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location  https://review.openstack.org/64427719:15
*** jamesmcarthur has joined #openstack-infra19:18
*** bgmccollum has joined #openstack-infra19:18
*** jamesmcarthur has quit IRC19:19
*** eharney_ has quit IRC19:20
*** jamesmcarthur has joined #openstack-infra19:23
*** dustinc|lunch is now known as dustinc19:23
*** wolverineav has quit IRC19:25
clarkbwoo ^ passed tests and is on its way to the gate. I think that is all of them now. /me sends meeting agenda and finds lunch19:26
mordredclarkb: \o/19:28
*** armax has joined #openstack-infra19:29
openstackgerritMerged openstack-infra/project-config master: Update zuul ansible location  https://review.openstack.org/64427319:31
openstackgerritMerged openstack-infra/zuul-base-jobs master: Update zuul ansible location  https://review.openstack.org/64427719:32
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424719:39
*** imacdonn_ has joined #openstack-infra19:39
*** imacdonn_ is now known as imacdonn19:40
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424719:40
*** dpawlik has joined #openstack-infra19:40
mordredfungi: feel like a git-review? https://review.openstack.org/#/c/643966/19:43
*** wolverineav has joined #openstack-infra19:44
fungithanks for the reminder. so far my morning has been like a freight train but i have a couple minutes to look at something now19:45
*** dpawlik has quit IRC19:45
clarkbthe opendev logo looks to be applied more broadly on gitea now19:46
clarkbcorvus: fyi ^ and thanks19:46
*** wolverineav has quit IRC19:48
*** wolverineav has joined #openstack-infra19:48
clarkbmordred: corvus the last thing I notice is in mobile the search bar is still that gitea green19:49
clarkbAfter lunch I'll see if I can figure out how ti make it opendev purple/pink19:49
mordredclarkb: it's not green for me on mobile?19:50
mordredclarkb: oh - well - the URL title bar is - maybe that's the thing you mean?19:51
mordredso yeah - we should find that setting19:51
clarkbyea19:51
fungii like how "opendev" is now the name of a purply-pink color19:53
*** whoami-rajat has quit IRC19:55
openstackgerritJames E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image  https://review.openstack.org/64438719:58
*** jamesmcarthur has quit IRC20:00
corvusclarkb: https://review.openstack.org/642845 is a simple ozj change20:00
fungiclarkb: is there a flag to switch to "mobile view" or is that only decided automagically? i can't find a ui widget nor do there seem to be selectable alternative stylesheets for it20:00
corvusfungi: i think to see this you must actually be on a phone20:01
*** tram- has joined #openstack-infra20:02
*** tram- has left #openstack-infra20:02
fungiweird20:03
corvuseven firefox's web tools don't show it20:03
*** pcrews has quit IRC20:07
clarkbas an alternative we can remove the coloring of the url bar aand use defauly20:08
*** jamesmcarthur has joined #openstack-infra20:09
*** wolverineav has quit IRC20:09
*** wolverineav has joined #openstack-infra20:10
*** jamesmcarthur has quit IRC20:10
*** jamesmcarthur has joined #openstack-infra20:10
*** irclogbot_1 has joined #openstack-infra20:12
*** dpawlik has joined #openstack-infra20:14
*** dklyle has quit IRC20:14
openstackgerritMerged openstack-infra/zuul master: Increase wait time of test_plugins  https://review.openstack.org/64437520:15
*** wolverineav has quit IRC20:16
*** wolverineav has joined #openstack-infra20:16
*** wolverineav has quit IRC20:16
*** wolverineav has joined #openstack-infra20:17
*** fresta_ has quit IRC20:17
*** dklyle has joined #openstack-infra20:18
*** dpawlik has quit IRC20:18
openstackgerritMerged openstack-infra/system-config master: Upgrade to gitea 1.7.4  https://review.openstack.org/64362320:19
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Use YAML anchors to avoid repeating irrelevant-files  https://review.openstack.org/64284520:19
corvusclarkb: http://logs.openstack.org/87/644387/3/check/system-config-build-image-gitea/e6d9a5d/zuul-info/inventory.yaml gra1 passed20:19
*** eharney_ has joined #openstack-infra20:21
*** Lucas_Gray has joined #openstack-infra20:21
*** irclogbot_1 has quit IRC20:25
*** irclogbot_1 has joined #openstack-infra20:27
*** e0ne has joined #openstack-infra20:28
openstackgerritMerged openstack-infra/git-review master: Use remote_url instead of remote for download  https://review.openstack.org/64396620:30
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests  https://review.openstack.org/64424720:30
*** kgiusti has left #openstack-infra20:36
clarkbcorvus: woot I'm guessing it was a skopeo bug that was fixed20:37
*** dave-mccowan has quit IRC20:39
clarkbmordred: corvus ThemeColorMetaTag is the value to change in gitea's setting/setting.go or templates/base/head.tmpl20:41
*** e0ne has quit IRC20:41
clarkbany idea if the contents of setting.go will read values out of a config file somewhere?20:41
*** dave-mccowan has joined #openstack-infra20:44
clarkbya there seems to be a mapping between setting.go and app.ini UI section20:45
openstackgerritMerged openstack-infra/puppet-zuul master: Update executor ansible on installation  https://review.openstack.org/64395720:48
*** raissa has joined #openstack-infra20:50
*** dpawlik has joined #openstack-infra20:51
clarkbheh ini.AllCapsUnderscore seems to be the magic20:52
clarkbhrm do we not set a custom app.ini yet?20:53
clarkbah it is in ansible not docker20:54
*** pcaruana has quit IRC20:54
*** dpawlik has quit IRC20:55
corvusShrews: https://review.openstack.org/640822 is up your alley :)20:59
*** trown is now known as trown|outtypewww20:59
openstackgerritClark Boylan proposed openstack-infra/system-config master: Set the gitea theme color to match the opendev pink  https://review.openstack.org/64442021:01
clarkbI think ^ should work to change that green color21:01
Shrewscorvus: +321:06
*** Lucas_Gray has quit IRC21:06
*** rh-jelabarre has quit IRC21:07
openstackgerritClark Boylan proposed openstack-infra/system-config master: Set the gitea theme color to match the opendev pink  https://review.openstack.org/64442021:08
clarkbcorvus: ^ sorry I forgot to convert it to the AllCapsUnderscore format :/ should be good now21:09
*** priteau has quit IRC21:12
*** armax has quit IRC21:13
clarkbcorvus: fyi I'm looking at fixing http://logs.openstack.org/12/643712/1/gate/opendev-buildset-registry/1016e6e/job-output.txt.gz#_2019-03-18_19_28_39_060210 which is a different fallout of changes to the intermediate registry setup21:16
openstackgerritMalek Karray proposed openstack-infra/storyboard-webclient master: Converting a search query into a browse  https://review.openstack.org/64442421:17
*** xek has quit IRC21:17
*** mgoddard has quit IRC21:20
*** raissa has quit IRC21:20
openstackgerritClark Boylan proposed openstack-infra/zuul-jobs master: Handle no metadata in zuul artifact items  https://review.openstack.org/64442821:23
clarkbcorvus: ^ I think that should fix it21:23
*** irclogbot_1 has quit IRC21:26
*** irclogbot_1 has joined #openstack-infra21:28
clarkbinfra-root I'm going to run `manage-projects -d openstack/deb-python-oslotest` on review.o.o to see if I can see why that project doesn't have the retired.config acl applied21:30
openstackgerritMerged openstack-infra/nodepool master: config: use yaml.safe_load instead of load  https://review.openstack.org/64384921:35
clarkbthat seems to do a bunch of nothing21:36
openstackgerritMerged openstack-infra/system-config master: Add nodepool_restart playbook  https://review.openstack.org/64082221:37
clarkbhttps://review.openstack.org/#/c/506219/2 is teh change that broke setting of retired acls21:55
clarkbthe configs_list property was updated to return only unretired projects and that is the list we iterate over to set project configs in manage_projects21:55
clarkbI updated manage_projects.py on review01 to print out the projects it was iterating over and none of the retired projects show up21:56
clarkb(I've since reverted that and `diff /opt/jeepyb/jeepyb/cmd/manage_projects.py /usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py` shows no delta so should be back to normal now21:56
fungiaha, chicken-and-egg21:57
fungiwe want to exclude retired projects from lots of things, but not from the routine which retires them ;)21:57
clarkbya I think we want manage projects to iterate over a complete list21:57
clarkband then most everything else can use the smaller list21:58
clarkbI'll write up a change but I probably won't run it soon since I'll be doing that driving thing tomorrow21:58
*** armax has joined #openstack-infra22:03
*** rfolco has quit IRC22:04
openstackgerritClark Boylan proposed openstack-infra/jeepyb master: Inspect all configs in manage-projects  https://review.openstack.org/64443222:04
clarkbwe may want to run that by hand and not with puppet as it may take some time to build out and configure all of the retired projects22:05
*** dpawlik has joined #openstack-infra22:05
*** jtomasek has quit IRC22:05
*** dpawlik has quit IRC22:10
corvusclarkb: hrm... why was there no metadata?22:14
clarkbcorvus: the change it failed on was the puppet4 enablement change in our groups.yaml file so I think it was because there were no image artifacts22:15
corvusclarkb: i see artifacts in http://logs.openstack.org/12/643712/1/gate/opendev-buildset-registry/1016e6e/zuul-info/inventory.yaml22:16
clarkboh huh and they don't have a metadata entry22:16
clarkbso maybe my fix isn't the right option there22:17
corvusit looks like our upgrade gitea to 1.7.4 change ran after the parent change (643622) merged, so we didn't see that exercised earlier22:18
corvusthose artifacts came from the 'upload' job vs the 'build' job, and this may be the first time that happened... they should be the same as far as this is concerned, maybe there's a difference.22:20
clarkbpush-to-intermediate-registry is what seems to set the metadata field22:21
corvusyeah, and i can't see a way for it not to do that22:21
corvusso maybe there's a bug in the artifact collection from a live item22:21
clarkbya just double checked the inheritance and we should run the push to intermediate role prior to uploading to docker because the buildset registry base job is parent of the upload to docker job22:24
clarkbI'm going to recheck that change again as there aren't any image builds in the pipeline that I know of and that should get the puppet4 change in22:24
corvusclarkb: ++22:24
corvusi'll look into whether this is a zuul bug22:24
*** kjackal_v2 has quit IRC22:30
clarkb#status Log modified jeepyb files on review01.o.o to debug why manage-projects isn't setting retired project acls. I have since restored those file contents and `diff /opt/jeepyb/jeepyb/cmd/manage_projects.py /usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py` shows no delta.22:31
openstackstatusclarkb: finished logging22:31
openstackgerritMerged openstack-infra/zuul master: Wait for command processor thread to stop  https://review.openstack.org/64185422:42
ianwdid we always need to set "OS_VOLUME_API_VERSION=1" when chatting to rax volume manager?  i feel like that's new, but haven't done it in a while ...22:52
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Fix metadata not passed in provides/requires  https://review.openstack.org/64444522:52
corvusclarkb: ^ definitely a bug22:52
corvuslet me rebase that22:53
clarkbianw: if I had to guess upstream tooling now defaults to version 2, but rax only has version 122:54
clarkbianw: so ya likely new22:54
*** tkajinam has joined #openstack-infra22:54
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Fix metadata not passed in provides/requires  https://review.openstack.org/64444522:54
*** wolverineav has quit IRC22:57
ianwclarkb: still need cc655f1e-2ce6-4657-b9c1-792a5861e97a | clarkb-test-afs01-ord-upgrade ?22:57
*** wolverineav has joined #openstack-infra22:57
clarkbianw: nope22:58
clarkbafs seems stable after the upgrades, that one can go away22:58
clarkbianw: are you volunteering to clear it out or should I get it?22:58
clarkbcorvus: fix lgtm22:59
ianwclarkb: i will, poking around in there doing this backup rotation22:59
*** armax has quit IRC22:59
ianwdone23:00
clarkbthanks23:00
*** elod has quit IRC23:01
clarkbcorvus: if you have a moment can you review https://review.openstack.org/643671 and https://review.openstack.org/#/c/644420/ two small gitea fixups23:03
*** wolverineav has quit IRC23:03
*** armax has joined #openstack-infra23:04
corvusyep!23:04
*** rcernin has joined #openstack-infra23:04
*** fresta has joined #openstack-infra23:05
corvusMar 18 22:57:30 ze01 puppet-user[27467]: (/Stage[main]/Zuul::Executor/Exec[zuul_manage_ansible]) Triggered 'refresh' from 1 events23:06
corvusthat happened ^23:06
corvusit didn't log any errors23:06
*** rkukura_ has joined #openstack-infra23:06
*** rkukura has quit IRC23:06
*** rkukura_ is now known as rkukura23:06
corvusif it supplied any output though, i don't know where to find it23:07
corvusanyone know where puppet exec output goes these days?23:08
corvuswell, it seems to have worked23:10
clarkbI think puppet logs it to syslog iff rc >023:10
clarkbotherwise it may not record it23:11
corvusso i think we're okay to restart one of the zuul executors to verify things are okay, then we can restart all of zuul to pick up the new feature23:11
clarkbexciting23:11
corvusi'm at ~eod... should we wait till tomorrow?23:11
clarkbMy ability to help tomorrow is likely lower than normal but I'm fine with that myself23:12
fungii'm good with helping tomorrow but starting to wind down myself for the evening23:17
corvusthat's probably best unless someone is really eager to follow through on it now :)23:18
openstackgerritMerged openstack-infra/system-config master: Update more dev servers to puppet 4  https://review.openstack.org/64371223:18
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Fail jobs when requirements not met  https://review.openstack.org/64439223:19
*** mriedem has quit IRC23:20
openstackgerritMerged openstack-infra/system-config master: Retry gitea repo setting HTTP POSTs  https://review.openstack.org/64367123:25
*** tosky has quit IRC23:48
openstackgerritMerged openstack-infra/system-config master: Set the gitea theme color to match the opendev pink  https://review.openstack.org/64442023:48
*** jamesmcarthur has quit IRC23:51
*** wolverineav has joined #openstack-infra23:54
*** wolverineav has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!