Friday, 2019-01-18

openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add pass-to-parent option for secrets  https://review.openstack.org/63167000:05
*** wolverineav has joined #openstack-infra00:06
fungiclarkb: yeah, launchpadlib and its dependencies are (finally) available through pypi, so dropping the distro packaged versions are likely fine00:06
clarkbI've done some quick sanity checking of the uncategorized failures and i think the bulk of them are related to the various neutron issues we've seen recently00:08
*** efried has joined #openstack-infra00:12
*** rh-jelabarre has joined #openstack-infra00:13
*** wolverin_ has joined #openstack-infra00:13
*** wolverineav has quit IRC00:15
*** eumel8 has quit IRC00:21
*** whoami-rajat has quit IRC00:21
*** hwoarang has quit IRC00:22
*** hwoarang has joined #openstack-infra00:23
*** mudpuppy has quit IRC00:34
*** mudpuppy has joined #openstack-infra00:34
*** ijw has quit IRC00:36
*** rlandy has quit IRC00:37
*** sthussey has quit IRC00:43
dabukalamsuperuser cert seems to have expired00:44
dabukalamnot sure if that's something -infra looks after00:44
*** betherly has joined #openstack-infra00:45
*** betherly has quit IRC00:50
clarkbit is not but wecan let them know00:50
*** ijw has joined #openstack-infra00:52
*** ijw has quit IRC00:56
clarkbdabukalam: https://superuser.openstack.org seems to have a valid cert00:59
ianwi see sdague has moved onto ibm's quantum computing00:59
fungidabukalam: when you say "sueruser" you mean https://superuser.openstack.org/ ? i'm seeing an expiration in april, but it might be a different cert on different cdn endpoints00:59
dabukalamthat's strange00:59
fungiianw: so has mtreinish00:59
ianwthe superposition of devstack being simultaneously constantly broken and not broken will probably serve him well :)01:00
*** slaweq has quit IRC01:00
fungiianw: devstack is neither working nor broken until you look in its logs01:00
dabukalamit's appeared to have fixed itself, yes I was talking about https://superuser.openstack.org/01:00
fungiinteresting01:00
*** slaweq has joined #openstack-infra01:00
*** wolverin_ has quit IRC01:06
*** wolverineav has joined #openstack-infra01:18
*** imacdonn has quit IRC01:20
*** whoami-rajat has joined #openstack-infra01:21
*** smcginnis is now known as smcginnis_away01:38
*** wolverineav has quit IRC01:39
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: config: add playbooks to job.toDict()  https://review.openstack.org/62134301:41
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: config: add tenant.toDict() method and REST endpoint  https://review.openstack.org/62134401:47
*** yikun has joined #openstack-infra01:50
*** kopecmartin|off has quit IRC01:50
*** kopecmartin has joined #openstack-infra01:51
*** zbitter has joined #openstack-infra01:51
*** jrist- has joined #openstack-infra01:53
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: WIP: Add API endpoint to get frozen jobs  https://review.openstack.org/60707701:53
*** lifeless_ has joined #openstack-infra01:55
*** vabada_ has joined #openstack-infra01:55
*** kopecmartin has quit IRC01:57
*** tridde has joined #openstack-infra01:57
*** kopecmartin has joined #openstack-infra01:59
*** mudpuppy has quit IRC02:00
*** jbadiapa has quit IRC02:00
*** dklyle has quit IRC02:00
*** apetrich has quit IRC02:00
*** lifeless has quit IRC02:00
*** larsks has quit IRC02:00
*** vabada has quit IRC02:00
*** iurygregory has quit IRC02:00
*** stevebaker has quit IRC02:00
*** sshnaidm has quit IRC02:00
*** zaneb has quit IRC02:00
*** trident has quit IRC02:00
*** mgoddard has quit IRC02:00
*** pfallenop has quit IRC02:00
*** jento has quit IRC02:00
*** jrist has quit IRC02:00
*** larsks has joined #openstack-infra02:00
*** mgoddard has joined #openstack-infra02:00
*** eernst has joined #openstack-infra02:01
*** pfallenop has joined #openstack-infra02:01
*** eernst has quit IRC02:02
*** openstackgerrit has quit IRC02:02
*** dims has quit IRC02:02
*** dims has joined #openstack-infra02:05
*** sshnaidm has joined #openstack-infra02:05
*** apetrich has joined #openstack-infra02:06
*** jbadiapa has joined #openstack-infra02:07
*** kopecmartin has quit IRC02:09
*** hongbin has joined #openstack-infra02:14
*** betherly has joined #openstack-infra02:17
*** betherly has quit IRC02:21
*** hamzy has joined #openstack-infra02:33
*** zbitter has quit IRC02:37
*** zbitter has joined #openstack-infra02:38
*** hwoarang has quit IRC02:40
*** hwoarang has joined #openstack-infra02:41
*** zbitter has quit IRC02:43
*** zbitter has joined #openstack-infra02:43
*** psachin has joined #openstack-infra02:46
*** zaneb has joined #openstack-infra02:47
*** zbitter has quit IRC02:48
*** betherly has joined #openstack-infra02:48
*** betherly has quit IRC02:52
*** wolverineav has joined #openstack-infra02:59
*** roman_g has quit IRC03:00
*** tridde is now known as trident03:06
*** betherly has joined #openstack-infra03:08
*** ykarel has joined #openstack-infra03:13
*** betherly has quit IRC03:13
*** jento has joined #openstack-infra03:26
*** HenryG_ has quit IRC03:28
*** betherly has joined #openstack-infra03:28
*** bobh has joined #openstack-infra03:28
*** HenryG has joined #openstack-infra03:31
*** rh-jelabarre has quit IRC03:33
*** betherly has quit IRC03:34
*** wolverineav has quit IRC03:38
*** hwoarang has quit IRC03:39
*** hwoarang has joined #openstack-infra03:39
*** wolverineav has joined #openstack-infra03:40
*** wolverineav has quit IRC03:40
*** zul has quit IRC03:42
*** udesale has joined #openstack-infra03:47
*** betherly has joined #openstack-infra03:49
*** diablo_rojo has quit IRC03:53
*** betherly has quit IRC03:53
*** ramishra has joined #openstack-infra04:04
*** jamesmcarthur has joined #openstack-infra04:05
*** bobh has quit IRC04:05
*** betherly has joined #openstack-infra04:09
*** betherly has quit IRC04:14
*** hwoarang has quit IRC04:16
*** anteaya has quit IRC04:17
*** hwoarang has joined #openstack-infra04:18
*** udesale has quit IRC04:26
*** betherly has joined #openstack-infra04:30
*** udesale has joined #openstack-infra04:32
*** betherly has quit IRC04:35
*** ramishra has quit IRC04:38
*** udesale has quit IRC04:38
*** jamesmcarthur has quit IRC04:39
*** ramishra has joined #openstack-infra04:39
*** Bhujay has joined #openstack-infra04:40
*** Bhujay has quit IRC04:41
*** Bhujay has joined #openstack-infra04:41
*** lpetrut has joined #openstack-infra04:42
*** hwoarang has quit IRC04:54
*** hwoarang has joined #openstack-infra04:55
*** tkajinam__ has joined #openstack-infra04:58
*** tkajinam_ has quit IRC05:00
*** hongbin has quit IRC05:06
*** lpetrut has quit IRC05:15
*** psachin has quit IRC05:23
*** hwoarang has quit IRC05:29
*** hwoarang has joined #openstack-infra05:29
*** wolverineav has joined #openstack-infra05:30
*** wolverineav has quit IRC05:37
*** lpetrut has joined #openstack-infra06:04
*** lpetrut has quit IRC06:14
*** quiquell|off is now known as quiquell06:18
*** kjackal has joined #openstack-infra06:23
*** jaosorior has joined #openstack-infra06:33
*** kopecmartin has joined #openstack-infra06:34
*** hwoarang has quit IRC06:36
*** hwoarang has joined #openstack-infra06:37
*** wolverineav has joined #openstack-infra06:39
*** jamesmcarthur has joined #openstack-infra06:40
*** jamesmcarthur has quit IRC06:44
*** lpetrut has joined #openstack-infra07:02
*** kjackal has quit IRC07:04
*** dpawlik has joined #openstack-infra07:22
*** pcaruana has joined #openstack-infra07:25
*** wolverineav has quit IRC07:28
*** kjackal has joined #openstack-infra07:29
*** pgaxatte has joined #openstack-infra07:30
*** armax has quit IRC07:34
tobiashinfra-root: it looks like we lost the openstackgerrit bot07:40
*** udesale has joined #openstack-infra07:47
*** iurygregory has joined #openstack-infra07:47
*** gfidente has joined #openstack-infra07:50
ianwtobiash: hrm, looking07:52
tobiashianw: thanks, but probably not urgent :)07:52
*** ykarel has quit IRC07:54
*** pcaruana has quit IRC07:55
*** pcaruana has joined #openstack-infra07:55
*** jpena|off is now known as jpena07:58
ianw#status log restarted gerritbot as it had dropped out07:58
openstackstatusianw: finished logging07:58
ianwnothing obvious in log07:59
*** rpittau has joined #openstack-infra07:59
*** ykarel has joined #openstack-infra08:00
*** ginopc has joined #openstack-infra08:02
*** jtomasek has joined #openstack-infra08:06
*** ccamacho has joined #openstack-infra08:13
*** ykarel has quit IRC08:17
*** ykarel has joined #openstack-infra08:18
*** tkajinam__ has quit IRC08:18
*** openstackgerrit has joined #openstack-infra08:18
openstackgerritMerged openstack/diskimage-builder master: Make sure $TMP_BUILD_DIR/mnt is owned by root  https://review.openstack.org/62974408:18
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider  https://review.openstack.org/59033508:20
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route  https://review.openstack.org/63003508:22
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: Add prepare-workspace-openshift role  https://review.openstack.org/63140208:29
*** ginopc has quit IRC08:29
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Implement zookeeper-auth  https://review.openstack.org/61915608:30
*** kjackal has quit IRC08:31
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Fix noop job toDict()  https://review.openstack.org/63040908:32
*** wolverineav has joined #openstack-infra08:32
*** dosaboy has quit IRC08:34
*** vultaire has quit IRC08:34
*** wolverineav has quit IRC08:36
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering  https://review.openstack.org/53786908:37
*** ginopc has joined #openstack-infra08:37
*** tosky has joined #openstack-infra08:38
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: wip: add status graph rendering  https://review.openstack.org/61000108:45
*** jpich has joined #openstack-infra08:53
*** e0ne has joined #openstack-infra08:55
tobias-urdininfra-root requesting reviews on https://review.openstack.org/#/c/631194/ so we can test requeing one of the failed release jobs, then hopefully fix all of them08:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: docs: add job's logs guide  https://review.openstack.org/58292108:56
*** adriancz has quit IRC08:57
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route  https://review.openstack.org/63003509:03
*** roman_g has joined #openstack-infra09:04
*** dosaboy has joined #openstack-infra09:10
*** dosaboy has quit IRC09:15
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: wip: add status graph rendering  https://review.openstack.org/61000109:15
*** dosaboy has joined #openstack-infra09:17
*** kjackal has joined #openstack-infra09:21
*** xek has joined #openstack-infra09:28
*** panda|off is now known as panda09:30
*** derekh has joined #openstack-infra09:34
openstackgerritFrank Kloeker proposed openstack-infra/project-config master: Add translation job for four-opens project  https://review.openstack.org/63173909:34
*** dtantsur|afk is now known as dtantsur09:35
openstackgerritMerged openstack-infra/zuul master: Add action to host results in zuul_json callback  https://review.openstack.org/63062209:43
*** jpich has quit IRC09:50
*** jpich has joined #openstack-infra09:51
openstackgerritMerged openstack-infra/zuul-jobs master: Fix upload-puppetforge role  https://review.openstack.org/63119409:51
*** jpich has quit IRC09:55
openstackgerritMerged openstack/diskimage-builder master: Change phase to check for dracut-regenerate in iscsi-boot  https://review.openstack.org/62794909:57
*** annp_ has quit IRC09:59
*** lpetrut has quit IRC10:08
*** dtantsur is now known as dtantsur|brb10:08
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Add Kata tenant Zuul configuration repository  https://review.openstack.org/63103210:20
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Define a specific Zuul tenant for Kata containers  https://review.openstack.org/63099410:20
*** eumel8 has joined #openstack-infra10:23
*** jpich has joined #openstack-infra10:27
*** Bhujay has quit IRC10:41
*** Bhujay has joined #openstack-infra10:59
*** electrofelix has joined #openstack-infra11:01
*** ykarel is now known as ykarel|lunch11:03
*** kopecmartin has quit IRC11:07
*** yamamoto has joined #openstack-infra11:18
*** yamamoto has quit IRC11:18
ssbarnea|roverclarkb: git-review fixed comment: https://review.openstack.org/#/c/628655/11:27
*** odyssey4me has joined #openstack-infra11:30
*** kopecmartin has joined #openstack-infra11:38
*** yamamoto has joined #openstack-infra11:48
*** yamamoto_ has joined #openstack-infra11:49
*** yamamoto has quit IRC11:53
*** yamamoto_ has quit IRC11:54
*** roman_g has quit IRC11:58
*** jpich has quit IRC11:58
*** jpich has joined #openstack-infra11:59
*** yamamoto has joined #openstack-infra12:02
*** dtantsur|brb is now known as dtantsur12:02
*** yamamoto has quit IRC12:03
*** yamamoto has joined #openstack-infra12:03
*** roman_g has joined #openstack-infra12:05
*** panda is now known as panda|lunch12:14
*** yamamoto has quit IRC12:15
*** yamamoto has joined #openstack-infra12:15
*** quiquell has quit IRC12:19
*** quiquell has joined #openstack-infra12:19
*** e0ne has quit IRC12:33
*** jpena is now known as jpena|lunch12:35
*** e0ne has joined #openstack-infra12:39
*** e0ne has quit IRC12:40
*** e0ne has joined #openstack-infra12:40
*** gfidente has quit IRC12:41
*** TheJulia is now known as needssleep12:42
*** yamamoto has quit IRC12:44
*** rpittau has quit IRC12:46
*** yamamoto has joined #openstack-infra12:48
*** dtantsur is now known as creepy_owlet12:50
*** panda|lunch is now known as panda12:55
*** udesale has quit IRC13:07
*** udesale has joined #openstack-infra13:08
*** gfidente has joined #openstack-infra13:10
*** rh-jelabarre has joined #openstack-infra13:11
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Revert "Revert "Cleanup down ports""  https://review.openstack.org/61437013:13
*** yamamoto has quit IRC13:13
*** bobh has joined #openstack-infra13:15
*** yamamoto has joined #openstack-infra13:15
openstackgerritSorin Sbarnea proposed openstack/gertty master: Add missing aliases for gerrit searches  https://review.openstack.org/61712013:16
*** Bhujay has quit IRC13:16
openstackgerritSorin Sbarnea proposed openstack/gertty master: Adds linting zuul job  https://review.openstack.org/58187113:17
*** bobh has quit IRC13:20
*** agopi has quit IRC13:22
*** rpittau has joined #openstack-infra13:25
*** jpena|lunch is now known as jpena13:26
*** smarcet has joined #openstack-infra13:28
*** kjackal has quit IRC13:28
*** yamamoto has quit IRC13:31
*** yamamoto has joined #openstack-infra13:31
*** trown|outtypewww is now known as trown13:35
*** rlandy has joined #openstack-infra13:36
tbarronI need to fix the legacy-manila-publishimage-generic job https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml#L60613:38
tbarronand perhaps modernize it to zuulv3 (non-legacy).  Is there13:38
tbarrona reason that it's still centrally defined?  Perhaps b/c it publishes to tarballs.o.o ?13:39
tbarronThis job runs infrequently but probably has not run correctly since the auto-conversion of legacy jobs for zuulv313:40
*** zul has joined #openstack-infra13:41
iurygregorytbarron, im moving some publish jobs in ironic to zuulv3 i know the parent job is publish-openstack-artifacts13:41
iurygregoryhttps://review.openstack.org/#/c/630713/13:42
* tbarron looks13:44
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid master: Fixed scope of deploy resource type usage  https://review.openstack.org/63165513:44
*** yamamoto has quit IRC13:45
toskytbarron: you can also check this one for sahara: https://review.openstack.org/#/c/532691/ and https://review.openstack.org/#/c/582676/, which was13:45
*** jamesmcarthur has joined #openstack-infra13:47
*** agopi has joined #openstack-infra13:47
*** xek has quit IRC13:47
*** xek has joined #openstack-infra13:48
tbarroniurygregory: tosky: thanks!13:50
iurygregorytbarron, yw13:50
tosky(I forgot ... was fixed in the second commit and it's regularly working since then)13:51
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid master: Fixed scope of deploy resource type usage  https://review.openstack.org/63165513:51
*** kjackal has joined #openstack-infra13:52
*** Bhujay has joined #openstack-infra13:54
*** wolverineav has joined #openstack-infra13:57
*** eharney has joined #openstack-infra13:58
*** udesale has quit IRC14:01
*** zul has quit IRC14:01
*** wolverineav has quit IRC14:01
*** udesale has joined #openstack-infra14:02
*** boden has joined #openstack-infra14:03
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: List changed files for all commits between refs  https://review.openstack.org/63179714:04
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Revert "Revert "Cleanup down ports""  https://review.openstack.org/61437014:05
*** kjackal has quit IRC14:07
*** kgiusti has joined #openstack-infra14:08
openstackgerritMerged openstack-infra/project-config master: kata: add dco-license jobs to kata projects  https://review.openstack.org/63099814:08
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: List changed files for all commits between refs  https://review.openstack.org/63179714:09
*** jamesmcarthur has quit IRC14:09
*** ykarel|lunch is now known as ykarel|away14:12
*** gfidente has quit IRC14:12
*** owalsh_ has joined #openstack-infra14:13
*** yamamoto has joined #openstack-infra14:13
*** gfidente has joined #openstack-infra14:14
*** yamamoto has quit IRC14:15
*** yamamoto has joined #openstack-infra14:15
*** owalsh has quit IRC14:16
*** mriedem has joined #openstack-infra14:16
*** ykarel|away has quit IRC14:17
*** owalsh_ is now known as owalsh14:18
*** yamamoto has quit IRC14:18
*** nhicher has joined #openstack-infra14:19
*** priteau has joined #openstack-infra14:25
*** lpetrut has joined #openstack-infra14:30
*** jrist- is now known as jrist14:34
*** jcoufal has joined #openstack-infra14:37
*** hongbin has joined #openstack-infra14:42
fungiinfra-root: following up on earlier discussions in here, does anyone object to no longer including infra-core in https://review.openstack.org/#/admin/groups/stackalytics-core since we abandoned the effort to take over hosting of the service? i expect we should also retire https://git.openstack.org/cgit/openstack-infra/puppet-stackalytics/14:50
Shrewsfungi: make it so14:55
*** hongbin has quit IRC14:55
ssbarnea|roverfungi: this woud not prevent people adding you as reviewers.  not sure what retirement would mean, but it makes sense not to confuse users that infra-root could help maintaining it.14:55
*** jtomasek has quit IRC14:56
ssbarnea|roverdid anyone got any replies regarding the source code? i tried to ask on twitter too, total silence.14:56
fungissbarnea|rover: openstack-infra/puppet-stackalytics was our attempt at setting up configuration management for deploying a stackalytics.openstack.org service, and we're not using it, so suggesting we follow typical repository retirement process there to avoid confusion14:56
fungithe openstack/stackalytics repo is, on the other hand, an unofficial project maintained by some mirantis staff14:57
fungiand the infra-core group has no need for review permissions there14:58
fungissbarnea|rover: as for the stackalytics.com source code, there was a response from boris to a tweet mark collier made saying they hadn't gotten around to pushing the updated source code into the public repository yet, but are planning to do so14:59
*** jtomasek has joined #openstack-infra14:59
*** armstrong has joined #openstack-infra15:00
mordredfungi: ++ to that plan - I don't see any reason for us to have core reviewer status there15:00
*** dpawlik has quit IRC15:00
*** dpawlik has joined #openstack-infra15:00
*** jtomasek has quit IRC15:00
*** markmcd has quit IRC15:01
*** jtomasek has joined #openstack-infra15:01
*** smarcet has quit IRC15:05
*** sthussey has joined #openstack-infra15:07
*** dpawlik has quit IRC15:11
*** jamesmcarthur has joined #openstack-infra15:14
openstackgerritSlawek Kaplonski proposed openstack-infra/project-config master: Add neutron-tempest-iptables_hybrid-fedora job to Neutron dashboard  https://review.openstack.org/63180615:14
*** smarcet has joined #openstack-infra15:16
*** xek has quit IRC15:23
*** xek has joined #openstack-infra15:23
dmsimardJust a heads up -- I'm not sure if anyone is using Ansible devel (2.8) but they introduced a changed that broke ara recently: https://storyboard.openstack.org/#!/story/200481315:27
dmsimardI'll have a fix released early next week15:27
*** quiquell is now known as quiquell|off15:30
*** markmcd has joined #openstack-infra15:33
*** smarcet has quit IRC15:36
pabelangerdmsimard: yah, I think the system-config devel job caught it also15:37
*** armax has joined #openstack-infra15:37
*** smarcet has joined #openstack-infra15:38
pabelangerdmsimard: http://logs.openstack.org/60/631360/2/check/system-config-run-base-ansible-devel/a01f024/job-output.txt15:40
pabelangeri think ianw created a bug in ansible/ansible15:41
dmsimardit's not really a bug in ansible15:41
dmsimardthey're changing how things work15:41
dmsimardhttps://github.com/ansible/ansible/commit/afdbb0d9d5bebb91f632f0d4a1364de5393ba17a15:41
dmsimardara will need to support both the old way and the new way (until the old way dies from deprecation)15:42
pabelangerah, so callback plugin in ARA is raising it?15:42
*** efried has quit IRC15:45
dmsimardyes because they've gotten rid of the variable that ara used to retrieve CLI args -- there's a new way to get args now.15:47
*** dklyle has joined #openstack-infra15:47
*** udesale has quit IRC15:48
*** efried has joined #openstack-infra15:51
clarkbfungi: ++ I think it only causes confusion15:54
clarkbinfra-root I have removed my WIP from https://review.openstack.org/#/c/631346/1 to upgrade production gerrit. I'll go and remove the backup files and chown review_site/lib/* to gerrit2:gerrit2 now15:55
clarkbcan you please reviwe that and if all looks good feel free to approve it. I think an approval now will end up in application around 1700UTC which is what I communicated yesterday15:55
*** smarcet has quit IRC15:55
mordredclarkb: looking. and exctied15:56
mordredclarkb: that war and that setup still requires plugins and stuff as jars that get downloaded by puppet, right?15:57
clarkbmordred: yes15:57
mordredclarkb: (thinking towards making the docker image with that war)15:57
clarkbmordred: aiui since we don't change major versions of gerrit the existing plugins we've got should be fine15:58
mordredawesome15:58
*** Bhujay has quit IRC15:58
clarkbhttps://review-dev.openstack.org/monitoring does indeed work15:58
clarkb#status log Cleaned up review.openstack.org:/home/gerrit2/review_site/lib in preparation for the Gerrit 2.13.12 upgrade via https://review.openstack.org/#/c/631346/115:59
openstackstatusclarkb: finished logging15:59
clarkbthen once we've got past that next on my plate is opendev base jobs repo16:00
*** pcaruana has quit IRC16:00
*** iurygregory is now known as iurygregory_mtg16:00
clarkb(I haven't forgotten it but did get distracted by e-r and double checking some gerrit stuff yesterday. Also contractors)16:00
fungiclarkb: approved16:02
*** smarcet has joined #openstack-infra16:03
smarcetfungi: clarkb: mordre: morning guys please review https://review.openstack.org/#/c/631655/ and https://review.openstack.org/#/c/625640/ thx u! :)16:04
*** anteaya has joined #openstack-infra16:07
AJaegerclarkb, want me to retire puppet-storyboard? I can do the steps including announcement...16:07
clarkbAJaeger: ya that may also help reduce confusion16:08
clarkbthanks16:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Retire puppet-storyboard (part 1)  https://review.openstack.org/63181716:13
AJaegerargh, wrong repo ;(((16:13
AJaegerI want stackalytics, not storyboard ;(((16:13
clarkbst tab16:14
fungiyeah, let's not retire puppet-storyboard until mordred has a chance to containerify it16:14
cmurphyi was so confused16:14
* AJaeger needs a weekend...16:15
fungiin that case, i have good news for you16:15
mordred++16:16
*** chandan_kumar is now known as codemonster16:17
AJaegerthanks, fungi! That's a relief ;)16:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Retire puppet-stackalytics (part 1)  https://review.openstack.org/63181716:18
*** lpetrut has quit IRC16:18
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-stackalytics master: Retire pupet-stackalytics  https://review.openstack.org/63181816:21
*** e0ne has quit IRC16:22
corvusgood morning!16:24
clarkbcorvus: hello16:25
clarkb(not sure if you saw my note above but I haven't forgotten the base jobs repo, it is next in line after gerrit war update happening as soon as things merge and puppet runs16:25
*** smarcet has quit IRC16:26
corvusclarkb: no worries; i've actually been wondering if we should call it something else; i'm going to wip it while i sketch something out16:26
clarkbk16:26
*** markmcd has quit IRC16:27
openstackgerritMerged openstack-infra/system-config master: Upgrade review.o.o to gerrit 2.13.12  https://review.openstack.org/63134616:29
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Retire puppet-stackalytics (part 1)  https://review.openstack.org/63181716:29
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Retire puppet-stackalytics (part 3)  https://review.openstack.org/63181916:29
*** iurygregory_mtg is now known as iurygregory16:29
*** smarcet has joined #openstack-infra16:30
clarkbbased on 2.13.12 change merging just now and where we are un current puppet run cycle I think we'll apply the upgrade around 1730UTC16:31
*** iurygregory has quit IRC16:31
corvusclarkb: ok, i think 'base-jobs' is fine.  it's ready when you are :)16:32
AJaegerclarkb: pushed all changes out to retire puppet-stackalytics, see https://review.openstack.org/#/q/topic:retire-puppet-stackalytics16:32
*** pgaxatte has quit IRC16:33
clarkbAJaeger: great I'll take a look after the gerrit upgrade and opendev base jobs change16:34
clarkbAJaeger: enjoy your weekend :)16:34
corvusi think the pass-to-parent stuff is going to really help with the base jobs change16:34
*** zul has joined #openstack-infra16:34
corvusi think it will mean that we can eliminate things like the zuul- and starlingx-specific afs jobs16:35
clarkboh ya you could have different secrets for different tenants in shared base jobs16:35
corvusyep16:35
*** rpittau has quit IRC16:36
mordred++16:37
mordredI'm very excited about the outcome of that16:37
mordredor at least the potential outcome16:37
mordredcorvus: I'm finishing up the bindep/zuul dockerfile stuff (turns out I can't write shell scripts)16:37
mordredcorvus: then will go review that change16:37
corvusmordred: cool, i'm about to push up a rev16:38
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add pass-to-parent option for secrets  https://review.openstack.org/63167016:38
AJaegerthanks, clarkb16:39
*** rlandy is now known as rlandy|brb16:39
*** markmcd has joined #openstack-infra16:43
*** jamesmcarthur has quit IRC16:44
*** jamesmcarthur has joined #openstack-infra16:47
*** mriedem is now known as superdan16:56
*** superdan is now known as mriedem16:56
*** rkukura has joined #openstack-infra16:56
*** efried is now known as fried_rice16:57
*** mriedem is now known as hansmoleman16:57
*** kopecmartin is now known as kopecmartin|off16:57
*** panda is now known as panda|off17:01
clarkbI expect puppet to upgrade gerrit at 17:20UTC based on when the last puppet run ran and how far after the start of the cron job it started17:01
clarkbinfra-root ^ fyi17:01
*** jpena is now known as jpena|away17:01
fungigood deal17:01
*** jpena|away is now known as jpena|off17:01
pabelangerso, with the new ansible playbooks on bridge.o.o, does the emergency file still work or is it another process now?17:02
clarkbpabelanger: emergency file still works but it is now in a different format17:02
clarkbit is a yaml file now17:02
pabelangerclarkb: is that the groups.yaml file with yamlgroup plugin?17:03
clarkbno, that file is what groups our inventory into groups based on globs. The emergency file is /etc/ansible/hosts/emergency.yaml17:03
clarkbworks almost identical to the old file but it has a .yaml suffix and the list of hosts to disable is a yaml list17:04
pabelangerah, I see now17:04
pabelangercool, thanks17:04
clarkbI did my maths wrong I think puppet is running now17:06
*** ramishra has quit IRC17:06
mordredgerrit is upgrading :)17:06
corvusclarkb: right on time :)17:06
clarkbif you are watching the puppet syslog remember we do sometimes see gerrit fail to start according to init but it is just slower than the 5 minute timeout in the init script17:07
clarkb(we are waiting for gerrit to start now according to a ps)17:08
clarkband its done according to puppet17:08
*** codemonster is now known as chandankumar17:08
clarkbgerrit web ui loads for me and reports a 2.13.12 version17:09
corvusmy gertty is back online17:09
clarkb#status log Upgraded review.openstack.org Gerrit to 2.13.12.ourlocalpatches. Please keep an eye on Gerrit for any unexpected behavior but initial indications look good.17:09
openstackstatusclarkb: finished logging17:09
clarkbalso I'm impressed with how well the cron application of puppet lined up with the time I sent in my email. I didn't actually do any modulo math as I figured being within ~half an hour was good enough :)17:11
*** smarcet has quit IRC17:12
corvusi expect such accuracy in the future!17:13
openstackgerritMonty Taylor proposed openstack-infra/bindep master: Add a dockerfile and scripts to make a bindep builder image  https://review.openstack.org/63183517:15
mordredcorvus: ^^ that works to build a zuul image for me locally - although I only wired up a build job (figure it's not so urgent we can't land ptp first)17:16
corvus17294 tasks of replication on the wall... 17294 tasks of replication... take forty-one down, replicate 'em around... 17253 tasks of replication on the wall...17:19
clarkboh good someone is watching that17:19
clarkbI wonder if upstream would be willing to take a patch to make that disableable17:19
corvusyeah, but i have this awful song stuck in my head17:19
clarkbwe can trigger replication ourselves if we want it after a restart17:19
* clarkb scribbles a note that that would make a good infra programming task if anyone really likes java17:20
*** lpetrut has joined #openstack-infra17:20
*** Dobroslaw has quit IRC17:22
openstackgerritMonty Taylor proposed openstack-infra/zuul master: WIP Build zuul containers with dockerfile not pbrx  https://review.openstack.org/63184017:22
mordredcorvus, clarkb :^^ that shows using the bindep image - although it's not complete from a zuul pov (we want to have some service-specific dockerfiles with addition deps and whatnot)17:22
corvusmordred: oh thanks, that makes assemble make much more sense17:23
*** ginopc has quit IRC17:23
mordredtobiash: ^^ on 613835 and 63184017:23
clarkbok I'm going to catch up on breakfast then sit down to review all the things. I've got base jobs and stackytics puppet and image building changes all queueud up17:24
*** jbadiapa has quit IRC17:24
*** jpich has quit IRC17:26
mordredcorvus: yeah - the idea is to have the builder image make a bunch of wheels and a file with the list of needed binary depends, then install-from-source will install them - so that the final image doesn't need dev tools installed17:26
clarkbmordred: ++ also so that you don't need all the intermediate object files laying around17:28
*** rkukura has quit IRC17:29
mordredclarkb: yah. exactly17:29
mordredclarkb: that process produces a 370M zuul container, compared to a 266M base image with pbrx ... but it's also using python:slim which is debian instead of python:alpine as a base image17:31
mordredso - in general, I think it's probably not a terrible increase17:32
*** wolverineav has joined #openstack-infra17:33
clarkbis the base image switch due to libc?17:36
clarkb(just thinking we may want to write down some of these early decisions and why they were made so in 6 months if someone wants to chagne things we can check if it was considered or not)17:36
clarkbmaybe just a QA addendum to our existing spec17:37
*** creepy_owlet is now known as dtantsur|afk17:38
*** wolverineav has quit IRC17:38
mordredclarkb: yeah - tobiash actually ran in to issues with his alpine-based images and switch to ubuntu-based images17:39
mordredclarkb: however - I plan on making a followup patch to that one so that we make a few different bindep base images matches the python base images17:40
mordredclarkb: so a bindep:slim, bindep:alpine, etc - since I think it won't be terribly onerous to support that17:40
mordredthat way someone who wants to build an image from a pbr/bindep using project can pick any reasonable target base image17:41
mordredclarkb: although - we might also want a bindep base image that doesn't do any python things but JUST does the bindep collection and output17:42
mordredclarkb: so that people could use it for rust/go/c++/java/whatever17:42
clarkbmordred: one thing that makes that weird is bindep runs in the context of the base distro17:42
clarkbso you'd need bindep for all the base distros17:42
clarkb(not a bad idea, just morework)17:42
mordredclarkb: yeah. that's why it's sticking to the python images for now17:42
mordredyeah17:42
*** armax has quit IRC17:44
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: docker: add ability to restrict repository names  https://review.openstack.org/63184817:45
corvusmordred, clarkb: ^ one more piece of the image building puzzle17:45
ssbarnea|roverclarkb: thanks for the gerrit upgrade! when to expect the next? :D17:46
clarkbssbarnea|rover: The next will likely be in a while. Its a bit more involved and I think mordred wants to do a transition to the docker image first17:47
*** lpetrut has quit IRC17:47
clarkb(whcih simplifies a lot of the put gerrit war and plugins and jar libs in place whcih is what I spent most of my time sorting out for this upgrade)17:47
ssbarnea|roveri wanted to ask why we do not sync production database to gerrit-dev as its size is an important fact for testing a new system and an upgrade.17:48
corvusthat's effectively a non-overrideable job variable, so we can create an "openstack-docker-image-upload" job which allows any openstack project to upload to dockerhub using a shared job, and we don't have to write a playbook for each one.17:48
*** rlandy|brb is now known as rlandy17:48
corvus(or, rather, we don't have to write a separate playbook for "openstack", "starlingx", "opendev", "zuul"...17:49
clarkbssbarnea|rover: not all of the data in the db is safe to sync to what we treat as a dev instance17:49
ssbarnea|roverwell, database upgrade and the reviewdb stuff could introduce some nasty surprises. but i like the idea of container, also we already have an official one.17:49
pabelangermordred: oh, are we moving away from pbrx to native dockerfiles?17:49
clarkbssbarnea|rover: but we usually do test the upgrade locally with the real db17:49
mordredcorvus: brilliant17:50
clarkbssbarnea|rover: and starting with the next version there is no db17:50
clarkbor rather an almost inconsequential db17:50
clarkbso investing in that now seems like a bad idea17:50
mordredpabelanger: yah - as we started doing more with the dockers, it became clear how many parts of the ecosystem expect and can handle dockerfiles - and combining that with multi-stage dockerfiles we can pretty much do the same thing pbrx was doing17:51
fungiclarkb: well, there is a db of consequence, it's just not a relational db17:51
corvusclarkb: i love the asterisk there: There is No DB* (except for the little bit of db that there is).17:51
ssbarnea|roverclarkb: is there a way for me to get a dump/rsync of current gerrit and test upgrade locally? i am curious how it copes at scale.17:51
mordredpabelanger: I want to get the bindep base image usable by s2i as well, so that people using openshift with s2i builder can use it successfully as well17:51
fungicorvus: and also the massive db they built inside git ;)17:51
clarkbssbarnea|rover: we will need to scrub it but that is probably something that can be arranged17:51
pabelangermordred: ack17:52
mordredpabelanger: there's a little more work involved there still - and since we're not using s2i I deferred it for now - but I tried to think ahead to make that suck less17:52
ssbarnea|roverat the moment i am playing with official gerrit container, got it running but I want to bypass the initial setup wizard and have it fully running from start.17:52
pabelangeryah, I've only used it once, in some openshift training. But that was a while ago, I am sure it is way different not17:52
clarkbssbarnea|rover: corvus got that set up with docker compose in the zuul quickstart17:53
clarkbssbarnea|rover: I don't know how viable it is for a production gerrit, I think it may rely on a feature that isn't safe for that?17:53
mordredpabelanger: yah. I like the design overall17:53
mordredssbarnea|rover: we're also publishing this now: https://hub.docker.com/r/opendevorg/gerrit17:54
corvusclarkb, ssbarnea|rover: yeah, i think my quickstart relies on bootstrap access granted via the 'become' auth scheme, which is dev-only.17:54
mordredssbarnea|rover: which is what we'll be using when we upgrade to 2.1517:54
corvusclarkb, ssbarnea|rover: you might be able to use become to bootstrap a system without network access, then switch to a reach auth scheme.  i think that would be fine.17:55
corvuss/reach/real/17:55
fungiso like, start gerrit listening only on the loopback, locally do the steps to create the admin via become, then restart gerrit on a safer config listening on reachable interfaces17:56
corvusya17:56
*** trown is now known as trown|lunch17:56
ssbarnea|rovercorvus: i am really missing a link to the source-code browser on zull quickstart page, https://git.zuul-ci.org/zuul -- is not clickable and maybe I do not want to clone to looks a few files.17:57
fungissbarnea|rover: https://git.zuul-ci.org/cgit/zuul is browseable, https://git.zuul-ci.org/zuul is the clone url17:58
fungiwhen we move to gitea we plan to have them both be the same url instead of different urls17:59
fungibut sure, including a link to browse the source might be helpful17:59
*** derekh has quit IRC18:00
clarkbcorvus: re https://review.openstack.org/#/c/631033/1 we decided then that we didn't want site-base-jobs or similar? I'm fine either way18:00
corvusclarkb: everyone seemed happy with opendev/base-jobs.  i am too, but i'd be equally happy with opendev/site-base-jobs or opendev/zuul-site-config.18:01
corvusmordred: ^ ?18:01
clarkbfor me we'll get used to the name and its something I expect we set up once and then rarely think about after :)18:02
clarkb(I'm just ready to hit approve and wanted to make sure there weren't still questions about the name)18:02
*** ykarel|away has joined #openstack-infra18:04
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add link to git browser in quickstart  https://review.openstack.org/63185518:05
corvusclarkb: afaik, no question18:05
corvusssbarnea|rover: ^^18:06
clarkbok approving now then18:06
corvus(i'm not sure the right way to say "pop one item off the stack and then see the message" in irc :)18:06
ssbarnea|rovercorvus: thanks! i was planning to add it myself but better :D18:07
ssbarnea|roveri wanted to ask if any of you tried to play with molecule for testing ansible roles.18:09
clarkbI think pabelanger has18:10
pabelangeryes, I have molecule job running in zuul.o.o18:10
pabelangerit is overkill, since we have nodepool, but want to keep it running to ensure local users were able to use it18:11
pabelangerssbarnea|rover: http://git.openstack.org/cgit/openstack/ansible-role-nodepool/18:12
pabelangerssbarnea|rover: tox -emolecule18:12
ssbarnea|roveri like it a lot as I can "unit-test" changes made to a role very quickly and locally.18:12
*** jaosorior has quit IRC18:13
pabelangeryes, that is why I added it. even wrote up testinfra validation18:13
ssbarnea|roverpabelanger: i have 3mo old change to add it to tripleo-quickstart, :)18:13
*** ccamacho has quit IRC18:14
pabelangerssbarnea|rover: for ansible-network, we are looking to use molecule as our entry-point, but done use docker, just nodes from nodepool18:15
pabelangers/done/don't18:15
ssbarnea|roverpabelanger: thanks for the link, the testinfra was one area i didn't reach yet.18:15
pabelangerssbarnea|rover: yah, I like it. There is also a lot more in system-config too, I haven't looked too much at it, but know it exists18:17
clarkbthe way corvus built our testing for system-config is to ahve nodesets that get plugged into our inventory as "ns1.openstack.org" etc and we run our actual setup against that18:17
clarkbthen testinfra tests conditions against that18:17
openstackgerritMerged openstack-infra/project-config master: Add opendev/base-jobs repo  https://review.openstack.org/63103318:18
*** priteau has quit IRC18:18
pabelangerclarkb: yah, I am hoping to do the same for ansible-network, just haven't had time to look into it.18:19
pabelangermaybe I'll try and get multinode windmill going first :)18:20
*** wolverineav has joined #openstack-infra18:23
mordredpabelanger: ++18:26
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Retire puppet-stackalytics (part 2b)  https://review.openstack.org/63186118:27
*** wolverineav has quit IRC18:28
*** diablo_rojo has joined #openstack-infra18:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Retire puppet-stackalytics (part 3)  https://review.openstack.org/63181918:30
*** hemna is now known as hemnaaway18:35
*** ykarel|away has quit IRC18:35
*** wolverineav has joined #openstack-infra18:35
*** kjackal has joined #openstack-infra18:39
openstackgerritMonty Taylor proposed openstack-infra/bindep master: Add a dockerfile and scripts to make a bindep builder image  https://review.openstack.org/63183518:41
*** martinkennelly has joined #openstack-infra18:42
openstackgerritMerged openstack-infra/nodepool master: Fix OpenShift driver typo  https://review.openstack.org/63139118:43
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add more testing for pass-to-parent  https://review.openstack.org/63186218:47
*** jamesmcarthur has quit IRC18:50
corvusit's REALLY bright outside.  like some sort of giant ball of fusion in the sky.18:51
corvusi vaguely remember such a thing from my distant youth.18:52
Shrewscorvus: #fakenews18:52
clarkbmordred: I left some comments on https://review.openstack.org/#/c/631835/2 to improve some corner case issues, I don't think they are necessarily deal breakers so I didn't -118:52
corvuscan't be true18:53
clarkbit is a perfect drizzly day here18:53
corvusclarkb, mordred: that's a good point clarkb makes re assemble -- that's actually a "use bindep to build a python package" thing.  but bindep normally doesn't care about python, eh?18:55
corvusbut it's just one extra little script file that happens to live on the bindep image18:57
corvusmaybe it needs a less generic name?18:58
clarkbthat might help clarify its purpose18:58
*** ijw has joined #openstack-infra19:02
*** zul has quit IRC19:03
*** bgmccollum has quit IRC19:08
mordredoh - well - the name 'assemble' is taken from the interface needed by s2i19:13
mordred(which we're not using yet but which I'd like to be able to get these images to be able to be used with)19:13
*** electrofelix has quit IRC19:13
corvusmordred: ah.  perhaps we need a bindep:python image then?  i'm just thinking, what about a c++ app that wants to use s2i+bindep?19:14
corvusmordred: or should we move assemble out of bindep entirely, and just make opendevorg/python-builder ?19:15
mordredcorvus: hrm. that's not a bad idea19:16
clarkbits also specific to a single distro (and bindep exists to try and abstract that away)19:17
corvusmaybe it's reaching a bit in the name of purity, but i guess i'm sensitive to the idea of hamstringing bindep's wider success by having it thought of as just a python thing.19:17
corvusclarkb: yeah, or that19:17
clarkbso saying this is the python thing that builds on top of python docker images (which happen to be a single distro)_19:17
clarkber ^ seems fine19:17
mordredyah19:17
mordredI think making opendevorg/python-builder is a great idea19:18
corvusalso, sticking another dockerfile in system-config is nbd.19:18
mordredand then if we have opendevorg/python-builder:slim and opendevorg/python-builder:alpine19:18
mordredyup19:18
mordredcorvus: by that, you mean it's not BD Wong, right?19:18
corvus*everything* is bd wong.  and bd wong is *everything*.19:18
corvusand your 30 seconds are up19:20
*** trown|lunch is now known as trown19:21
*** bgmccollum has joined #openstack-infra19:27
*** e0ne has joined #openstack-infra19:29
*** bgmccollum has quit IRC19:31
*** wolverineav has quit IRC19:36
*** zul has joined #openstack-infra19:37
mordredcorvus, clarkb: actually - if we make python-builder in system-config - maybe it should just install bindep from pip instead of from source?19:37
*** wolverineav has joined #openstack-infra19:37
clarkbmordred: that should work. We're pretty good about making bindep releases whenever new features or bug fixes get in (since most people consume it that way)19:39
*** wolverineav has quit IRC19:39
mordredyeah19:39
*** wolverineav has joined #openstack-infra19:40
*** bgmccollum has joined #openstack-infra19:48
mordredclarkb: I think I just realized though - that actually we may not need the "install javascript tools" step in the image itself19:54
mordredwe can just run install-js-tools.sh in zuul's dockerfile19:54
*** wolverineav has quit IRC19:56
mordredhrm. although that could pollute the python-builder image's view of what the needed distro packages are19:57
mordredI'm going to leave it for now and figure out how to refactor it out later19:57
*** wolverineav has joined #openstack-infra19:57
corvusmordred: i think i lost thread; what are the next steps?19:59
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add python-builder docker image  https://review.openstack.org/63187820:01
mordredcorvus: ^^20:01
*** bgmccollum has quit IRC20:01
mordredcorvus: I have some initial thoughts about removing the chunk where we install js tools on the builder image, but as they have no negative impact on the resulting real images I think it's fine for now20:02
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add more testing for pass-to-parent  https://review.openstack.org/63186220:02
*** wolverineav has quit IRC20:02
corvusmordred: gotcha20:03
corvusmordred: that needs to be rebased on https://review.openstack.org/631606 and that could use a +3 :)20:06
*** hansmoleman is now known as mriedem20:08
openstackgerritMonty Taylor proposed openstack-infra/zuul master: WIP Build zuul containers with dockerfile not pbrx  https://review.openstack.org/63184020:09
mordredcorvus: I shall now do that20:09
mordredcorvus: there's the zuul patch updated to use the python-builder image instead20:10
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add python-builder docker image  https://review.openstack.org/63187820:11
mordredcorvus: done20:12
clarkbfungi: is storyboard prod ready to be xenialed now that dev server made the switch?20:13
clarkbfungi: the only known bug was with the openid lib that was fixed by pinning?20:14
fungiclarkb: yeah, we could do that next instead of testing the database move on -dev next20:14
clarkbdo you have a preference for one or the other? I'm making sure I'm caught up before you go on vacation20:15
fungigot sideswiped by sending contributor codes and finally just starting to come up for air20:15
corvusi got me a code!20:15
corvusall 8 of me20:16
fungicorvus: good, it worked!20:16
fungiclarkb: either can be done first, doesn't really matter. i also haven't deleted the old storyboard-dev yet, and i note bup is complaining about a missing bup directory or something20:16
fungiclarkb: oh, and there are some puppet errors about database collation i need to look into as well20:16
clarkbfungi: for bup it probably wants to ssh as bup-storyboard-dev0120:16
fungilikely20:16
fungiwhich is fine, we don't really need to be backing this server up, but will need to fix it one way or the other20:17
clarkbwhich is good then we can delete the old backups after a little while (since they are append only these server upgrades are good opportunity to hit reset on thsoe)20:17
*** bgmccollum has joined #openstack-infra20:19
clarkbok I may take a quick stab at cleaning up the bup situation and that monday since I can probably do that without committing to being at my desk all day20:19
fungii'm going to go find myself a very late lunch and then hopefully start back on poking at storyboard20:20
clarkbenjoy20:20
*** bgmccollum has quit IRC20:20
fungithanks!20:21
corvusfungi: i'll join you for lunch, virtually20:22
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Replace javascript install with a hook mechanism  https://review.openstack.org/63188020:26
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Use install-js-tools.sh in Dockerfile  https://review.openstack.org/63188120:26
mordredcorvus, clarkb: ^^ how do those two patches look as an approach to not having js tools built in to the python-builder image but still letting us build zuul images properly?20:27
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Use install-js-tools.sh in Dockerfile  https://review.openstack.org/63188120:29
*** whoami-rajat has quit IRC20:30
*** wolverineav has joined #openstack-infra20:33
*** kopecmartin|off has quit IRC20:33
openstackgerritMerged openstack-infra/system-config master: Rename docker role vars  https://review.openstack.org/63160620:34
*** kopecmartin has joined #openstack-infra20:36
openstackgerritMerged openstack-infra/zuul master: Convert zuul_return into action plugin  https://review.openstack.org/59116820:37
*** wolverineav has quit IRC20:38
*** bgmccollum has joined #openstack-infra20:38
mordredWOW. we wind up senselessly installing python2.7 and python3.5 in the builder image20:44
mordredluckily it's the builder image, so who cares20:44
mordredbut zuul's bindep lists python3-dev (which the docker image doesn't need but bindep can't know that) so we get python3.5-dev from stretch - then when we install the javascript tools, nodejs depends on python-minimal which causes python2.7 to be installed20:45
clarkbwait node depends on python?20:46
*** dpawlik has joined #openstack-infra20:46
mordredyeah20:46
mordredclarkb, fungi, SpamapS: anybody remember off the top of your head how to tell dpkg that a package is installed when it isn't?20:47
clarkbI do not20:47
mordredjamespage: ^^ you know lots of things20:47
*** jcoufal has quit IRC20:48
toskymordred: I think that you need to create a dummy package with equivs20:49
*** jamesmcarthur has joined #openstack-infra20:49
mordredtosky: ugh. that sounds like too much effort20:49
toskyiirc it's really easy20:49
mordredtosky: I thought there was some trick you could to of telling dpkg that a package was installed20:49
toskymordred: apart from manually editing the database? :)20:50
mordredtosky: hehe20:50
mordredthanks - that gave me the right words to google for20:50
clarkbtosky: careful mordred might like that idea20:50
mordredtosky: https://eric.lubow.org/2010/creating-dummy-packages-on-debian/20:50
*** dpawlik has quit IRC20:50
toskymordred: yep, that one20:51
* mordred is thinking of making this builder image make a python3-dev dummy package since it's working on the python:slim base image - which has python and its dev headers installed from source20:52
corvusi wouldn't bat an eye if mordred wrote a perl one-liner to edit the dpkg database :)20:52
mordredcorvus: all tasks in life can be accomplished with a perl one-liner20:52
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Use install-js-tools.sh in Dockerfile  https://review.openstack.org/63188120:53
mordredcorvus, clarkb: ^^ ok - that works now - but would still love feedback on the hook var approach20:53
corvusclarkb: i feel like the zuul unit tests are failing due to host slowness more often.  i have no real data.20:54
openstackgerritMerged openstack-infra/zuul master: Add link to git browser in quickstart  https://review.openstack.org/63185520:54
clarkbmordred: I think your depends on in https://review.openstack.org/#/c/631840/2 is broken (the depends on string is missing)20:54
*** e0ne has quit IRC20:55
clarkbcorvus: http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001849.html that may be related?20:55
clarkbthough I thought we had zk in a tmpfs at this point so it may be some other area of slowness20:56
corvusi'll spot check some providers20:56
*** jamesmcarthur has quit IRC20:57
corvusclarkb: hrm.  3/3 different providers, none of them gra20:57
clarkbcorvus: ok so probably not that issue then20:58
clarkb(which makes sense since we tmpfs)20:58
clarkbmordred: is that preferable to having the zuul dockerfile run the js script before the assemble call?20:58
clarkbmordred: or does the ordering matter enough here that we need to have assmble invoke it after some other stuff20:58
*** martin_ken has joined #openstack-infra20:59
corvusyeah, just looking at it, i'd almost expect that to just be a RUN line21:00
corvusbefore assemble21:00
openstackgerritMonty Taylor proposed openstack-infra/zuul master: WIP Build zuul containers with dockerfile not pbrx  https://review.openstack.org/63184021:02
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Use install-js-tools.sh in Dockerfile  https://review.openstack.org/63188121:02
mordredcorvus, clarkb: so the reason it's not a RUN line is becaues then it would infect the bindep calls to find out what packages need to be installed in the final image21:03
mordredbecause it would be installing packages21:03
clarkbif a package is listed in both bindep and the plugin script then we won't install it properly in the ifnal result?21:04
*** wolverineav has joined #openstack-infra21:04
corvusoh, the 'bindep -b' gives missing packages21:04
mordredright - becuase if we install it via install-js-tools.sh (or via a dep of one of the things we install) before we run assemble - then when bindep -b runs it won't be in the list21:05
mordredyeah21:05
*** wolverineav has quit IRC21:05
mordredI mean - I suppose I could just get rid of the b and have it print all of the package?21:05
*** wolverineav has joined #openstack-infra21:05
clarkbmordred: oh ya then install the full set and let package manager figure out what is redundant21:05
mordredyeah. well - that doesn't seem to be a thing21:06
clarkbI think that should work, it goes against normal bindep documentedp rocess though21:06
clarkboh21:06
mordredah - there it is21:06
mordredbindep -l newline21:06
mordredI think that would work for this images if y'all think that's a better approach than the envvar hook21:07
corvusi kinda do; the envar thing feels a little inside out21:07
clarkbya its more straightforward for people to just do what they need as per normal and let the magic be a small scope21:08
mordredkk. sounds good to me21:08
openstackgerritCristopher Lemus proposed openstack-infra/project-config master: stx-test: Add new repo for test tools  https://review.openstack.org/63188421:15
*** anteaya has quit IRC21:21
*** rlandy has quit IRC21:25
*** wolverineav has quit IRC21:29
*** wolverineav has joined #openstack-infra21:30
*** gfidente has quit IRC21:30
*** wolverineav has quit IRC21:31
*** wolverineav has joined #openstack-infra21:31
openstackgerritMonty Taylor proposed openstack-infra/zuul master: WIP Build zuul containers with dockerfile not pbrx  https://review.openstack.org/63184021:36
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add python-builder docker image  https://review.openstack.org/63187821:36
mordredcorvus, clarkb: ^^ ok - there it is all together and squashed and using RUN in the zuul dockerfile21:36
mordredI also did the thing tosky suggested with building an equiv dummy package for python3-dev and installing it21:37
mordredtosky: thanks!21:37
mordredtosky: you're right - totally easy to do21:37
corvusinfra-root: i think zuul is broken21:37
corvushttps://review.openstack.org/63188421:37
toskyhappy to hear that!21:38
corvusi think the action plugin change needs executor restart21:38
mordredcorvus: looking21:38
corvussee http://logs.openstack.org/84/631884/1/check/openstack-zuul-jobs-linters/6a6c9bb/ara-report/result/94c5a71b-b482-43b7-bb48-5c35ed22d407/21:38
corvusif others concur, i'll try that now.21:38
corvuszuul change in question: https://review.openstack.org/59116821:39
mordredcorvus: let's do it21:39
mordredcorvus: I'm not sure I understand why it needs an executor restart21:40
mordredbut if it fixes it - awesome21:40
corvusmordred: i'm not either21:41
*** agopi has quit IRC21:41
mordredcorvus: oh - wait - I think there needs to be an empty module corresponding to the action plugin21:41
openstackgerritCristopher Lemus proposed openstack-infra/project-config master: stx-test: Add new repo for test tools  https://review.openstack.org/63188421:41
mordredcorvus: which then makes me wonder how the patch passed tests if that is true21:41
corvusmordred: yeah, it should have been tested21:42
corvusmordred: also, let's make sure this wasn't a point-in-time fluke and see if later builds work21:42
*** cristoph has joined #openstack-infra21:43
mordredcorvus: ++21:43
mordredcorvus: fwiw - in the ansible source tree, there is an action plugin for "include_vars" - and a python file in the modules dir that is a no-op21:44
mordredhowever - that file has the doc strings - so it might just be that that's where docstrings go21:44
corvusmmm.21:44
corvusokay, i've seen other jobs succeed21:45
mordredcorvus: cool. so maybe just point-in-time21:45
corvusso i think that may have been a false alarm and that job just happened to catch the moment of replacement.21:45
mordredyeah21:46
*** trown is now known as trown|outtypewww21:47
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add python-builder docker image  https://review.openstack.org/63187821:50
clarkbwe've seen that before21:51
openstackgerritMonty Taylor proposed openstack-infra/zuul master: WIP Build zuul containers with dockerfile not pbrx  https://review.openstack.org/63184021:52
mordredok. now those should both be good - and the zuul Dockerfile is still reasonable and understandable I think21:53
*** dpawlik has joined #openstack-infra21:56
*** priteau has joined #openstack-infra21:56
*** kjackal has quit IRC21:58
mordredclarkb, corvus: I think I'm happy with the python-builder image now21:59
*** mriedem is now known as mriedem_away21:59
mordredthe zuul patch is still WIP obvs - but will get that finished up in just a bit22:00
*** jtomasek has quit IRC22:00
mordredI think there's enough in it though to see usage of the first one22:00
mordredbut I think it's ok to review/land/publish the system-config one22:01
*** kgiusti has left #openstack-infra22:02
*** yboaron has joined #openstack-infra22:05
*** armstrong has quit IRC22:09
*** priteau has quit IRC22:10
*** sean-k-mooney has joined #openstack-infra22:11
sean-k-mooneyo/22:11
sean-k-mooneyis http://logs.openstack.org/25/629025/9/check/openstack-tox-functional-with-sudo/2f51882/job-output.txt.gz#_2019-01-18_20_18_28_219621 new/expected22:11
clarkbsean-k-mooney: it is a known issue with clouds reusing IPs22:13
clarkbperiodically we dig up enough log data to hand over to them and have them clean things up22:14
clarkbthe joys of dogfooding :)22:14
sean-k-mooneyok so i can recheck or no ?22:14
clarkbyes you can recheck22:14
clarkbcloudnull: ^ fyi we may need to clear dup IPs in rax again22:14
sean-k-mooneyclarkb: if it helps it ran on ubuntu-xenial-rax-ord-000194347322:15
clarkbsean-k-mooney: I think taht region info and the public IP, 104.130.195.251, are the pieces of data we collect to help them track it down22:16
sean-k-mooneythis http://logs.openstack.org/25/629025/9/check/openstack-tox-functional-with-sudo/2f51882/job-output.txt.gz#_2019-01-18_20_14_01_559024  ya it useful in general to debug issues from a cloud provider point of view22:17
sean-k-mooneyam is there an elastic-recheck query/bug for this?22:17
clarkbI don't recall /me looks22:18
clarkbsean-k-mooney: http://status.openstack.org/elastic-recheck/index.html#1721093 that one maybe22:18
clarkbhttp://status.openstack.org/elastic-recheck/index.html#1793370 possibly that too. The main signature is the ssh key error though22:19
sean-k-mooneywell better question shoudl i write one/file one if not22:19
sean-k-mooneyya22:19
clarkbyou should, but I think those two may already cover it (you should convince yourself of that)22:19
clarkbbasically what happens is cloud resuses IP on our test node (or our test node is reusing it) then later the other node wins the arp fight22:19
corvusmordred, clarkb: okay, weird -- the same job failed again: http://logs.openstack.org/84/631884/2/check/openstack-zuul-jobs-linters/e6be72c/job-output.txt.gz22:20
mordredcorvus: uhm22:20
*** boden has quit IRC22:20
corvus2 different executors22:21
sean-k-mooneyclarkb: ok ill take a look at both exisitng queries and add it to my list. ill proably get to it on monday but ill take a look breifly tonight.22:21
sean-k-mooneyclarkb: its fine to recheck however right.22:22
clarkbsean-k-mooney: yes it is22:22
sean-k-mooneycool jsut wanted to confim this was not hiting lots of jobs currently22:22
pabelangermordred: corvus: I think we need to update tox.ini to now use the action plugin path, not library22:22
corvusOH!22:23
pabelangerthat is ansible-lint failing to find22:23
corvusit's the job itself :)22:23
mordredahhhhhh22:23
mordredthat makes SO MUCH MORE SENSE22:23
corvusEFRIDAY22:23
* mordred hands pabelanger a plump goose22:23
mordredwhich should not be confused with a glum moose22:23
corvuspabelanger: you wanna fix that?22:24
pabelangersure, give me a few22:24
mordredor a plum papoose22:24
mordredclearly my brain has already hit the weekend22:24
clarkbIm there myself and Im further west22:25
*** xek has quit IRC22:25
clarkbis the issue that ansible-lint needs a hint that the thing is valid and lievs in zuul?22:28
mordredclarkb: yeah22:28
clarkbsean-k-mooney: also thanks for checking. And I'll see if I can manage to follow up with cloudnull next week for cleaning those up. I'm "traveling"22:29
fungi"lunch" obtained, catching up now22:29
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189322:30
sean-k-mooneywell i try not to recheck without rootcauseing first and when i see a staus of  POST_FAILURE i assume there is an infra issue22:31
pabelangerI think that is the fix, we'll know soon22:31
sean-k-mooneythat said its the first time i have see  POST_FAILURE and had logs to look at as typically that hapensif the log server is donw/full22:31
sean-k-mooneyat least in my experience22:32
fungimordred: not sure if anybody answered you yet, but you probably want https://packages.debian.org/equivs22:32
clarkbsean-k-mooney: ya the reason for that is because the failure happens talking to the remote test node but the zuul executor runs fine. So we are able to get the logs from the zuul executor22:32
*** jistr has quit IRC22:32
clarkbsean-k-mooney: if it fails due to the log server being sad then ya, we tend to not get anything22:32
sean-k-mooneywhich is pretty understandable it there is an issue with loging its not surpising if you get no logs22:33
sean-k-mooneywhich is why i was surprised when i did this time22:33
*** jistr has joined #openstack-infra22:33
mordredfungi: tosky hooked me up - https://review.openstack.org/#/c/631878/4/docker/python-builder/Dockerfile lines 20 and 21 and https://review.openstack.org/#/c/631878/4/docker/python-builder/python3-dev.control22:34
mordredfungi: works liek a champ22:34
sean-k-mooneyanyway enjoy your weekend/travels o/22:34
fungiawesome22:34
mordredyah22:34
fungiconsensus on equivs!22:35
mordredfungi: also - that change is ready for review if you're in the mood for such a thing22:35
fungiglad i'm not the only one carrying that obscure knowledge in my noggin22:35
mordrednoggins are good for obscure knowledge storage22:36
fungiknoggin storage22:36
*** wolverineav has quit IRC22:38
*** wolverineav has joined #openstack-infra22:40
*** eharney has quit IRC22:40
*** wolverineav has quit IRC22:45
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189622:47
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189722:48
*** wolverineav has joined #openstack-infra22:48
*** jistr has quit IRC22:49
*** jistr has joined #openstack-infra22:50
*** dpawlik has quit IRC22:56
*** cristoph has quit IRC23:05
clarkbI've approved ^ but now I must go find wurst for dinner23:05
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189623:09
openstackgerritMerged openstack-infra/project-config master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189323:09
fungiWURST!23:11
*** wolverineav has quit IRC23:13
*** wolverineav has joined #openstack-infra23:14
*** wolverineav has quit IRC23:15
*** wolverineav has joined #openstack-infra23:15
fungii have a dedicated sausage grinder as of roughly a month ago, so plan on making some proper wurst here at home once travel settles back down23:21
openstackgerritMerged openstack-infra/zuul-jobs master: Set ANSIBLE_ACTION_PLUGINS for linters  https://review.openstack.org/63189723:24
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route  https://review.openstack.org/63003523:26
openstackgerritMerged openstack-infra/zuul master: Add pass-to-parent option for secrets  https://review.openstack.org/63167023:38
openstackgerritMerged openstack-infra/zuul master: Add more testing for pass-to-parent  https://review.openstack.org/63186223:38
*** martinkennelly has quit IRC23:39
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page  https://review.openstack.org/63004123:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route  https://review.openstack.org/63007823:55
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page  https://review.openstack.org/63007923:55
*** tosky has quit IRC23:55
*** wolverineav has quit IRC23:57
*** wolverineav has joined #openstack-infra23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!