Friday, 2018-05-04

*** stevebaker has joined #openstack-infra00:01
clarkbGerrit is the team name in github maybe it is case sensitive?00:01
clarkbor at least the search may be?00:01
clarkbteams_dict = dict(zip([t.name.lower() for t in teams], teams)) would imply that is the case00:02
clarkb(note the use of lower there)00:02
*** gyee has quit IRC00:02
clarkbI think we are ok here. But that bug should be fixed00:02
clarkb(and double checked confirmed to actually be a bug)00:02
ianwclarkb: umm, if you want to tell me exactly what to watch :)00:03
corvusclarkb: does this mean... that we flip created on all the projects from true to false and false to true on every alternate run of the program?00:04
clarkbcorvus: no I think once it is false it stays false beacuse the get repo won't exception00:04
clarkbianw: /var/log/manage_projects on review01.o.o should be noops unless we add projects and the contents of the file edited in https://review.openstack.org/#/c/566170/ should be what that chagne says (I manually edited the file since review01 was in emergency)00:04
*** markvoelker has quit IRC00:05
clarkbianw: more than happy to finish this up tomorrow if you don't want to deal with it00:05
*** tosky has quit IRC00:05
clarkbbut I better work on a fire otherwise dinner will be late00:05
clarkbcorvus: its definitely buggy I think00:05
clarkbcorvus: we should set created to true if we either manage to create it or it already exists00:05
clarkband we should handle varying case in team names if necessary00:05
clarkbwe don't have a backup of queued gerrit processes so I don't think we broke replication to github00:06
clarkband spot checking a project it lgtm00:06
ianwclarkb: alright i can monitor it.  want me to take it out of emergency?00:06
clarkbianw: I'm good with it if you are00:07
corvusclarkb: okay, so the gerrit-in-team bug causes created-in-github to be set to true the first time through; then the second time through, gerrit-in-team is cached, so created-in-github gets set to false, then stays false.00:07
clarkbcorvus: right00:07
*** jistr has quit IRC00:07
corvusclarkb: okay, so the fix is still to update created-in-github in that method (just like gerrit-in-team is)00:08
corvusclarkb: and then also, fix the case issue for gerrit-in-team00:08
clarkbyes I think so00:08
*** hamzy has joined #openstack-infra00:09
ianwclarkb: done and will monitor00:09
*** jistr has joined #openstack-infra00:13
*** mpeterson has quit IRC00:16
*** dingyichen has joined #openstack-infra00:20
*** mpeterson has joined #openstack-infra00:23
*** ssbarnea_ has quit IRC00:28
*** gouthamr has quit IRC00:30
*** dmellado has quit IRC00:31
*** stevebaker has quit IRC00:33
*** d0ugal has quit IRC00:36
*** gouthamr has joined #openstack-infra00:38
*** dmellado has joined #openstack-infra00:38
*** camunoz has quit IRC00:38
*** stevebaker has joined #openstack-infra00:39
openstackgerritKendall Nelson proposed openstack-infra/project-config master: Add Heat to StoryBoard  https://review.openstack.org/56619300:50
*** edmondsw has joined #openstack-infra00:53
*** gouthamr has quit IRC00:58
*** stevebaker has quit IRC00:59
*** dmellado has quit IRC00:59
*** d0ugal has joined #openstack-infra01:03
openstackgerritKendall Nelson proposed openstack-infra/project-config master: Add Heat to StoryBoard  https://review.openstack.org/56619301:04
*** edmondsw has quit IRC01:05
*** markvoelker has joined #openstack-infra01:05
*** anteaya has quit IRC01:12
*** spsurya has joined #openstack-infra01:14
*** dpawlik has joined #openstack-infra01:15
*** annp has joined #openstack-infra01:16
*** edmondsw has joined #openstack-infra01:18
*** dpawlik has quit IRC01:20
*** gouthamr has joined #openstack-infra01:23
*** stevebaker has joined #openstack-infra01:24
*** dmellado has joined #openstack-infra01:26
*** yamahata has quit IRC01:30
*** gouthamr has quit IRC01:32
*** dmellado has quit IRC01:32
*** stevebaker has quit IRC01:33
*** rosmaita has quit IRC01:35
*** mrjazzercise has quit IRC01:36
*** VW has joined #openstack-infra01:39
*** markvoelker has quit IRC01:40
*** rlandy|bbl is now known as rlandy01:45
*** hongbin has joined #openstack-infra01:51
*** gouthamr has joined #openstack-infra01:56
*** dmellado has joined #openstack-infra01:56
*** stevebaker has joined #openstack-infra01:58
*** dayou_ has joined #openstack-infra02:02
*** r-daneel has quit IRC02:06
*** dmellado has quit IRC02:09
*** stevebaker has quit IRC02:09
*** dmellado has joined #openstack-infra02:13
*** stevebaker has joined #openstack-infra02:13
*** armaan has joined #openstack-infra02:13
*** armaan has quit IRC02:14
*** hongbin has quit IRC02:17
*** edmondsw has quit IRC02:18
*** rcernin has quit IRC02:19
*** rcernin has joined #openstack-infra02:20
*** gouthamr has quit IRC02:22
*** gouthamr has joined #openstack-infra02:23
*** gouthamr has quit IRC02:27
*** gouthamr has joined #openstack-infra02:32
*** ykarel|away has joined #openstack-infra02:36
*** markvoelker has joined #openstack-infra02:37
*** gouthamr has quit IRC02:37
*** stevebaker has quit IRC02:37
*** hongbin has joined #openstack-infra02:37
*** dmellado has quit IRC02:38
*** germs has quit IRC02:39
*** germs has joined #openstack-infra02:40
*** zhangfei has joined #openstack-infra02:40
*** psachin has joined #openstack-infra02:41
*** ykarel|away has quit IRC02:44
*** esberglu has quit IRC02:47
*** yamamoto has quit IRC02:51
*** yamamoto has joined #openstack-infra02:51
*** VW has quit IRC02:51
*** VW has joined #openstack-infra02:52
*** zhangfei has quit IRC02:52
dmsimardclarkb: wow I had never seen this spec before https://specs.openstack.org/openstack-infra/infra-specs/specs/deploy-ci-dashboard.html02:55
*** VW has quit IRC02:57
*** yamamoto has quit IRC02:58
*** zhangfei has joined #openstack-infra03:07
*** markvoelker has quit IRC03:10
*** dave-mccowan has quit IRC03:14
*** dpawlik has joined #openstack-infra03:16
*** dhill_ has quit IRC03:16
*** dpawlik has quit IRC03:21
*** andreas_s has joined #openstack-infra03:30
*** andreas_s has quit IRC03:34
*** felipemonteiro__ has joined #openstack-infra03:36
*** stevebaker has joined #openstack-infra03:37
*** gouthamr has joined #openstack-infra03:38
*** dmellado has joined #openstack-infra03:39
*** felipemonteiro_ has joined #openstack-infra03:39
dmsimardhttp://ci-watch.tintri.com/project?project=nova o_O03:39
*** lpetrut has joined #openstack-infra03:41
*** felipemonteiro__ has quit IRC03:43
*** nicolasbock has quit IRC03:44
*** felipemonteiro_ has quit IRC03:45
*** gouthamr has quit IRC03:52
*** mwarad has joined #openstack-infra03:57
openstackgerritIan Wienand proposed openstack-dev/pbr master: Special case long_description_content_type  https://review.openstack.org/56517703:59
*** yamamoto has joined #openstack-infra03:59
*** rcernin has quit IRC03:59
*** rcernin has joined #openstack-infra04:00
*** edmondsw has joined #openstack-infra04:00
*** germs has quit IRC04:02
*** hongbin has quit IRC04:02
*** germs has joined #openstack-infra04:02
*** germs has quit IRC04:02
*** germs has joined #openstack-infra04:02
*** stevebaker has quit IRC04:03
*** gyankum has joined #openstack-infra04:04
*** dmellado has quit IRC04:04
*** ykarel|away has joined #openstack-infra04:05
*** edmondsw has quit IRC04:05
*** ykarel|away is now known as ykarel04:05
*** yamamoto has quit IRC04:06
*** markvoelker has joined #openstack-infra04:08
*** r-daneel has joined #openstack-infra04:10
*** slaweq has joined #openstack-infra04:11
*** slaweq has quit IRC04:15
*** janki has joined #openstack-infra04:15
*** lpetrut has quit IRC04:21
*** lpetrut has joined #openstack-infra04:25
*** pgadiya has joined #openstack-infra04:27
*** r-daneel_ has joined #openstack-infra04:35
*** r-daneel has quit IRC04:36
*** r-daneel_ is now known as r-daneel04:36
*** dmellado has joined #openstack-infra04:39
*** gouthamr has joined #openstack-infra04:40
*** markvoelker has quit IRC04:41
*** Douhet has quit IRC04:41
*** Douhet has joined #openstack-infra04:42
*** jaosorior has joined #openstack-infra04:42
*** stevebaker has joined #openstack-infra04:43
*** lpetrut has quit IRC04:44
*** gouthamr has quit IRC04:49
*** gouthamr has joined #openstack-infra04:50
*** gouthamr has quit IRC04:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove Bandit from Infrastructure Systems  https://review.openstack.org/56553704:57
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove Bandit from Infrastructure Systems  https://review.openstack.org/56553704:59
*** kjackal has joined #openstack-infra05:06
*** ianw has quit IRC05:06
*** ramishra has joined #openstack-infra05:07
*** krtaylor has quit IRC05:09
*** NobodyCam has quit IRC05:15
*** berendt has quit IRC05:15
*** patrickeast has quit IRC05:15
*** sdake has quit IRC05:15
*** TheJulia has quit IRC05:15
*** pas-ha has quit IRC05:15
*** srwilkers has quit IRC05:15
*** hogepodge has quit IRC05:15
*** vdrok has quit IRC05:15
*** charz has quit IRC05:15
*** udesale has joined #openstack-infra05:15
*** sdake has joined #openstack-infra05:15
*** sdake has quit IRC05:15
*** sdake has joined #openstack-infra05:15
*** charz has joined #openstack-infra05:15
*** srwilkers has joined #openstack-infra05:16
*** TheJulia has joined #openstack-infra05:16
*** hogepodge has joined #openstack-infra05:16
*** berendt_ has joined #openstack-infra05:16
*** vdrok has joined #openstack-infra05:16
*** pas-ha has joined #openstack-infra05:16
*** NobodyCam has joined #openstack-infra05:16
*** dpawlik has joined #openstack-infra05:17
*** dpawlik has quit IRC05:21
*** ramishra has quit IRC05:21
*** lpetrut has joined #openstack-infra05:22
*** lpetrut has quit IRC05:29
*** Gorian has quit IRC05:34
*** iranzo has joined #openstack-infra05:38
*** markvoelker has joined #openstack-infra05:38
*** dhajare has joined #openstack-infra05:40
*** quiquell|off is now known as quiquell05:40
*** Gorian has joined #openstack-infra05:42
*** threestrands has quit IRC05:47
*** slaweq has joined #openstack-infra05:47
openstackgerritMerged openstack-infra/project-config master: Add horizon into required-projects for nodejs jobs for magnum-ui  https://review.openstack.org/56471105:51
*** slaweq has quit IRC05:52
*** kjackal has quit IRC05:53
openstackgerritMerged openstack-infra/project-config master: Trivial: Update pypi url to new url  https://review.openstack.org/56304905:55
*** ianw has joined #openstack-infra05:55
*** ramishra has joined #openstack-infra05:56
*** pgadiya has quit IRC05:57
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/56621706:03
*** hemna_ has quit IRC06:07
*** gfidente has joined #openstack-infra06:08
*** gfidente has joined #openstack-infra06:08
*** markvoelker has quit IRC06:11
*** quiquell is now known as quiquell|afk06:12
*** dayou_ has quit IRC06:20
*** e0ne has joined #openstack-infra06:21
*** pcichy has joined #openstack-infra06:23
*** lpetrut has joined #openstack-infra06:28
*** hashar has joined #openstack-infra06:28
*** dbecker has quit IRC06:30
*** dmsimard has quit IRC06:39
*** dmsimard has joined #openstack-infra06:39
*** yolanda_ has joined #openstack-infra06:42
*** pcaruana has joined #openstack-infra06:43
*** dbecker has joined #openstack-infra06:44
*** yolanda has quit IRC06:45
*** jtomasek has joined #openstack-infra06:45
*** r-daneel has quit IRC06:49
*** krtaylor has joined #openstack-infra06:50
*** alex_xu has quit IRC06:50
*** alex_xu has joined #openstack-infra06:51
*** e0ne has quit IRC06:59
*** stevebaker has quit IRC07:01
*** dmellado has quit IRC07:01
*** dayou_ has joined #openstack-infra07:04
*** rcernin has quit IRC07:05
*** slaweq has joined #openstack-infra07:05
*** slaweq has quit IRC07:06
*** slaweq has joined #openstack-infra07:06
chrisyang_0660Hi team, I was testing our CI system and find that it just can't make a new build.07:07
chrisyang_0660I checked the zuul log and found that something seems wrong with the zuul-merger.07:07
chrisyang_0660The example of error message shows that "Unable to find commit for ref master/Zc35f3d1ca65f466283d5df2e44a8404f"07:07
chrisyang_0660Does anyone know what happens and how to fix this?07:07
chrisyang_0660The detail log of merge-debug.log is as follows:07:07
chrisyang_0660https://gist.github.com/yljh21328/cabc0726b05aceb06fd54e5e2012488e07:07
*** quiquell|afk is now known as quiquell07:08
*** markvoelker has joined #openstack-infra07:08
fricklerchrisyang_0660: just guessing, as this isn't obvious from your logs: the address of review.openstack.org has changed, do you have outgoing firewall rules that may need updating?07:10
*** dpawlik has joined #openstack-infra07:10
*** dmellado has joined #openstack-infra07:12
evrardjphello07:16
*** dpawlik has quit IRC07:16
*** zoli is now known as zoli|wfh07:17
evrardjpmay I have some reviews on https://review.openstack.org/#/c/562619/ now that their dependant patches have merged?07:17
*** dpawlik has joined #openstack-infra07:19
*** tesseract has joined #openstack-infra07:20
*** florianf has joined #openstack-infra07:20
*** dmellado_ has joined #openstack-infra07:24
*** ssbarnea_ has joined #openstack-infra07:25
*** andreas_s has joined #openstack-infra07:25
*** jaosorior has quit IRC07:25
*** alexchadin has joined #openstack-infra07:32
*** ykarel is now known as ykarel|lunch07:32
*** dmellado_ has quit IRC07:34
chrisyang_0660frickler: I have tried to flush the dns cache. But still get the same error message.07:34
*** amoralej|off is now known as amoralej07:34
*** wolverineav has joined #openstack-infra07:34
*** edmondsw has joined #openstack-infra07:36
*** bauzas is now known as bauwser07:37
*** edmondsw has quit IRC07:41
*** markvoelker has quit IRC07:42
*** kukacz has quit IRC07:46
*** tosky has joined #openstack-infra07:47
*** jpena|off is now known as jpena07:48
*** ccamacho has joined #openstack-infra07:51
*** jpich has joined #openstack-infra07:51
*** kukacz_ has joined #openstack-infra07:55
*** wolverineav has quit IRC07:57
*** yamahata has joined #openstack-infra07:57
*** wolverineav has joined #openstack-infra07:57
*** janki has quit IRC07:57
*** ssbarnea_ has quit IRC07:59
*** kjackal has joined #openstack-infra08:00
*** dmellado_ has joined #openstack-infra08:01
*** kukacz_ has quit IRC08:01
*** kukacz_ has joined #openstack-infra08:01
*** wolverineav has quit IRC08:02
*** dchen has joined #openstack-infra08:04
*** dingyichen has quit IRC08:05
*** pcichy has quit IRC08:05
*** pcichy has joined #openstack-infra08:05
*** lucas-afk is now known as lucasagomes08:07
*** kukacz_ has quit IRC08:07
*** kukacz_ has joined #openstack-infra08:11
*** dmellado_ has quit IRC08:11
*** kukacz_ has quit IRC08:14
*** kukacz_ has joined #openstack-infra08:14
*** ykarel|lunch is now known as ykarel08:15
*** salv-orlando has joined #openstack-infra08:15
*** salv-orlando has quit IRC08:15
*** salv-orlando has joined #openstack-infra08:16
*** yamahata has quit IRC08:23
*** panda|off is now known as panda08:24
*** derekh has joined #openstack-infra08:25
*** pblaho has joined #openstack-infra08:28
*** stephenfin is now known as finucannot08:29
*** lyarwood has quit IRC08:34
*** andreas_s has quit IRC08:38
*** andreas_s has joined #openstack-infra08:39
*** markvoelker has joined #openstack-infra08:39
*** e0ne has joined #openstack-infra08:40
*** germs has quit IRC08:43
*** kjackal has quit IRC08:43
*** germs has joined #openstack-infra08:44
*** germs has quit IRC08:44
*** germs has joined #openstack-infra08:44
*** lbragstad has quit IRC08:45
*** lbragstad has joined #openstack-infra08:45
*** dmellado_ has joined #openstack-infra08:50
*** stevebaker has joined #openstack-infra08:53
*** salv-orlando has quit IRC08:53
hwoaranghello infra. anyone to look at https://review.openstack.org/#/c/554620/ again? thank you :)08:54
*** zhangfei has quit IRC08:54
chrisyang_0660frickler: Our zuul credential seems not working with review.openstack.org. After re-configure, it works fine. :)08:55
fricklerhwoarang: +2, but don't have time to check the mirror run today, hope some other infra-root can pick this up later09:01
*** andreas_s has quit IRC09:01
*** pcichy has quit IRC09:01
hwoarangthank you frickler much appreciated09:01
*** andreas_s has joined #openstack-infra09:02
*** janki has joined #openstack-infra09:03
*** vtapia has quit IRC09:08
*** zhangfei has joined #openstack-infra09:10
*** markvoelker has quit IRC09:12
evrardjpwhile we are at infra requests, I'd be super happy to have this reviewed too frickler :) https://review.openstack.org/#/c/562619/09:15
*** ssbarnea_ has joined #openstack-infra09:18
fricklerevrardjp: +2, too09:21
evrardjpwe are on a row! :)09:21
evrardjpthanks frickler !09:22
evrardjpyou're the deal!09:22
*** udesale_ has joined #openstack-infra09:29
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/56621709:29
*** udesale__ has joined #openstack-infra09:31
*** udesale has quit IRC09:32
*** dtantsur|afk is now known as dtantsur09:33
*** udesale_ has quit IRC09:34
*** vabada has quit IRC09:38
*** yamamoto has joined #openstack-infra09:49
*** vtapia has joined #openstack-infra09:52
*** yamamoto has quit IRC09:53
*** salv-orlando has joined #openstack-infra09:57
*** armaan has joined #openstack-infra10:04
*** wolverineav has joined #openstack-infra10:07
*** markvoelker has joined #openstack-infra10:08
*** wolverineav has quit IRC10:12
*** kjackal has joined #openstack-infra10:12
openstackgerritMerged openstack-infra/project-config master: Add os_masakari repo base jobs  https://review.openstack.org/56261910:13
*** udesale_ has joined #openstack-infra10:13
*** udesale__ has quit IRC10:16
fricklerinfra-root: FYI, not sure if we are using this anywhere, but if you are seeing ansible inventory runs fail, it could be this: https://bugs.launchpad.net/os-client-config/+bug/176881310:22
openstackLaunchpad bug 1768813 in os-client-config "config.get_cache_expiration_time() function missing in 1.31.0 release" [Undecided,New]10:22
*** zoli|wfh is now known as zoli|lunch10:22
*** andreas_s has quit IRC10:25
*** ethfci has joined #openstack-infra10:27
*** andreas_s has joined #openstack-infra10:27
*** r-daneel has joined #openstack-infra10:29
*** salv-orlando has quit IRC10:31
*** andreas_s has quit IRC10:35
*** zhangfei has quit IRC10:35
ianwhwoarang / frickler : i've cherry-picked that and running it manually first to make sure it's ok, then we can commit10:35
*** kjackal has quit IRC10:36
*** lpetrut has quit IRC10:39
hwoarangthank you ianw10:41
ianwhwoarang: i don't have a heap of time to debug it, but i don't think it works :/  http://paste.openstack.org/show/720356/10:42
*** markvoelker has quit IRC10:42
hwoaranglooks like we are missing some mkdir -p10:44
ianwthe rsync doesn't do that?10:44
hwoarangmaybe not recursively10:45
ianwit's always an adventure with rsync alphabet soup command options ... rsync -rlptDvz10:45
hwoarangi see the script already does mkdir -p for the rest of the repos10:45
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: openstack_project: Mirror some additional openSUSE repositories  https://review.openstack.org/55462010:46
hwoarangianw: could you try that please ^^10:46
ianwhwoarang: yep, already running with that :)10:47
ianwit seems happier10:47
hwoaranggood :)10:47
*** zhangfei has joined #openstack-infra10:47
ianwit's doing a bunch of aarch64/ppc64le etc stuff10:48
ianwit's probably not big enough to worry about10:49
hwoaranghmm that's in the containers repo10:50
hwoarangthat will go away after 42.3 is EOL so should be ok for now... it's not too big i think10:50
*** spsurya has quit IRC10:56
*** lathiat has joined #openstack-infra10:57
*** kjackal has joined #openstack-infra10:57
*** lathiat_ has quit IRC10:58
*** wolverineav has joined #openstack-infra11:02
*** zoli|lunch is now known as zoli|wfh11:02
*** andreas_s has joined #openstack-infra11:03
*** alexchadin has quit IRC11:04
*** salv-orlando has joined #openstack-infra11:07
*** lpetrut has joined #openstack-infra11:09
*** dave-mccowan has joined #openstack-infra11:13
*** kjackal has quit IRC11:15
*** alexchadin has joined #openstack-infra11:17
*** lucasagomes is now known as lucas-pto11:17
*** udesale__ has joined #openstack-infra11:22
*** udesale__ has quit IRC11:22
*** udesale__ has joined #openstack-infra11:23
*** udesale__ has quit IRC11:24
*** udesale_ has quit IRC11:24
*** udesale__ has joined #openstack-infra11:24
*** pcaruana has quit IRC11:27
*** quiquell is now known as quiquell|lunch11:28
*** udesale__ has quit IRC11:29
*** salv-orlando has quit IRC11:31
*** BobBall has quit IRC11:33
*** markvoelker has joined #openstack-infra11:39
*** pcaruana has joined #openstack-infra11:39
*** nicolasbock has joined #openstack-infra11:42
*** pbourke has quit IRC11:48
*** linkmark has quit IRC11:51
*** rosmaita has joined #openstack-infra11:53
*** edmondsw has joined #openstack-infra11:53
*** ldnunes has joined #openstack-infra11:53
*** kaisers1 has joined #openstack-infra11:55
*** kjackal has joined #openstack-infra11:55
*** kaisers has quit IRC11:56
*** annp has quit IRC11:57
*** pcaruana has quit IRC11:59
*** jpena is now known as jpena|lunch12:03
*** jistr is now known as jistr|mtg12:11
*** pcaruana has joined #openstack-infra12:12
*** markvoelker has quit IRC12:12
*** alexchadin has quit IRC12:15
*** ssbarnea_ has quit IRC12:16
*** dtantsur is now known as dtantsur|brb12:17
*** ethfci has quit IRC12:21
mnasermorning infra :>12:21
*** markvoelker has joined #openstack-infra12:24
*** quiquell|lunch is now known as quiquell12:24
*** electrofelix has quit IRC12:26
*** dhajare has quit IRC12:28
*** kgiusti has joined #openstack-infra12:29
*** jistr|mtg is now known as jistr12:31
*** ssbarnea_ has joined #openstack-infra12:33
*** rlandy has joined #openstack-infra12:34
*** vivsoni_ has joined #openstack-infra12:35
*** vivsoni has quit IRC12:35
*** trown|outtypewww is now known as trown12:37
*** armaan has quit IRC12:38
*** armaan has joined #openstack-infra12:38
openstackgerritMerged openstack-infra/project-config master: Stop gating on tripleo-incubator  https://review.openstack.org/56583612:43
odyssey4meoh dear, it looks like the ara-report is giving a 500 error dmsimard - any idea what's up with that?12:44
*** udesale has joined #openstack-infra12:44
dmsimardodyssey4me: looking12:45
*** yamamoto has joined #openstack-infra12:46
*** yamamoto has quit IRC12:48
*** yamamoto has joined #openstack-infra12:49
*** ssbarnea_ has quit IRC12:49
odyssey4medmsimard an example: http://logs.openstack.org/78/564778/14/check/openstack-ansible-functional-ubuntu-xenial/933a197/logs/ara-report/12:49
dmsimardodyssey4me: yeah I reproduced the issue and found the problem, trying to see how to fix it12:49
*** kjackal has quit IRC12:50
*** jcoufal has joined #openstack-infra12:53
*** yamamoto has quit IRC12:53
*** andreas_s has quit IRC12:54
*** jpena|lunch is now known as jpena12:54
*** udesale has quit IRC12:54
*** udesale has joined #openstack-infra12:54
*** alexchadin has joined #openstack-infra12:55
*** felipemonteiro has joined #openstack-infra12:55
*** andreas_s_ has joined #openstack-infra12:56
*** andreas_s_ has quit IRC12:56
*** andreas_s_ has joined #openstack-infra12:56
*** ssbarnea_ has joined #openstack-infra12:57
*** aojea has joined #openstack-infra12:59
*** yamamoto has joined #openstack-infra13:00
dmsimardodyssey4me: it works now but I haven't found the root cause, I'll have to investigate more to understand13:01
*** efried is now known as fried_rice13:02
dmsimardARA uses Ansible to load it's configuration and it was all of a sudden stack tracing: http://paste.openstack.org/raw/720366/13:02
*** Goneri has joined #openstack-infra13:02
dmsimardHowever, logs-dev.o.o did not exhibit the issue and in theory the only difference is that logs-dev has this patch enabled https://review.openstack.org/#/c/558688/13:03
openstackgerritDan Radez proposed openstack-infra/project-config master: Add initial job to zuul for networking-ansible  https://review.openstack.org/56537313:05
dmsimardasked #ansible-devel if it rings them a bell13:06
*** mriedem has joined #openstack-infra13:06
dmsimardinfra-root: https://review.openstack.org/#/c/558688/ is ready to merge, it's currently applied manually and static.o.o is in the emergency file, see ^ for details on the issue13:06
*** dtantsur|brb is now known as dtantsur13:09
*** gyankum has quit IRC13:09
*** pbourke has joined #openstack-infra13:11
*** amoralej is now known as amoralej|lunch13:12
*** psachin has quit IRC13:13
*** bobh has joined #openstack-infra13:14
*** r-daneel has quit IRC13:16
*** andreas_s_ has quit IRC13:17
*** eharney has joined #openstack-infra13:18
*** andreas_s_ has joined #openstack-infra13:19
*** esberglu has joined #openstack-infra13:28
*** zoli|wfh is now known as zoli|pto13:28
*** zoli|pto is now known as zoli13:29
*** gouthamr has joined #openstack-infra13:30
pabelangerdmsimard: it has 2 +2's if you want to monitor and +3, wfm13:34
dmsimardianw: thanks for the late +2 :)13:34
*** VW has joined #openstack-infra13:34
*** VW has quit IRC13:34
dmsimardpabelanger: ack, thanks13:34
*** VW has joined #openstack-infra13:35
*** andreas_s_ has quit IRC13:35
*** camunoz has joined #openstack-infra13:36
*** eernst has joined #openstack-infra13:36
*** andreas_s has joined #openstack-infra13:37
*** esberglu has quit IRC13:39
*** esberglu has joined #openstack-infra13:39
*** e0ne has quit IRC13:39
*** andreas_s has quit IRC13:40
*** andreas_s has joined #openstack-infra13:40
*** pbourke has quit IRC13:41
*** pbourke has joined #openstack-infra13:41
*** e0ne has joined #openstack-infra13:43
*** andreas_s has quit IRC13:44
*** yamamoto has quit IRC13:45
*** andreas_s_ has joined #openstack-infra13:46
TheJuliais the success stuff not working ? Seems last entry was the first of the month and I just tried it and nothing :(13:51
*** armaan has quit IRC13:52
*** r-daneel has joined #openstack-infra13:52
*** amoralej|lunch is now known as amoralej13:53
openstackgerritMerged openstack-infra/puppet-openstackci master: Enable tuned WSGI settings for logs.openstack.org  https://review.openstack.org/55868813:54
dmsimardTheJulia: success stuff ? you mean #thanks ?13:54
dmsimardor maybe #success is a thing, I forget13:55
TheJuliaI think it was supposed to start with success13:55
TheJuliano leading character13:55
TheJuliaat least that is the example I spotted13:55
TheJuliagah, the example is wrong13:55
dmsimardTheJulia: pretty sure it needs a leading # like most/all bot commands13:55
TheJulianevermind, ignore me13:55
dmsimardTheJulia: send a patch if the doc is wrong :)13:56
dmsimardmordred: I'm starting to understand why you use shade instead of openstackclient. Wow.13:57
*** rajinir has joined #openstack-infra13:57
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: WIP: Add fedora mirror to nodepool dsvm jobs  https://review.openstack.org/56610213:58
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora mirror to nodepool dsvm jobs  https://review.openstack.org/56610213:59
*** r-daneel has quit IRC13:59
*** r-daneel has joined #openstack-infra13:59
TheJuliadmsimard: I think it was on a superuser page, I closed it to try and free memory on my laptop14:01
*** quiquell is now known as quiquell|off14:06
mordreddmsimard: it's just easier for me14:06
openstackgerritMerged openstack-infra/nodepool master: Fix test patching of clouds.yaml file locations  https://review.openstack.org/56613814:07
*** armaan has joined #openstack-infra14:08
*** kiennt26 has joined #openstack-infra14:10
*** bobh has quit IRC14:11
*** bobh has joined #openstack-infra14:12
*** Guest46098 has quit IRC14:13
*** Guest46098 has joined #openstack-infra14:13
*** melwitt is now known as jgwentworth14:15
*** pcichy has joined #openstack-infra14:17
funginifra-root, config-core, infra-puppet-core et al... my internet service to the house is still out this morning with no eta for a fix (waiting on them to call me back to schedule sending out a tech to see what the road crew cut/crushed/shredded this time). i'm hobbling along on a very slow wireless modem in the meantime so may not be especially responsive14:17
fungierm infra-root ^14:18
*** salv-orlando has joined #openstack-infra14:18
*** salv-orlando has quit IRC14:18
*** alexchadin has quit IRC14:19
pabelangerfungi: ack14:23
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Remove debian-jessie from nodepool dsvm testing  https://review.openstack.org/56632714:26
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora mirror to nodepool dsvm jobs  https://review.openstack.org/56610214:28
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora-28 to nodepool dsvm  https://review.openstack.org/55921114:28
*** hemna_ has joined #openstack-infra14:31
*** e0ne has quit IRC14:32
*** ramishra has quit IRC14:33
*** lpetrut has quit IRC14:34
*** ssbarnea_ has quit IRC14:35
*** eernst has quit IRC14:36
*** pbourke has quit IRC14:39
*** germs has quit IRC14:39
*** zhangfei has quit IRC14:40
*** germs has joined #openstack-infra14:40
*** germs has quit IRC14:40
*** germs has joined #openstack-infra14:40
*** andreas_s_ has quit IRC14:41
*** woid has joined #openstack-infra14:41
*** eharney has quit IRC14:42
*** hongbin has joined #openstack-infra14:43
*** armaan has quit IRC14:44
*** agopi|out is now known as agopi14:44
*** armaan has joined #openstack-infra14:45
*** yamamoto has joined #openstack-infra14:45
*** r-daneel has quit IRC14:45
*** iranzo has quit IRC14:45
*** agopi has quit IRC14:45
*** agopi has joined #openstack-infra14:46
*** e0ne has joined #openstack-infra14:46
corvusfungi: i believe the correct proword is 'nifty-root'14:47
funginow it will be, for sure14:47
fungimay the fourth be with you all14:47
corvusand also with you14:48
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Add tox-py36 jobs  https://review.openstack.org/56588114:48
*** jistr is now known as jistr|tpb14:49
*** yamamoto has quit IRC14:52
*** ssbarnea_ has joined #openstack-infra14:52
*** r-daneel has joined #openstack-infra14:52
*** andreas_s has joined #openstack-infra14:52
*** dpawlik has quit IRC14:52
*** pbourke has joined #openstack-infra14:53
*** dhill_ has joined #openstack-infra14:53
*** eernst has joined #openstack-infra14:53
fungiconfig-core: i'm about to start importing heat's lp bugs into storyboard, if anyone is up for a second +2 on https://review.openstack.org/56619314:54
corvusfungi: +214:56
fungithanks!14:56
*** dmellado has quit IRC14:57
*** dmellado_ is now known as dmellado14:57
openstackgerritGraham Hayes proposed openstack-infra/yaml2ical master: Allow for meeting to be schedualed every 4 weeks.  https://review.openstack.org/56633514:58
*** gyankum has joined #openstack-infra14:59
*** Michael__ has joined #openstack-infra15:00
*** Michael__ has quit IRC15:00
*** andreas_s has quit IRC15:00
*** michaeldorner has joined #openstack-infra15:00
openstackgerritPaul Belanger proposed openstack/diskimage-builder master: Add support for Fedora 28, remove EOL Fedora 26  https://review.openstack.org/56633715:01
michaeldornerHi there! Are there any documentation about the numbers of committers (developer with write access to the repository) in OpenStack? Thanks alot!15:01
*** armaan has quit IRC15:02
*** armaan has joined #openstack-infra15:02
*** jistr|tpb is now known as jistr15:05
*** r-daneel has quit IRC15:06
*** r-daneel has joined #openstack-infra15:06
*** eharney has joined #openstack-infra15:07
fungimichaeldorner: we're not structured that way15:10
fungimichaeldorner: we operate through a communal code review process15:10
fungimichaeldorner: and technically only our continuous integration and gating system has write access to the git repositories we host15:11
fungi(well, and some sysadmins for the code review system)15:11
*** pcaruana has quit IRC15:13
fungimichaeldorner: probably the closest thing we have to the concept of "committers" is core review teams, who are the contributors with responsibility of approving proposed changes to their respective repositories15:14
openstackgerritMerged openstack-infra/project-config master: Add Heat to StoryBoard  https://review.openstack.org/56619315:14
mnasermichaeldorner: perhaps if we know what your end goal is we can help you get the right answer :)15:15
*** ramishra has joined #openstack-infra15:17
*** anteaya has joined #openstack-infra15:17
*** pbourke has quit IRC15:18
fungimichaeldorner: because you've got me curious, i'm doing a fresh run of https://git.openstack.org/cgit/openstack-infra/system-config/tree/tools/who-approves.py to find out how many core reviewers (people with approval rights) there are across all official openstack team deliverable repositories15:19
fungii'll let you know once it's done, but may be ~15 minutes or so to complete the requisite queries15:19
openstackgerritGraham Hayes proposed openstack-infra/yaml2ical master: Allow for meeting to be schedualed every 4 weeks.  https://review.openstack.org/56633515:19
*** dave-mccowan has quit IRC15:20
*** ramishra has quit IRC15:22
*** pbourke has joined #openstack-infra15:26
*** nicolasbock has quit IRC15:26
*** dave-mccowan has joined #openstack-infra15:27
*** dayou_ has quit IRC15:29
*** e0ne has quit IRC15:31
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet master: Update to support puppet v5  https://review.openstack.org/56634615:31
fungimmm, who-approves.py still wants to use the old gitweb interface we lost in the gerrit 2.13 upgrade15:33
openstackgerritJames E. Blair proposed openstack-infra/ansible-role-puppet master: Update to support puppet v5  https://review.openstack.org/56634615:33
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Support puppet 5 on ubuntu  https://review.openstack.org/56489815:34
corvuswe're going to find out if i've learned how to ansible :)15:34
radezHey folks, these groups were created for me yesterday: https://review.openstack.org/#/admin/groups/?filter=networking-ansible15:36
*** ykarel is now known as ykarel|away15:36
*** hashar is now known as hasharAway15:36
radezis this the right venue to req to be added to them? re: https://review.openstack.org/#/c/565370/15:37
clarkbradez: yup I'll take a look momentarily15:37
radezthx clarkb15:37
clarkbfungi: I'm glad the recent road work here didn't lead to destroy the Internets15:39
clarkbradez: done15:40
radezclarkb++15:40
radezthx for the quick response15:40
*** nicolasbock has joined #openstack-infra15:40
*** bnemec is now known as beekneemech15:42
*** dansmith is now known as superdan15:45
clarkbradez: no problem. A quick and easy first task of the morning.15:45
clarkbNow to double check that those ci jobs being accused of running on reused nodes didn't run on reused nodes15:46
*** armaan has quit IRC15:49
*** armaan has joined #openstack-infra15:49
openstackgerritMerged openstack-infra/project-config master: Add initial job to zuul for networking-ansible  https://review.openstack.org/56537315:50
*** dpawlik has joined #openstack-infra15:57
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora-28 to nodepool dsvm  https://review.openstack.org/55921115:57
*** michaeldorner has quit IRC15:59
dhellmanndoes the wiki still send email when watched pages are changed? I'm trying to figure out if I'm not receiving them or if they're not being sent.16:00
clarkbdhellmann: it sent me mail as recently as may 1st for a page I watch16:00
*** gyee has joined #openstack-infra16:00
clarkb(the infra meeting agenda page)16:00
*** eernst has quit IRC16:01
dhellmannclarkb : can you give me a search string to help me find similar messages? looking for "wiki openstack" is not producing useful results16:01
clarkbthe from is noc@openstack.org mailed by wiki.openstack.org. Subject is "OpenStack page foo/bar/etc has been changed by $user"16:01
dhellmanncool, thanks16:02
*** dpawlik has quit IRC16:02
*** eernst has joined #openstack-infra16:04
*** rkukura has joined #openstack-infra16:06
*** germs has quit IRC16:08
*** EmilienM is now known as EvilienM16:09
rajinirZuul Merger on out thirdparty is starting to fail. Has anything chnaged?16:09
rajinirERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f6e42f90e90>16:09
rajinirERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f6e42f90e90>16:09
dhellmannwell, I'm not finding any wiki messages at all. I have a rule for that email address to sort them into a folder that exists. and I used to get email reliably. :-(16:10
clarkbdhellmann: any idea when the last one you got was?16:10
dhellmannno, I don't have any old ones (they expired out of the mailbox and were expunged)16:11
*** armaan has quit IRC16:11
clarkbdhellmann: and is it to a redhat mx? could be possible that specific destinations are filtering them out16:11
dhellmannno, it's to my doughellmann.com address16:11
clarkbdhellmann: we'll have to look at logs probably16:11
dhellmannthat's the email I have set up LP, and that's the email the wiki thinks I have16:11
*** armaan has joined #openstack-infra16:11
mordredI don't see any doughellmann entries in the exim logs16:11
clarkbrajinir: zuulv3 work is fairly full steam ahead. Are you running v3 or v2?16:11
rajinirv216:11
rajinirclarkb: zull v216:12
dhellmannis there a sandbox page I can use to trigger something?16:12
clarkbrajinir: v2 shouldn't have changed at all since the last tag we made prior to v3 branch merging16:12
dhellmannI didn't have the "send me emails I send to others" flag turned on for my most recent edit16:12
mordredbut I do see entries for cboylan16:12
*** Guest59082 has quit IRC16:12
clarkbrajinir: do those log entries come with tracebacks?16:12
rajinirclarkb: Stopped working since May2 10.30 AM CST16:12
*** fried_rice is now known as fried_rolls16:13
clarkbdhellmann: I'm not aware of one but you could create one :)16:13
rajinirClark: error from merger-debug.log https://www.irccloud.com/pastebin/pPa6qmEN/Merger%20error16:13
mordreddhellmann: I'm watching the exim logs at the moment16:13
*** r-daneel has quit IRC16:13
clarkbdhellmann: if you give me a link I can make edits too so that you can test the other user making changes approach16:13
dhellmannlet me make a change on the TC Tracker page16:13
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Fix typo with _logConsole function  https://review.openstack.org/56635616:13
mordred++16:13
*** r-daneel has joined #openstack-infra16:13
dhellmannhttps://wiki.openstack.org/w/index.php?title=Technical_Committee_Tracker&action=edit16:14
dhellmannok, my edit was saved16:14
mordreddhellmann: that sent an email to thierry16:14
clarkbrajinir: stderr: 'fatal: Could not read from remote repository. is the clue we need. We moved our gerrit install and it has a new ip address now. You are likely getting firewall filtered?16:14
clarkbrajinir: I would try testing connectivity to review.o.o independent of zuul and work from there16:15
*** kiennt26 has quit IRC16:15
rajinirclark: Is there a email that I missed. What was the old ip?16:15
mordreddhellmann: do you have the right notifications set on https://wiki.openstack.org/wiki/Special:Preferences#mw-prefsection-echo ?16:16
rajinirclark: ping of ping review.openstack.org  works16:16
dhellmannmordred as far as I can tell I have all notifications turned on16:16
clarkbrajinir: there were many emails from pabelanger. ping uses icmp packets and not tcp to ports 443 and 2941816:17
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Remove debian-jessie from nodepool dsvm testing  https://review.openstack.org/56632716:17
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora mirror to nodepool dsvm jobs  https://review.openstack.org/56610216:17
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Add fedora-28 to nodepool dsvm  https://review.openstack.org/55921116:17
clarkbrajinir: I would try curl/wget for web traffic and ssh directly for ssh traffic16:17
dhellmannenable email from other users; send me copies; email me when a page or a file is changed; email me for minor edits -- all checked yes16:17
dhellmannemail address as confirmed 201416:17
mordreddhellmann: those are different options than I'm seeing on my preferences page16:18
dhellmannI have no "real name" set; do I need that?16:18
mordredoh - wow. there are two different sets of email settings16:18
dhellmannmordred : ok, I went to my name at the top of the page and clicked "preferences" then scrolled down16:18
dhellmannoh, yay16:18
dhellmannhow did you get to that page you linked earlier?16:18
mordreddhellmann: there's the email settings on User Profile - but there's also a notifications panel16:18
pabelangerclarkb: rajinir: here is one of the emails for IP information: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html16:18
rajinirclarkb: ssh works. "ssh -p 29418 openstack-ironic-ci@review.openstack.org gerrit stream-events" , able to see responses16:19
dhellmannok, I have turned on all of those as well except "email from other user" which it won't let me set on16:19
*** derekh has quit IRC16:21
clarkbrajinir: from the merger host? what about `curl https://review.openstack.org` ?16:22
mordreddhellmann: have you made any additional edits you would expect to email you?16:23
dhellmannI'm going to do one now16:23
clarkband is now a good time for me to make an edit?16:23
rajinirpabelanger, clarkb:  "curl https://review.openstack.org" works16:23
dhellmannI changed open-id to sync my email on each login, then logged out, and now I'm going to make an edit16:23
*** rpioso|afk is now known as rpioso16:23
dhellmannmordred : done16:24
clarkbrajinir: what does `git remote -v` show you in that repo?16:24
mordreddhellmann: I also edited the page - and I see that an email was sent to you16:24
clarkbrajinir: specific for the origin remote16:24
dhellmannmordred : I can confirm that I received it16:24
mordredwoot16:24
mordredI do not know why it stopped working for you16:24
dhellmannso I wonder which change was needed; the notifications thing or the open-id thing16:24
mordredit's a mystery16:24
dhellmannI received the email from your change, but not mine, but that's ok16:25
dhellmannI don't need to be notified when I change a page16:25
mordreddhellmann: not related to that - but thank you for your lower-constraints work - it totally just appropriately failed a test where I should have bumped a min16:25
dhellmann\o/16:25
dhellmannmordred, clarkb : thanks for your help fixing my notification emails16:26
mordred\o/16:26
mordredit's  like a day of accomplishments16:26
rajinir<clarkb> that repo error message cam from the merger-debug.log. I'm not sure where I should run the "git remote -v" in16:27
clarkbrajinir: /var/lib/zuul/git/openstack/ironic says the log file16:28
rajinirAh: that folder is missing16:28
clarkbrajinir: if you restart the merger I believe it will reclone missing repos16:29
clarkbthat might be the easiest fix if the problem is the repo does not exist16:29
fricklerrajinir: clarkb: maybe a restart of the merger is needed in order to pick up the new address from DNS?16:29
fricklerthere was a similar issue earlier today16:30
*** jpich has quit IRC16:31
rajinirclarkb: same error, its not recloning the missing repos16:32
rajinirclarkb: after the restart16:32
rajinirclarkb: shall I manually clone it16:32
clarkbrajinir: does /var/lib/zuul/git exist?16:32
*** dtantsur is now known as dtantsur|afk16:32
clarkband and are permissions set to allow zuul to write to it?16:33
rajiniryes16:33
rajinirit has other projects16:33
rajinirhttps://www.irccloud.com/pastebin/ueslbSjV/16:33
clarkbya you can try manually cloning it, that shouldn't be necessary though so debugging why clones aren't happening may be useful16:33
fungidhellmann: sorry, catching back up... mediawiki doen't send you notifications of your own edits. that "send me copies of messages i send to other users" option is, i believe, for the mediawiki contact feature where you can send specific messages to other users. i don't believe it's associated at all with page watches16:34
dhellmannfungi : ah, that makes more sense16:35
rajinirclarkb: Ironic project is under the folder /var/lib/zuul/git/openstack16:35
rajinirhttps://www.irccloud.com/pastebin/DKmmxpsL/16:36
clarkbrajinir: did you clone it or did it clone it?16:37
rajinirclarkb: It was already there16:37
rajinirclarkb: I missed it before sorry16:38
clarkbrajinir: and when you were testing ssh before you were on that host?16:38
clarkbsince that remote says ssh that is what we need to ensure is working16:38
rajinirclarkb: Yes same host16:38
*** eharney has quit IRC16:38
pabelangermaybe ssh -v review.openstack.org -p2941816:38
pabelangersee what the output says16:39
clarkbthe other thing to consider is the user and the ssh key16:39
clarkbyou'll want to test that as the zuul user to make sure it has access to the key16:39
pabelangeryah, was thinking that too16:39
rajinirpabelanger> :clarkb: permission denied "ssh -v review.openstack.org -p29418"16:40
clarkbrajinir: you might have to specify the user16:41
rajinirhttps://www.irccloud.com/pastebin/DwJPVQWX/16:41
pabelangeryah, you are trying root16:41
pabelangerbut you routed to review.o.o16:41
rajinirpabelanger> the ssh keys for root and zuul are same, I can retry as zuul16:42
clarkbrajinir: the problem isn't the local user its the remote user. You can't ssh to review.o.o as root on the review.o.o16:43
fungirajinir: running `ssh review.openstack.org ...` as the root account infers that you want to authenticate as root@review.openstack.org16:43
rajiniroh sorry16:43
*** trown is now known as trown|lunch16:43
pabelangerssh -v openstack-ironic-ci@review.openstack.org -p2941816:43
rajinirssh -v openstack-ironic-ci@review.openstack.org -p29418 works fine16:43
rajinirhttps://www.irccloud.com/pastebin/Ls70a7F6/16:44
pabelangerI'd try again as your zuul user to confirm16:44
*** shoogz has quit IRC16:44
*** armaan has quit IRC16:45
rajinirpabelanger: zuul user has issue16:45
pabelangerpermission denied?16:45
rajinirpabelanger:yes permission denied16:45
pabelangeroh16:46
pabelangeryou might need to use -i to path of your private key16:46
pabelangerwill be value in /etc/zuul.conf16:46
pabelangerotherwise, you private key might not be correct any more16:46
rajinirpabelanger:  "ssh -i /var/lib/zuul/ssh/id_rsa -v openstack-ironic-ci@review.openstack.org -p29418" worked16:48
pabelangerokay, I'd expect zuul-merger to still work then16:48
rajinirzuul.conf is referring to /var/lib/zuul/ssh/id_rsa16:48
rajinircd /etc16:49
rajinirzuul.conf:server=review.openstack.org16:49
pabelangerrajinir: and zuul-merger is running a zuul user?16:49
*** udesale has quit IRC16:50
rajinirBoth are zuul and zuul-merger are running as root. It always was16:50
*** dkliban has left #openstack-infra16:50
pabelangerAh, okay, they don't need to use root but that is a change for another day16:51
rajinirpabelanger: ok16:51
rajinirpabelanger: yes defintely a key issue16:53
rajinirpanelanger: When I run "git fetch --tags -v origin" in the openstack/ironic fit folder which the merger is doing, I see the same error16:53
pabelangerand root user can read /var/lib/zuul/ssh/id_rsa ?16:54
pabelangereg16:54
pabelanger# ssh -i /var/lib/zuul/ssh/id_rsa -v openstack-ironic-ci@review.openstack.org -p2941816:54
pabelangeras root16:54
*** shoogz has joined #openstack-infra16:55
rajinirYes, root user can run that as well16:55
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: De-gitweb-ify who-approves.py  https://review.openstack.org/56636516:55
*** armaan has joined #openstack-infra16:55
*** jpena is now known as jpena|off16:55
fungimichaeldorner: script needed some fixes ^16:55
rajinirpabelanger: .git/config16:56
rajinirhttps://www.irccloud.com/pastebin/AlVbA6dw/16:56
pabelangerrajinir: and if you deleted the repo, zuul-merger won't reclone it? or get the same error16:56
fungimichaeldorner: at present count we have 760 distinct people with permission to approve changes to one or more of the 801 deliverable repositories for official openstack project teams16:57
rajinirpabelanger: I can rename it and restart to see if that fixes the problem16:57
pabelangerI just checked zm01.o.o debug logs and can see no issue with ironic16:57
pabelangerso, maybe something it breaking git connection on network?16:58
pabelangerrajinir: this is corporate network?16:58
rajinirpabelanger: nope, we have a special lab16:58
rajinirwith exceptions16:59
pabelangersee if zuul-merger reclones, but might need to talk with NOC team and confirm new IP is whitelisted16:59
pabelangerthat's about the only thing I can think off16:59
clarkbyou can try running those git commands manually too17:00
clarkband even strace them to see what is going on17:00
rajinirok17:07
*** panda is now known as panda|off17:07
rajinirwill try now17:07
*** tesseract has quit IRC17:09
*** myoung|ruck is now known as myoung|ruck|biab17:13
*** chandankumar is now known as huntman17:16
*** huntman is now known as groot17:16
*** esberglu has quit IRC17:17
*** germs has joined #openstack-infra17:18
*** germs has quit IRC17:18
*** germs has joined #openstack-infra17:18
*** germs has quit IRC17:19
*** pblaho has quit IRC17:19
*** germs has joined #openstack-infra17:19
*** AR2018 has joined #openstack-infra17:20
openstackgerritEric Brown proposed openstack-infra/project-config master: Remove Bandit from Infrastructure Systems  https://review.openstack.org/56553717:20
pabelangerclarkb: fungi: mind a review on https://review.openstack.org/565869/ removes some testing on trusty for system-config17:23
pabelangerwe also should figure out the next step to migrate the servers we missed last time we migrated from trusty -> xenial. We'll need to schedule outages for most of them17:24
pabelangereg: static.o.o / lists.o.o17:24
clarkbetherpad and ethercalc17:24
pabelangeryah17:25
clarkbmaybe we wait for post summit at this point for those17:25
clarkbbut ya17:25
clarkbpabelanger: start by generating an up to date list of trusty nodes?17:25
pabelangersure17:25
fungilists.o.o we deliberately delayed as we're going to want to do an in-place upgrade on it like last time so we can preserve its ip address17:25
*** tosky has quit IRC17:27
*** tosky has joined #openstack-infra17:30
openstackgerritAndreas Florath proposed openstack/diskimage-builder master: WIP: Docker matrix build  https://review.openstack.org/41434717:31
openstackgerritMerged openstack-infra/nodepool master: Fix typo with _logConsole function  https://review.openstack.org/56635617:31
pabelangerclarkb: do you remember the syntax you used? lsb_release -d on all hosts?17:33
clarkbI think -c ?17:34
pabelanger-c works too17:34
pabelangerclarkb: ianw: any objections is I delete the old mirror-update.openstack.org server from rax?  I think we can say mirror-update01.o.o is working fine17:37
clarkbI'm fine with it17:38
pabelangerack, will delete17:38
*** groot is now known as chandankumar17:39
fungiwfm17:43
*** ykarel|away has quit IRC17:43
*** rkukura has quit IRC17:44
*** yamamoto has joined #openstack-infra17:50
*** trown|lunch is now known as trown17:52
*** pcaruana has joined #openstack-infra17:54
*** yamamoto has quit IRC17:54
*** eharney has joined #openstack-infra17:56
*** aojea has quit IRC17:58
*** dpawlik has joined #openstack-infra17:58
*** dpawlik has quit IRC18:03
pabelangerfungi: clarkb: okay, I've started work on https://etherpad.openstack.org/p/infra-sprint-xenial-upgrades-part2 even included link to ethercalc for us to decided when to next have virtual sprint18:05
pabelangerthere is about 30 servers left to upgrade to xenial18:05
rajinirpabelanger> Temporarily I changed the remote url to use https to get it working18:07
pabelangerrajinir: ack, good to know18:08
*** pcichy has quit IRC18:13
rajinir<pabelanger,clarkb>  Thank you for your time to narrow down the issue.18:14
*** Goneri has quit IRC18:14
mordredpabelanger, fungi, clarkb: this may be a dumb question - but should we consider upgrading those services directly to bionic?18:16
fungimordred: perhaps? i mean, for lists.o.o almost certainly (that's what we did for precise-trusty) but in general rebuilding from scratch has been good hygiene to help us track down bootstrap-oriented regressions in our configuration management18:18
clarkbmordred: we don't currently have a puppet option for bionic (corvus is working through that)18:19
clarkbmordred: hence the we need to sort out future of config management18:19
mordredfungi: oh - yeah, I mean doing a rebuild - but on bionic18:19
*** janki has quit IRC18:19
clarkbif we had working puppet I think we would but right now getting to xenial is probably a bigger leap in general18:20
mordredclarkb: and yes, this is an excellent point ... I guess maybe what I should have said is "maybe we should wait to do this sprint until we know what the future of config management is, and then once we know, perhaps we should just rebuild the new services directly on bionic"18:20
clarkbxienal to bionic seems mostly the same from a server standpoint18:20
clarkball the changes were desktop side of house18:20
mordredbut we can also do trusty-xenial and then later xenail-bionic too18:20
fungimordred: oh! asking whether we should skip from trusty to bionic?18:22
*** rosmaita has quit IRC18:22
clarkbfungi: yes18:22
*** dtruong2 has joined #openstack-infra18:22
*** dtruong2 has joined #openstack-infra18:22
fungiseems kinda soon to be putting production workloads on bionic, and unfortunately it's mostly important systems remaining18:23
*** kgiusti has quit IRC18:27
corvuswe have a year of trusty left -- i think deferring until we have future-of-config-mgmt sorted out seems reasonable, because, especially if we go a non-puppet route, a jump from trusty to bionic seems like it might work out well18:32
*** owalsh is now known as owalsh-afk18:32
corvusi mean, hopefully we'll have that question answered within the next couple of weeks at most anyway, then we can make a plan based on that18:32
corvus(i say especially for a non-puppet route because both non-puppet options mean our operating system version is less (or not at all) tied to our config management version)18:33
mordredcorvus: ++18:34
fungibut also if we end up deciding to switch to non-puppet that helps us not have to worry about <bionic support in whatever we have to write from scratch18:35
mordredyup18:37
*** eernst has quit IRC18:38
*** gyankum has quit IRC18:49
*** fried_rolls is now known as fried_rice18:49
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Add openstacksdk to nodepool-functional-py35-src  https://review.openstack.org/56638718:50
*** eernst has joined #openstack-infra18:57
*** dtruong2 has quit IRC18:59
*** rosmaita has joined #openstack-infra19:00
*** pcaruana has quit IRC19:10
pabelangerheads up, power is out here. I'm tethered to a GSM modem ATM19:14
fungipabelanger: woo! at least i have power so i don't have to worry about my gsm modem running out of battery19:16
pabelangerfungi: yah, I have 1hr still but going to offline for a bit to conserve19:18
*** florianf has quit IRC19:18
pabelangerI'll check back periodically19:18
*** Goneri has joined #openstack-infra19:19
clarkbI'm worried what is going to happen to me now19:19
*** eernst has quit IRC19:20
*** eernst has joined #openstack-infra19:20
fungiwe're all being strategically targeted as part of some coordinated scheme to take down the infrastructure19:20
*** eernst has quit IRC19:28
clarkbso I've poked around possible options for infra team dinner. I think Monday is our best scheduling option. As far as venues go I'm having a hard time finding places that will take groups and not make a giant production out of it19:31
clarkbhowever tap and barrel and craft beer market seems like maybe they would work19:31
clarkbEvilienM: ^ it was mentioned i should bug you19:31
fungione of those where we just sort of all show up and head to different tables?19:31
clarkbya it may be, otherwise its paying deposits for set dinners and I'm not in the mood for setting that up or figuring out how to pay for it19:32
fungiagreed. somethnig informal where we don't all have one huge table nor necessarily get seated/served at exactly the same time seems fine to me19:34
dmsimard+119:34
EvilienMclarkb: me or the evil?19:37
clarkbEvilienM: the bar hopping french canadian :P19:38
bkeroDanger, danger19:39
* fungi wonders if EvilienM looks like EmilienM with a mustache and goatee, a la the star trek "mirror mirror" episode19:40
bkeroAlmost certainly. Can we get a gimp edit to prove it? :D19:41
*** eernst has joined #openstack-infra19:42
openstackgerritAnita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1  https://review.openstack.org/55797919:44
EvilienMclarkb: I'll come back to you after my lunch19:47
EvilienMbut I have a lot of options for you19:47
openstackgerritAnita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1  https://review.openstack.org/55797919:47
anteayapabelanger, what is wrong with the power where you are?19:51
anteayahow did it happen that it is absent?19:52
*** ykarel|away has joined #openstack-infra19:53
* fungi expects it is faced with an insurmountable insulator19:55
anteayaha ha ha19:56
anteayaperhaps a racoon19:56
anteayaah, they have a wind warning: https://weather.gc.ca/city/pages/on-147_metric_e.html19:57
anteayayes, a tree taking out a line would be quite the insulator19:57
TheJuliaWell, it might not be insulating... where as it could be shunting back ground instead19:58
*** e0ne has joined #openstack-infra19:59
anteayaTheJulia, good point19:59
*** dpawlik has joined #openstack-infra19:59
fungii'm heading out to get some food, back later20:01
openstackgerritAnita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1  https://review.openstack.org/55797920:03
*** dpawlik has quit IRC20:04
*** bgmccollum_ is now known as bgmccollum20:08
clarkbcorvus: I'm going to finish up lunch but then I think it would probably be a good time ti restart the zuul scheduler to pick up the config inclusions fix20:14
clarkbmore of a heads up than anything else I guess20:14
clarkbmordred: ^ also pinging you as you were workign on getting openstacksdk type fixes out the door?20:14
corvusclarkb: sounds good; things don't seem crazy busy20:16
*** jcoufal has quit IRC20:18
*** gfidente has quit IRC20:20
*** armaan has quit IRC20:23
*** armaan has joined #openstack-infra20:23
TheJuliaanteaya: Of course, the happy medium between the two is the voltage getting high enough to jump the air gap and arc20:24
anteayaTheJulia, oh that sounds fun20:24
anteayaactually it sounds like large arcs are headed my way20:24
anteayaso I think I may be unplugging for a bit until it finishes20:25
*** anteaya has quit IRC20:26
*** Goneri has quit IRC20:29
EvilienMclarkb, fungi : I have some suggestions for you then, based on what you want. Feel free to look on their websites:20:31
EvilienMhttp://www.elephantcastle.com/marine_bldg20:32
EvilienMhttp://donnellygroup.ca/the-blackbird/ this one is good but not if you're a lot20:33
EvilienMhttps://theportsidepub.com/ is really nice but might be noisy, probably not too busy for a Monday20:33
EvilienMand my favourite place: http://www.joefortes.ca/20:34
EvilienMthis place is AMAZING20:34
EvilienMon my list, there is https://www.backandforthbar.com/ as well. The food isn't amazing (more snacks than actual meals) but a good place to hangout20:35
*** hasharAway is now known as hashar20:36
*** ykarel|away has quit IRC20:44
*** camunoz has quit IRC20:45
clarkbEvilienM: thanks!20:49
EvilienMI hope it helped20:49
clarkbcorvus: ok I'm just about ready to restart the scheduler. I'm gonna double check with release team as there was earlier mention of maybe doing some bug fix releases for modred20:52
corvusclarkb: cool.  pipelines look clear20:52
clarkbcorvus: zuul==3.0.3.dev64  # git sha 7816dfc looks correct to me on zuul0120:54
corvusayup20:54
clarkbrelease team says go for it20:54
clarkbcorvus: dump queues then stop, start zuul-scheduler then restart zuul-web and reenqueue once openstack tenant is ready?20:55
corvusclarkb: yep.  see if zuul-web survives though.20:55
clarkbok I'm starting that now20:56
corvusi thought it might have last time we restarted, though i know of no fix.  so it's just more data collection.20:56
clarkbscheduler is starting back up again20:57
clarkbzuul web seems happy other than the tenant isn't ready yet20:57
*** bobh has quit IRC20:57
*** trown is now known as trown|outtypewww20:57
*** bobh has joined #openstack-infra20:58
clarkbenqueuing jobs now21:00
corvusyep, and my status page refreshed without incident21:00
clarkbtook about 4 minutes to laod which seems about right ( so I think theory about gerrit caches is probably accurate)21:01
corvuscool, that's actually even a little fast i think (i had 6m in my mind)21:01
clarkbenqueueing of check is done now too21:05
clarkb(gate was done first)21:05
clarkbcorvus: I wonder if the asyncio gearman is what makes zuul-web happy now21:05
clarkbit may still be blocking on those old connections until the timeout but new connections hit new gearman server and all is happy21:06
*** VW_ has joined #openstack-infra21:09
*** e0ne has quit IRC21:11
*** mriedem is now known as mriedem_yard21:13
corvusclarkb: could be!21:13
*** VW has quit IRC21:13
openstackgerritMerged openstack-infra/zuul master: Add zuul systemd drop-in files for CentOS 7  https://review.openstack.org/56507421:13
openstackgerritMerged openstack-infra/zuul master: Skip attempting python3-devel installation on CentOS 7  https://review.openstack.org/56508021:13
*** VW_ has quit IRC21:13
*** ldnunes has quit IRC21:17
openstackgerritMerged openstack-infra/zuul master: Add start and end timestamp to task and play result in zuul_json callback  https://review.openstack.org/56388821:17
*** myoung|ruck|biab is now known as myoung|ruck21:18
pabelangerand power is back21:19
*** dave-mccowan has quit IRC21:19
*** felipemonteiro has quit IRC21:20
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add logo to docs  https://review.openstack.org/56640621:22
pabelangerclarkb: I think glean is broken on tumbleweed: http://logs.openstack.org/11/559211/5/experimental/nodepool-functional-py35-suse-src/7973ce7/controller/logs/screen-nodepool-launcher.txt.gz21:22
pabelangerI'll maybe give it a go on monday see if I can find anything specific21:23
*** bobh has quit IRC21:23
pabelangeralso, any ideas how we can avoid the double timestamps?21:23
pabelangerMay 04 17:04:30.154714 ubuntu-xenial-inap-mtl01-0003873903 nodepool-launcher[25044]: 2018-05-04 17:04:30,154 DEBUG nodepool.ProviderManager: Creating new ProviderManager object for devstack21:23
clarkbpabelanger: re timestamps I believe expectation is systemd provides them for devstack services. To fix it we can stop adding timestamps to our log formatters when run under devstack21:24
pabelangerclarkb: kk, I'll look at the logging handlers21:25
clarkband ya looks like it starts glean for ens3 but then it doesn't finish successfully/21:26
*** dbecker has quit IRC21:38
*** hashar has quit IRC21:42
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add logo to docs  https://review.openstack.org/56640621:47
*** edmondsw has quit IRC21:47
*** edmondsw has joined #openstack-infra21:47
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Remove initial stub release note  https://review.openstack.org/56640921:48
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Add logo to docs  https://review.openstack.org/56641021:48
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Add logo to docs  https://review.openstack.org/56641121:50
openstackgerritJames E. Blair proposed openstack-infra/zuul-base-jobs master: Add logo to docs  https://review.openstack.org/56641221:51
*** hamzy_ has joined #openstack-infra21:51
*** edmondsw has quit IRC21:52
openstackgerritJames E. Blair proposed openstack-infra/zuul-sphinx master: Add logo to docs  https://review.openstack.org/56641321:52
*** hamzy has quit IRC21:54
*** wolverineav has quit IRC21:54
*** wolverineav has joined #openstack-infra21:55
*** AR2018 has quit IRC21:55
*** wolverineav has quit IRC21:59
openstackgerritClark Boylan proposed openstack-infra/jeepyb master: Cache created-in-github properly  https://review.openstack.org/56641622:01
openstackgerritClark Boylan proposed openstack-infra/jeepyb master: Check github groups case insensitively  https://review.openstack.org/56641722:01
clarkbcorvus: fungi pabelanger ^ followup from yesterday. I am going to WIP those because I want to test them probably on review-dev first then review.o.o before we merge22:01
clarkbhowever I think they are ready for review if you want ot take a look given what we learned yesterday22:01
openstackgerritJames E. Blair proposed openstack-infra/puppet-mailman master: Exit 0 if MAILMAN_SITE_DIR is not set  https://review.openstack.org/56641822:02
*** slaweq has quit IRC22:03
corvusclarkb, fungi, pabelanger: ^ that should get things working on lists.o.o again.  I manually tried it with an "apt-get upgrade" there and it completes now.22:03
clarkbcorvus: another approach may be to have a default site dir that dpkg can update?22:04
clarkbbut maybe best to avoid getting out of sync across site dirs22:04
corvusclarkb: we don't actually care about anything it's doing anyway.22:05
corvusclarkb: it's just trying to figure out what languages are used, but we manage that ourselves22:05
corvusclarkb: comment on that first change; second lgtm.22:07
*** ccamacho has quit IRC22:10
*** jtomasek has quit IRC22:10
clarkbaha! the return value is really replicate or not, that also explains why we set it to true if we add gerrit to the group22:11
*** ccamacho has joined #openstack-infra22:12
*** AR2018 has joined #openstack-infra22:16
openstackgerritClark Boylan proposed openstack-infra/jeepyb master: Check github groups case insensitively  https://review.openstack.org/56641722:18
openstackgerritClark Boylan proposed openstack-infra/jeepyb master: Cache created-in-github properly  https://review.openstack.org/56641622:18
clarkbcorvus: ^ I reordered them since I realized that created-in-github change is much bigger and likely to go in second22:19
*** AR2018 has quit IRC22:19
*** r-daneel has quit IRC22:21
*** AR2018 has joined #openstack-infra22:24
*** markvoelker has quit IRC22:29
*** AR2018 has quit IRC22:29
*** rlandy has quit IRC22:30
clarkbdoesn't look like manage_projects has ever been run on new review-dev22:41
*** rosmaita has quit IRC22:41
*** wolverineav has joined #openstack-infra22:41
clarkbnot surprising, but means we don't have any existing cache data there either which is likely a good thing (means we can watch it through that progression)22:41
clarkbI'll work on that next week22:41
clarkbnow the good weather is calling me to get some work done outside22:41
*** ssbarnea_ has quit IRC22:43
*** wolverineav has quit IRC22:45
fungiokay, i'm done with friday shenanigans and catching up22:47
*** wolverineav has joined #openstack-infra22:49
*** rpioso is now known as rpioso|afk22:50
*** linkmark has joined #openstack-infra22:52
*** wolverineav has quit IRC22:53
*** fried_rice is now known as efried22:57
*** hongbin has quit IRC23:01
*** wolverineav has joined #openstack-infra23:03
*** wolverineav has quit IRC23:05
*** bobh has joined #openstack-infra23:06
*** wolverineav has joined #openstack-infra23:06
openstackgerritMerged openstack-infra/puppet-mailman master: Exit 0 if MAILMAN_SITE_DIR is not set  https://review.openstack.org/56641823:08
*** wolverineav has quit IRC23:10
*** slaweq has joined #openstack-infra23:11
*** krtaylor has quit IRC23:13
*** slaweq has quit IRC23:15
*** bobh has quit IRC23:17
*** yamamoto has joined #openstack-infra23:21
*** VW has joined #openstack-infra23:24
*** wolverineav has joined #openstack-infra23:26
*** yamamoto has quit IRC23:26
*** wolverineav has quit IRC23:31
*** VW has quit IRC23:34
*** wolverineav has joined #openstack-infra23:34
*** VW has joined #openstack-infra23:35
*** eernst has quit IRC23:37
*** wolverineav has quit IRC23:39
*** wolverineav has joined #openstack-infra23:40
*** wolverineav has quit IRC23:40
*** wolverineav has joined #openstack-infra23:40
*** wolverin_ has joined #openstack-infra23:43
*** wolverineav has quit IRC23:45
*** wolverin_ has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!