Monday, 2018-08-06

openstackgerritMerged openstack-infra/puppet-accessbot master: Add alumni to accessbot  https://review.openstack.org/58811500:07
openstackgerritMerged openstack-infra/puppet-accessbot master: Add mlock setting and op_channel option  https://review.openstack.org/58813400:07
*** efried1 has joined #openstack-infra00:16
*** efried has quit IRC00:20
*** efried1 is now known as efried00:20
*** auristor has quit IRC00:31
*** dingyichen has joined #openstack-infra00:33
*** auristor has joined #openstack-infra00:34
*** xarses has joined #openstack-infra01:10
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route  https://review.openstack.org/55097801:13
*** jtomasek has joined #openstack-infra01:13
*** pbourke has quit IRC01:15
*** pbourke has joined #openstack-infra01:17
*** jtomasek has quit IRC01:18
openstackgerritMerged openstack-infra/project-config master: Add OpenStack IRC Wall of Fame  https://review.openstack.org/58811801:19
openstackgerritMerged openstack-infra/project-config master: Enforce channel modes with accessbot  https://review.openstack.org/58813901:19
*** chason has joined #openstack-infra01:21
openstackgerritIan Wienand proposed openstack-infra/project-config master: accessbot config: refactor  https://review.openstack.org/58899901:31
*** kaiokmo has quit IRC01:33
*** hongbin has joined #openstack-infra01:39
tonybI'm seeing the build-wheel-mirror jobs filaing pretty quick in https://review.openstack.org/#/c/588441 looking at the logs it seems like something's up with puppet http://logs.openstack.org/41/588441/1/check/build-wheel-mirror-centos-7/5ff4e54/job-output.txt.gz#_2018-08-05_22_55_55_637217 and01:53
tonybhttp://logs.openstack.org/41/588441/1/check/build-wheel-mirror-ubuntu-bionic/780ccc6/job-output.txt.gz#_2018-08-05_22_57_20_263688 Is that a know issue?01:53
openstackgerritIan Wienand proposed openstack-infra/puppet-accessbot master: Small refactor, remove global alumni and mode keys  https://review.openstack.org/58900101:55
ianwtonyb: hrm02:06
ianwwe merged changes to install_puppet iirc02:07
ianwno, ok, that's same for a while02:08
tonybianw: Okay, I really have very little idea how to debug it02:09
ianwi think i see what's happened ..02:13
tonybianw: Thanks fro looking02:13
ianwhttp://logs.openstack.org/41/588441/1/check/build-wheel-mirror-ubuntu-bionic/780ccc6/job-output.txt.gz#_2018-08-05_22_57_10_18252402:18
ianwis where it all starts02:18
tonybianw: Ahh okay02:25
tonybianw: that url is a 40402:26
ianwyeah ... and so then we *think* we've installed puppet 3, but really we've installed 502:27
ianwthen it blows up in install_modules.sh02:27
tonybOkay.02:29
*** psachin has joined #openstack-infra02:30
ianwthere is no puppet4 for bionic -> https://tickets.puppetlabs.com/browse/PA-1869?focusedCommentId=558667&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-55866702:31
tonybianw: and the modules aren't puppet5 compatible?02:35
ianwthat's a question ... but the *install* scripts aren't 5 enabled :)02:36
*** EmilienM has quit IRC02:39
*** EmilienM has joined #openstack-infra02:39
openstackgerritIan Wienand proposed openstack-infra/system-config master: Support puppet5 for bionic  https://review.openstack.org/58900702:53
ianwtonyb: ^ i'll add info in review comments02:54
tonybianw: Thanks02:54
*** vivsoni has joined #openstack-infra02:56
ianwtonyb: ok, comment added.  i'll look at a role to install afs from ansible02:59
*** bdodd_ has joined #openstack-infra03:08
*** bdodd has quit IRC03:09
*** udesale has joined #openstack-infra03:52
tonybianw: I reall appreciate you looking at it03:55
*** d34dh0r53 has quit IRC04:00
*** cloudnull has quit IRC04:00
*** eglute_s has quit IRC04:00
*** cloudnull has joined #openstack-infra04:04
*** eglute has joined #openstack-infra04:04
*** d34dh0r53 has joined #openstack-infra04:04
*** cloudnull has quit IRC04:05
*** d34dh0r53 has quit IRC04:05
*** eglute has quit IRC04:05
*** cloudnull has joined #openstack-infra04:05
*** eglute has joined #openstack-infra04:06
*** d34dh0r53 has joined #openstack-infra04:06
*** hongbin has quit IRC04:09
*** ykarel has joined #openstack-infra04:20
*** bdodd has joined #openstack-infra04:22
*** bdodd_ has quit IRC04:24
AJaegerconfig-core, would appreciate reviews of https://review.openstack.org/588920 https://review.openstack.org/587631 https://review.openstack.org/583415 https://review.openstack.org/586749 https://review.openstack.org/583849 https://review.openstack.org/580559  https://review.openstack.org/588181 , please04:50
*** viks_ has joined #openstack-infra05:10
*** icey has quit IRC05:13
*** icey has joined #openstack-infra05:14
*** hwoarang has joined #openstack-infra05:28
*** quiquell has joined #openstack-infra05:33
*** udesale has quit IRC05:34
*** janki has joined #openstack-infra05:38
*** lathiat_ is now known as lathiat05:44
*** lathiat has quit IRC05:44
*** lathiat has joined #openstack-infra05:44
*** jtomasek has joined #openstack-infra05:58
*** mschuppert has joined #openstack-infra05:59
*** AJaeger has quit IRC06:01
*** verdurin has quit IRC06:01
*** AJaeger has joined #openstack-infra06:04
*** nicolasbock has joined #openstack-infra06:05
*** verdurin has joined #openstack-infra06:05
*** pcaruana has joined #openstack-infra06:07
*** udesale has joined #openstack-infra06:11
*** chandankumar has joined #openstack-infra06:31
*** apetrich has joined #openstack-infra06:42
*** fabo_ has quit IRC06:51
*** fabo_ has joined #openstack-infra06:51
*** fabo_ is now known as fabo06:51
*** rcernin has quit IRC06:52
*** slaweq has quit IRC06:55
*** slaweq has joined #openstack-infra06:58
*** ramishra has joined #openstack-infra07:03
*** amoralej|off is now known as amoralej07:09
*** ajo has joined #openstack-infra07:09
*** jhesketh_ has joined #openstack-infra07:10
quiquellGood morning07:11
quiquellWe have see this error at one of our reviews http://logs.openstack.org/95/583195/18/check/tripleo-ci-centos-7-containers-multinode/769edac/logs/undercloud/var/log/containers/nova/nova-scheduler.log.txt.gz#_2018-08-03_22_57_32_13807:11
quiquellDon't know if it can be caused by a infra issue07:11
*** jhesketh has quit IRC07:11
*** Bhujay has joined #openstack-infra07:21
*** ccamacho has joined #openstack-infra07:23
*** ginopc has joined #openstack-infra07:24
*** zoli is now known as zoli|wfh07:25
*** zoli|wfh is now known as zoli07:26
*** Bhujay has quit IRC07:30
*** Bhujay has joined #openstack-infra07:39
*** tosky has joined #openstack-infra07:39
*** jpenag is now known as jpena07:40
*** Emine has joined #openstack-infra07:41
openstackgerritFrank Kloeker proposed openstack-infra/project-config master: Setup two additional translation jobs for Rocky  https://review.openstack.org/58905207:45
eumel8^^07:46
eumel8AJaeger: any news for the missing translation jobs?07:49
*** dpawlik has joined #openstack-infra07:50
*** ykarel is now known as ykarel|lunch07:53
*** rpittau has joined #openstack-infra07:53
*** Bhujay has quit IRC07:54
*** dingyichen has quit IRC07:56
*** dtantsur|afk is now known as dtantsur07:59
*** gfidente has joined #openstack-infra08:00
quiquellDo we have a replacement for DEVSTACK_GATE_TIMEOUT at zuulv3 ?08:02
quiquellI see zuul.project.timeout08:03
openstackgerritFrode Nordahl proposed openstack-infra/project-config master: Enforce Py3 for `charm-glance`  https://review.openstack.org/58906708:07
*** Emine has quit IRC08:08
*** yamamoto has quit IRC08:09
*** jaosorior has joined #openstack-infra08:21
*** Emine has joined #openstack-infra08:32
*** yamamoto has joined #openstack-infra08:43
*** Bhujay has joined #openstack-infra08:45
*** olivierbourdon38 has joined #openstack-infra09:00
*** ykarel|lunch is now known as ykarel09:03
*** yamamoto has quit IRC09:15
*** olivierbourdon38 has quit IRC09:23
*** panda is now known as panda|rover|off09:25
*** olivierbourdon38 has joined #openstack-infra09:29
*** apetrich has quit IRC09:34
*** Emine has quit IRC09:36
*** roman_g has joined #openstack-infra09:41
*** adrianc has quit IRC09:43
*** yamamoto has joined #openstack-infra09:50
ssbarneaapparently, #oooq channel is not protected by the new spam measure. what do we need to include it in the list of channels that allow only registered users?09:51
*** Emine has joined #openstack-infra09:51
*** Emine has quit IRC09:52
*** Emine has joined #openstack-infra09:53
quiquellssbarnea: Cool09:56
ssbarneai found the file to modify, making a CR now,  hopefully it will work09:56
quiquellssbarnea: Puff so painful, please work :-)09:58
*** vivsoni has quit IRC10:00
openstackgerritSorin Sbarnea proposed openstack-infra/project-config master: Add #oooq channel to accesslist  https://review.openstack.org/58910310:00
AJaegereumel8: no - I cannot help further with those, you need expert help later today10:02
ssbarneaquiquell: i am not sure if we want "mask: full_mask" or not.10:03
AJaegerssbarnea: #oooq is not openstack-infra administrated10:03
AJaegerAh, I see your change...10:03
ssbarneaAJaeger: please let me know if other magic is needed to make it happen. I checked ChanServ and openstackinfra user is listed as operator on this channel, so it should be able to do its magic, right?10:04
AJaegerssbarnea: if you're op on #oooq, you can do it yourself as well...10:04
AJaegerssbarnea: we run a test on your change, if that passes, all permissions are ok;)10:05
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: Mirror the openSUSE Cloud Master repository  https://review.openstack.org/58910510:05
ssbarneai was not sure it had testing. glad to hear that. how about full_mask option?10:06
ssbarneais it needed or not really? I see very feww channels having it in the config.10:07
ianwssbarnea: have you given openstackinfra chanserver permissions?10:09
ianwssbarnea: see comments in https://review.openstack.org/589001, it might make the mask: option more clear.  it's for permissions for chanserv access for users not part of infra.  if you just want it like everything else, leave it out10:10
*** yamamoto has quit IRC10:19
*** sambetts_ has joined #openstack-infra10:23
*** sambetts_ is now known as sambetts|afk10:24
openstackgerritColleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests  https://review.openstack.org/58111210:27
ssbarneaianw: "access #oooq list" on ChanServ reported: "openstackinfra         +AFRefiorstv [modified 16w 5d 18h ago]" which makes me believe it has correct permissions.10:28
*** vivsoni has joined #openstack-infra10:29
*** hughsaunders has quit IRC10:32
*** udesale has quit IRC10:34
*** hughsaunders has joined #openstack-infra10:35
AJaegerssbarnea: tests passed, so it's fine ;)10:35
ssbarnealets merge, we got enough spam for one year :D10:36
AJaegerssbarnea: need to wait for another config-core to review https://review.openstack.org/589103  ...10:36
AJaegerssbarnea: is it that bad? Then I'll do it single core ...10:37
ssbarneai guess we can wait a little bit, ianw ^^10:38
*** dave-mccowan has joined #openstack-infra10:47
ianwssbarnea: it *should* work ... the only problem is that we re-run the access bot on config file changes, and it's been getting up towards 30 minutes and puppet kills it10:54
ianwso basically, if it's not setup in ... an hour or so, can look further10:55
*** dave-mccowan has quit IRC11:03
openstackgerritMerged openstack-infra/project-config master: Add #oooq channel to accesslist  https://review.openstack.org/58910311:06
*** jpena is now known as jpena|lunch11:14
*** zoli is now known as zoli|lunch11:22
*** rh-jelabarre has joined #openstack-infra11:33
admcleodhi there, just checking before i do, should i include a test for this: https://review.openstack.org/#/c/589125/ (i assume i can just add one with and one without scriptpath defined to some of the pre-existing tests)11:39
AJaegeradmcleod: best ask on #openstack-jjb for anything related to jjb (=jenkins job builder)11:41
admcleodAJaeger: thanks :}11:42
*** slagle has joined #openstack-infra11:49
*** jcoufal_ has joined #openstack-infra11:55
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Add tempest-ansible project under tripleo  https://review.openstack.org/58912911:56
*** boden has joined #openstack-infra11:58
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Add tempest-ansible project under tripleo  https://review.openstack.org/58912912:03
*** rosmaita has joined #openstack-infra12:03
*** radez has joined #openstack-infra12:05
*** jcoufal_ has quit IRC12:06
*** zoli|lunch is now known as zoli|wfh12:06
*** zoli|wfh is now known as zoli12:06
*** jcoufal_ has joined #openstack-infra12:07
*** e0ne has joined #openstack-infra12:08
*** jcoufal_ has quit IRC12:10
*** jcoufal_ has joined #openstack-infra12:10
*** yamamoto has joined #openstack-infra12:11
*** e0ne has quit IRC12:13
*** dhill_ has quit IRC12:15
*** jpena|lunch is now known as jpena12:18
*** vivsoni has quit IRC12:21
*** jiapei has joined #openstack-infra12:22
*** vivsoni has joined #openstack-infra12:22
*** vivsoni has quit IRC12:23
*** vivsoni_ has joined #openstack-infra12:23
*** quiquell is now known as quiquell|lunch12:32
*** dtantsur is now known as dtantsur|brb12:36
*** trown|brb is now known as trown12:38
*** tpsilva has joined #openstack-infra12:39
*** Sigyn has quit IRC12:41
*** Sigyn has joined #openstack-infra12:42
*** agopi has quit IRC12:43
*** hs366 has joined #openstack-infra12:47
*** fabo has quit IRC12:49
*** yamamoto has quit IRC12:50
*** mdrabe has joined #openstack-infra12:50
*** dave-mccowan has joined #openstack-infra12:52
*** apetrich has joined #openstack-infra12:52
*** quiquell|lunch is now known as quiquell|off12:57
*** panda|rover|off is now known as panda|rover-ish12:58
*** eharney has joined #openstack-infra13:00
*** quiquell|off has quit IRC13:01
*** dave-mccowan has quit IRC13:03
*** kgiusti has joined #openstack-infra13:05
*** agopi has joined #openstack-infra13:08
*** agopi_ has joined #openstack-infra13:08
*** rosmaita has quit IRC13:10
*** agopi has quit IRC13:12
*** edmondsw has joined #openstack-infra13:13
*** jroll has quit IRC13:19
*** jroll has joined #openstack-infra13:19
*** jcoufal has joined #openstack-infra13:21
*** rosmaita has joined #openstack-infra13:23
*** janki has quit IRC13:24
*** agopi_ is now known as agopi13:29
*** jcoufal has quit IRC13:30
*** jcoufal_ has quit IRC13:30
*** jcoufal has joined #openstack-infra13:30
*** Bhujay has quit IRC13:31
*** olivierbourdon38 has quit IRC13:38
*** mriedem has joined #openstack-infra13:41
*** olivierbourdon38 has joined #openstack-infra13:41
*** MasterofJOKers has quit IRC13:41
*** MasterofJOKers has joined #openstack-infra13:43
*** lbragstad has joined #openstack-infra13:43
*** olivierbourdon38 has quit IRC13:44
*** olivierbourdon38 has joined #openstack-infra13:44
*** olivierbourdon38 has quit IRC13:46
*** olivierbourdon38 has joined #openstack-infra13:47
*** clarkb has joined #openstack-infra13:47
*** olivierbourdon38 has quit IRC13:47
*** zoli is now known as zoli|afk13:51
*** olivierbourdon38 has joined #openstack-infra13:59
*** gary_perkins has joined #openstack-infra14:04
*** spotz has joined #openstack-infra14:08
*** efried1 has joined #openstack-infra14:16
*** efried has quit IRC14:16
*** efried1 is now known as efried14:16
*** Bhujay has joined #openstack-infra14:18
*** apetrich has quit IRC14:19
*** kklimonda_ has joined #openstack-infra14:23
*** Tahvok_ has joined #openstack-infra14:23
*** kklimonda has quit IRC14:26
*** Tahvok has quit IRC14:26
*** zxiiro has quit IRC14:26
*** eventingmonkey has quit IRC14:26
*** smcginnis[m] has quit IRC14:26
*** Tengu has quit IRC14:26
*** szaher has quit IRC14:26
*** quite has quit IRC14:26
*** pfallenop has quit IRC14:26
*** dtantsur|brb has quit IRC14:26
*** Tahvok_ is now known as Tahvok14:26
*** kklimonda_ is now known as kklimonda14:26
*** eventingmonkey has joined #openstack-infra14:27
*** eharney has quit IRC14:28
*** dtantsur has joined #openstack-infra14:28
*** Bhujay has quit IRC14:29
*** calbers has quit IRC14:30
*** dpawlik has quit IRC14:31
*** codenix has quit IRC14:31
*** calbers has joined #openstack-infra14:31
*** zxiiro has joined #openstack-infra14:38
*** Tengu has joined #openstack-infra14:38
*** zoli|afk is now known as zoli14:38
*** zoli is now known as zoli|wfh14:39
*** e0ne has joined #openstack-infra14:43
*** eglute has quit IRC14:45
*** bobh has joined #openstack-infra14:45
*** yamamoto has joined #openstack-infra14:47
*** eglute has joined #openstack-infra14:49
*** jcoufal_ has joined #openstack-infra14:50
*** jcoufal_ has quit IRC14:50
*** jiapei has quit IRC14:51
*** gyee has joined #openstack-infra14:52
dhellmannclarkb : hey, I have ~130 patches ready to go to move the zuul job settings for Infra repos. When do you want those, relative to the PTG and other things the team is working on?14:53
*** jiapei has joined #openstack-infra14:53
AJaegerconfig-core, I would appreciate reviews of https://review.openstack.org/588920 https://review.openstack.org/587631 https://review.openstack.org/583415 https://review.openstack.org/586749 https://review.openstack.org/583849 https://review.openstack.org/580559  https://review.openstack.org/588181 , please14:54
*** dave-mccowan has joined #openstack-infra14:58
clarkbdhellmann: we tend to try avoiding making operational changes (to services directly) when RCs and stuff happen for openstack. That might be a good time to work through some refactoring14:59
dhellmannclarkb : yeah, I figured this week would be bad. do you want them at all before the ptg?14:59
dhellmannif not I can scrap these and regenerate them later so they're up to date14:59
*** rpioso|afk is now known as rpioso15:00
fungiscas: i've pushed all the branches and tags from openstack/openstack-chef-repo to openstack/openstack-chef so take a look at let me know if it's what you're expecting15:00
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add information to Ref and TimerTrigger repr  https://review.openstack.org/58918415:01
clarkbdhellmann: before ptg is probably fine, chances are many of us will pck off reviews like that during quiet times between now and then15:01
clarkbdhellmann: using a common topic for the infra change swill likely also help with that15:02
fungichange swill is such a great term15:02
dhellmannclarkb : I'm proposing them all with python3-first to attach them to the goal15:02
corvuseumel8, AJaeger: ^ it's not clear from the logs what's happening with openstack-manuals propose-translation-updates.  i've proposed a change to zuul to add more debug info.15:03
clarkbdhellmann: we can probably filter most of them with project:^openstack-infra/.* against that topic15:03
dhellmannyeah, that would likely work15:03
AJaegercorvus: thanks for following up on our discussion from yesterday.15:03
dhellmannfungi : you didn't find anything else about why those patches weren't triggering updates to storyboard, did you?15:03
AJaegercorvus: it'S also all other repos where we have stable/rocky set up15:03
corvusAJaeger: fascinating, that's useful info15:04
corvusAJaeger: was july 18 the branch point for rocky?15:04
scasfungi: i have some ancillary changes for finalizing the rehoming, but the history looks as i'm expecting. tanks nice man15:04
corvus(for manuals)15:04
AJaegercorvus: for manuals: We have no rocky15:04
AJaegercorvus: not sure when the other repos created their stable branches15:05
fungidhellmann: no, i wasn't in a great position to experiment with that over the weekend. basically the gerrit logs are entirely silent (other than noting the patches were pushed and verified review comments from zuul and fetches from zuul mergers...). normally gerrit logs every interaction it makes with storyboard and i see it logging similar interactions for other reviews both before and after the one15:05
fungiyou linked15:05
dhellmannclarkb : I've been thinking that we want to do some sort of coordination so that while the job setting migrations are in motion we don't change things in project-config to make something obsolete. Do you have any ideas about how to do that?15:06
dhellmannfungi : could it have had to do with volume or something? submitting so many patches together?15:06
clarkbdhellmann: we could add a temporary rule to the linter thing AJaeger wrote that would fail additive changes to project-config (then remove it when we think the migration is reasonably complete)15:07
AJaegercorvus: stable/rocky branches seem to have been created 26th July15:07
dhellmannclarkb : would that apply to everyone? or just to a team with the changes in progress?15:08
clarkbdhellmann: it would apply to everyone15:08
dhellmannok15:09
corvusdhellmann, clarkb: perhaps we could just let the reviewers know?15:09
dhellmannyeah, I would be OK with that, too15:10
clarkbjillr: on https://review.openstack.org/#/c/583415/6 the new repo uses tripleo.config instead of tripleo-ansible.config for gerrit acl config. Is that intentional?15:10
corvuslike "no additive changes" seems like a pretty easy thing for reviewers to deal with (and then we don't have to deal with writing exceptions to the linter)15:10
*** eharney has joined #openstack-infra15:10
dhellmannI had originally thought we would hold off proposing the patch to remove settings from project-config until the others had landed, but maybe we want to propose it so it shows up in the "conflicts with" list in gerrit15:10
clarkbcorvus: dhellmann ya if we want to keep a running update for which projects are migrating on the infra list that could work too15:10
dhellmannwhere is that list?15:11
clarkbdhellmann: sorry, the openstack-infra mailing list15:11
dhellmannah15:11
fungidhellmann: not sure regarding volume being related. an easy test would be to rebase one and see if a comment for it appears in sb (there would normally be a comment for every new patch set mentioning the story in its commit message)15:11
dhellmannfungi : I edited a couple based on AJaeger's feedback and they did not15:12
*** pcaruana has quit IRC15:12
dhellmannit doesn't do any sort of matching to verify that a task for the given project exists, does it?15:12
dhellmanneven if there's no task id in the message15:12
clarkbchecking in on packethost we now have non zero error count launching nodes, but the mirror is still happily running15:13
*** e0ne has quit IRC15:14
dhellmannclarkb , corvus : in the interest of reducing the number of places we have to track status, how about if we use the project-config reviews to indicate that a team has started the migration (a patch to remove their settings with W-1) or is done (a patch to remove their settings without W-1)?15:15
clarkbdhellmann: I don't know that we would poll a list of reviews and read through all the diffs to generate that list in our heads15:16
clarkbdhellmann: I think most of us tend to read mailing list email, but don't necessarily scan the list of reviews before picking off next review in the queue15:16
dhellmannhmm, ok15:17
*** janki has joined #openstack-infra15:17
*** eharney has quit IRC15:18
AJaegerdhellmann: I consider the project-config one nice so that I can check for odd problems, like the moving of legacy jobs15:21
dhellmannAJaeger : so you want us to go ahead and propose those patches early?15:21
*** eharney has joined #openstack-infra15:21
AJaegerdhellmann: yes, that's my personal preference.15:22
dhellmannok15:22
cmurphyhey clarkb, i see my gerrit email folder is overflowing, any puppet4 highlights from last week?15:22
openstackgerritMerged openstack-infra/project-config master: Add releasenote job to python-blazarclient  https://review.openstack.org/58055915:22
openstackgerritMerged openstack-infra/project-config master: Move ironic-tempest job for Neutron to "in tree"  https://review.openstack.org/58818115:22
clarkbcmurphy: ya, I got a few things merged and mordred helped out with reviews. Uh ask-staging is in the middle of migrating to ask-staging01 so we sort of hung up on being able to test that future parser migraiton15:23
clarkbcmurphy: also I left a few comments on some changes let me get links15:23
cmurphyclarkb: yeah i noticed i couldn't reach ask-staging.o.o and i wasn't sure if that was expected?15:23
*** olivierbourdon38 has quit IRC15:24
cmurphyor if some iptables got mangled15:24
clarkbcmurphy: I think ianw was working on that, we just need to followup with where it is and possibly help get it properly deployed15:24
cmurphyah ok15:25
clarkbcmurphy: https://review.openstack.org/#/c/581335/1 you saw comment there. With the *bot rspec change updates one concern I had was I think many(most/all) will connect to freenode15:25
clarkbcmurphy: and we have had to +r our channels due to spamming, so might want to avoid having test bots spam real channels as well15:26
cmurphyhaha good point15:26
cmurphythere is at least one other module where it makes a real connection to a real channel15:26
*** hamerins has joined #openstack-infra15:27
*** jcoufal has quit IRC15:27
*** yamamoto has quit IRC15:27
clarkbcmurphy: then the other big change is we (mostly mordred) have deployed bridge.openstack.org to replace puppetmaster.openstack.org. It is a puppetless bastion host that will either do ansible + puppet on remote nodes or just asnible per the new spec15:27
*** ykarel has quit IRC15:28
clarkbcmurphy: I don't think we need to do anything specific around puppet-4 for this, mostly just an fyi that this is happening. Part of the motivation is that puppetmaster is trusty and we needed a newer host anyway15:28
cmurphysounds good15:28
fungidhellmann: the its-storyboard plug-in for gerrit is _supposed_ to have basically two independent actions it can be called on to perform. one is to act on story footers (leave a comment on the story linking back to the change for review) and on task footers (adjust the state of the task based on the state of the change under review) and as far as i'm aware those are called by totally separate calls15:28
fungiinto the plug-in so lack of one shouldn't influence the other. we can test that hypothesis by amending the commit message to include a task footer i suppose15:28
*** jcoufal has joined #openstack-infra15:28
*** jcoufal_ has joined #openstack-infra15:29
*** jcoufal has quit IRC15:29
dhellmannfungi : done for https://review.openstack.org/58881015:30
dhellmannand the story was indeed updated15:31
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Remove legacy-muranoclient-dsvm-functional-mysql-backend for muranoclient  https://review.openstack.org/58674915:31
clarkbcmurphy: we did reorganize the hierdata dirs from groups/ fqdn/ to host-vars/ group-vars/ so that ansible could load them too15:31
cmurphyclarkb: just on bridge or on puppetmaster too?15:31
AJaegerclarkb: thanks for reviewing!15:32
clarkbcmurphy: both15:32
clarkbAJaeger: I skipped those that had depends on that were not merged15:32
cmurphyclarkb: ah interesting15:32
AJaegerclarkb: sorry for including those ;(15:32
*** jcoufal has joined #openstack-infra15:32
*** jcoufal has quit IRC15:32
cmurphyclarkb: how much of bridge is ninja'd versus committed in system-config? i glanced at the recent changes in system-config and the bridge playbooks seem a bit minimal15:32
cmurphymordred: ^15:33
clarkbcmurphy: I think we got the base changes merged that you could spin up a new one15:33
clarkbcmurphy: but its still a work in progress until bridge.o.o is ansibling things15:33
clarkbcmurphy: there are launch node updates and base playbooks to configure the host15:34
clarkba good exercise would probably be to replace it once we've got the playbooks in a happy spot using only those playbooks though15:34
cmurphy++15:35
fungidhellmann: i'm starting to wonder if our its rules are incorrect... the comment-on-status-update rule is supposed to trigger an add-standard-comment action on patchset-created but it's possible it's not getting triggered by story footers https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n20515:36
mordredcmurphy, clarkb : yeah it's been being MOSTLY stood up by the patches in that stack15:36
mordredthere are a few chicken-and-egg issues that are mostly related to the secrets that aren't fully ansibled15:36
mordredso - we COULD totally write just a little bit more code to have the current bridge able to set up a new bridge15:37
fungidhellmann: we have two commentlinks rules related to storyboard, "story" (for story tags) and "its-storyboard" (for task tags) https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n20515:37
AJaegercorvus, so for the translation jobs: Merge https://review.openstack.org/589184 and restart zuul would be the next steps, correct?15:38
mordredby and large though, I have avoided installing things or setting things up without using the ansible in that stack15:38
fungidhellmann: there may be an implicit interaction there where the commentlink rule has to match the plug-in name, which would explain the current behavior if so (gerrit 2.13 totally changed around the way its rules are configured, so we had to redo this after the last upgrade)15:38
mordredcmurphy: https://review.openstack.org/#/c/588326/ is definitely worth a look15:38
mordredthat's the first stab at making sure the base ansible playbook gets run before we run puppet on nodes so that our integration tests will have all the pieces15:39
fungidhellmann: i'll add to my to do list to experiment with a few possible solutions with review-dev/storyboard-dev15:39
cmurphymordred: neat, i hadn't looked at the open stack yet, only at what was landed15:39
corvusAJaeger: yes; at the moment i don't have enough to go on to dig further15:40
dhellmannfungi : ok, thanks. I'm not familiar enough with the gerrit side of this to understand the rules in that review.pp file15:40
fungidhellmann: they're pretty magic from my perspective too, unfortunately15:40
mordredcmurphy: cool. yeah - the open stack is where most of the fun is ... https://review.openstack.org/#/c/588394 needs to land before we can run more of the playbooks - since the next step would be to run the playbook that does the modules install15:41
fungidhellmann: the documentation for how gerrit its plug-ins work is sparse at best15:41
* dhellmann nods15:41
fungieven zaro had a tough time explaining how this works, and he wrote the its-storyboard plug-in15:42
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add information to Ref and TimerTrigger repr  https://review.openstack.org/58918415:42
toskydear infra, we (tempestconf) are hit by a timeout error on a test which seems to be happen only on ovh nodes, while rax and packethost work15:44
toskycompare for example the failed and passed jobs on https://review.openstack.org/#/c/580981/15:44
toskyor the first run and the recheck for https://review.openstack.org/#/c/588593/15:45
toskydoes it ring any bell?15:45
clarkbtosky: it helps if you can link to specific job logs15:46
*** ginopc has quit IRC15:46
toskyfailures: http://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/d943583/zuul-info/15:47
toskyhttp://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/b0e0792/zuul-info/15:47
toskyworking: http://logs.openstack.org/81/580981/7/check/python-tempestconf-tempest-packstack-admin/27d3863/15:47
toskyand http://logs.openstack.org/81/580981/7/check/python-tempestconf-tempest-packstack-admin/d729479/15:48
*** jaosorior has quit IRC15:48
fungitosky: when you say "timeout error" here it looks like you mean an ssh timeout reaching into a cirros instance started by the test?15:49
clarkbtosky: http://logs.openstack.org/81/580981/7/gate/python-tempestconf-tempest-packstack-admin/d943583/job-output.txt.gz#_2018-08-06_14_07_31_997944 looks like metadata service failed to provide the ssh key then ssh authentication failed because the ssh key wasn't configured15:49
toskyfungi: yes, that one15:49
fungi"SSHTimeout: Connection to the 172.24.4.21 via SSH timed out."15:49
toskythe same job works on other providers15:49
*** jaosorior has joined #openstack-infra15:49
toskyanother example from a different review: failure: http://logs.openstack.org/93/588593/1/check/python-tempestconf-tempest-packstack-admin/5040efe/15:51
toskyand working: http://logs.openstack.org/93/588593/1/check/python-tempestconf-tempest-packstack-admin/b8fdc81/15:51
fungii don't see any possible conflicting addresses/routes on the node where one of those ran, so rules out my first theory15:52
clarkbfungi: tosky I don't think this is a failing of the provider, neutron metadata service didn't do its job15:52
*** Emine has quit IRC15:53
dhellmannclarkb, corvus : how about something like this as an announcement? http://lists.openstack.org/pipermail/openstack-infra/2018-August/006039.html15:55
*** apetrich has joined #openstack-infra15:55
dhellmann(I'm trying to get this correct so I can write instructions for champions so I don't have to do all of these patches myself)15:55
toskyclarkb: but the root cause for that may still related to the provider15:56
toskysize of instances, network, etc15:56
clarkbtosky: it could be, but I think we need to identify why metadata service failed before we make that leap15:56
corvusdhellmann: oh, it's not all at once?  i thought it was like "just don't approve changes which add stuff to projects for a while"15:57
*** dklyle has joined #openstack-infra15:57
clarkbtosky: is this a single node job? if so then networking shouldn't matter its all localhost anyway15:57
toskyclarkb: single node job15:58
dhellmanncorvus : well, I didn't want to assume that all of them would be proposed together all at once15:58
*** efried has quit IRC15:58
dhellmanncorvus : if you want to do it all at once, we could just pick a start date for that freeze15:58
dhellmannalthough I expect proposing the patches to take a few days15:59
clarkbtosky: in any case I wouldn't make the jump that this is a provider problem just yet. We need to understand why metadata service failed to provide the ssh public key to the instance15:59
corvusdhellmann: AJaeger is the most prolific project-config reviewer, i'm happy with whatever works for him16:00
jillrclarkb: no, that is not intended - thank you will update.16:01
clarkbdhellmann: yup I think that should work thanks16:05
dhellmannI keep anticipating this being a huge number of patches that we're going to need time to absorb. Let me generate them all and see what we actually have.16:06
openstackgerritJill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo  https://review.openstack.org/58341516:06
*** efried has joined #openstack-infra16:10
*** ramishra has quit IRC16:10
*** jpena is now known as jpena|off16:12
*** janki has quit IRC16:15
*** bobh has quit IRC16:18
*** SamYaple has quit IRC16:37
*** SamYaple has joined #openstack-infra16:39
*** betherly_ is now known as betherly-afk16:40
*** e0ne has joined #openstack-infra16:47
fungiheaded out to get some lunch, back soon16:50
*** evrardjp has quit IRC16:51
*** hongbin has joined #openstack-infra16:52
AJaegerinfra-root, could you retire stable/newton for tripleo, please? the change for eol tagging has been merged, tonyb has details on what to do in http://lists.openstack.org/pipermail/openstack-dev/2018-July/132346.html . mwhahaha is ready to go...16:53
AJaegerinfra-root, side effect of this: fixes remaining error in http://zuul.openstack.org/api/config-errors16:53
*** bobh has joined #openstack-infra16:55
*** olivierbourdon38 has joined #openstack-infra16:56
clarkbAJaeger: do you know where that eol_branch scrpt lives?16:56
openstackgerritJames E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines  https://review.openstack.org/58838316:56
corvusclarkb: ^ that has the severity filter actually filter out debug lines so crm114 (and any future processors) don't see them16:57
*** apetrich has quit IRC16:57
AJaegerclarkb: good questions. I hope ianw or jhesketh_ know where eol_branch.sh lives16:58
AJaegeror tonyb ^16:58
openstackgerritJames E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines  https://review.openstack.org/58838316:58
AJaegerclarkb: Running find right now, codesearch did not help ;(16:59
clarkbcorvus: one comment on ps2, a cleanup that was missed in the refactor16:59
corvuson it17:00
openstackgerritJames E. Blair proposed openstack-infra/puppet-log_processor master: Add severity info to logstash and filter out DEBUG lines  https://review.openstack.org/58838317:00
corvusclarkb: i think that should have all the fixes now :)17:00
clarkbyup lgtm17:01
*** bobh has quit IRC17:02
*** bobh has joined #openstack-infra17:02
AJaegerclarkb: https://github.com/openstack-infra/release-tools/commit/39a82df080dda0b3028be623f016354ff0544520#diff-6b15029c01f7d1bce80961f6a7ea121e17:02
*** amoralej is now known as amoralej|off17:03
AJaegerclarkb: we retired release-tools repo and that repo has the script in it.17:03
AJaegerNO idea where we moved it afterwards ;(17:03
clarkbinteresting, dhellmann may know?17:03
AJaegerso, script is https://github.com/openstack-infra/release-tools/blob/0abf645cc36aadce365ba08328e98864272d64ed/eol_branch.sh17:03
AJaegerclarkb, dhellmann I found no reviews for it either. dhellmann, smcginnis, what's a good place to move eol_branch.sh to?17:05
*** hemna_ has joined #openstack-infra17:06
smcginnisHmm, maybe under openstack/releases in the tools directory? Although that doesn't really look like a "release" thing per se.17:07
*** bobh has quit IRC17:07
AJaegerIt's run by infra at EOL. clarkb, do we have a good place in infra instead?17:07
clarkbAJaeger: tonyb normally runs it fwiw, we have escalated his gerrit perms for him to run that script17:08
AJaegerclarkb: ah17:09
dhellmannAJaeger : you could put it in project-config with the rest of the release tool scripts, since we're going to want a copy there when we can automate this step anyway17:10
dhellmannotherwise, in releases/tools like smcginnis suggested would be good17:10
dhellmannI think we didn't realize any more of those scripts were necessary, :-/17:10
AJaegerno worries, we found it ;)17:10
smcginnisIf we can automate it evetually, project-config does sound like a good place then. I think a little more appropriate in the infra domain than the releases domain.17:11
AJaegerok17:14
corvusmordred: were you going to drop the "ansible_python_interpreter: python3" setting on bridge from https://review.openstack.org/587985 ?17:14
corvusmordred: is 'with_first_found' going away? i actually really like 'with_first_found' -- it makes perfect sense as a short-circuited while loop.  the replacement is a lot of boilerplate for the same thing.  :|17:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Import eol_branch.sh  https://review.openstack.org/58923617:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove eol tagging from eol_branch.sh  https://review.openstack.org/58923717:19
AJaegertonyb, smcginnis, dhellmann, clarkb ^17:19
*** zoli|wfh is now known as zoli|gone17:19
*** zoli|gone is now known as zoli17:19
corvusmordred: related to that, i left a -1 for an important question on https://review.openstack.org/58854617:20
clarkbAJaeger: thanks will take a look shortly.17:21
corvusclarkb, mordred: question on https://review.openstack.org/58855317:22
AJaegerclarkb: shall I ask tonyb to coordinate with infra-root? Then I'll send an email out...17:22
clarkbAJaeger: yes I think we should dobule check that version of the script is up to date (via tonyb) and then we can run it for tonyb if that is easier or bump tonyb's gerrit perms so that he can run it17:23
AJaegerclarkb: ok, will write email in case he does not read here17:24
corvushttps://review.openstack.org/588626 failed the puppet4-apply test on centos-7 -- is that issue known?17:24
*** yamamoto has joined #openstack-infra17:24
clarkbcorvus: yes it is the ssh keys issue 2018-08-03 17:44:19 +0000 Puppet (err): Failed to apply catalog: Could not parse line "HijxEUeWHhFJDIVRGUf0Ef0nrmBv fungi-openstack-2015" at /root/.ssh/authorized_keys:27 from http://logs.openstack.org/26/588626/1/check/infra-puppet-apply-4-centos-7/ff37fb3/applytest/puppetapplytest27.final.out.FAILED17:28
clarkbthis is not puppet 4 specific and has happened since we moved to adding root keys with glean on the test images17:28
corvusclarkb: i'm not up to speed on that; is there some backlog i should look for?17:28
clarkbcorvus: https://review.openstack.org/#/c/501887/2 was where I last tried to poke at it. cmurphy was looking at it more recently too iirc, not sure if there was anything new learned17:30
clarkbI think ianw had a glean change too /me looks for that17:30
corvusclarkb: is this a non-deterministic failure?17:30
clarkbcorvus: we don't know yet why it happens but it isn't consistent17:31
clarkbif you recheck chances are it iwll pass17:31
clarkbhttps://review.openstack.org/#/c/585574/ is the glean change ianw wrote that was related to this17:31
corvusclarkb: 501887 failed but the logs have beend deleted :(17:31
clarkbcorvus: ya :/17:31
clarkbcorvus: it also only seems to affect centos17:32
mordredcorvus: I am not sure for sure if it's going away going away, but I *believe* that is the intent. I think the reason they're wanting to make it go away is that it's not actually a loop but looks like one to people - so apparently it seems like ansible users get confused by it17:37
mordredcorvus: however, there's definitely a bug in that patch that your -1 points out17:37
*** dhill_ has joined #openstack-infra17:38
*** olivierbourdon38 has quit IRC17:40
*** evrardjp has joined #openstack-infra17:41
*** vivsoni_ has quit IRC17:44
*** vivsoni_ has joined #openstack-infra17:44
*** psachin has quit IRC17:45
*** bobh has joined #openstack-infra17:48
openstackgerritMerged openstack-infra/zuul master: Add information to Ref and TimerTrigger repr  https://review.openstack.org/58918417:49
clarkbmordred: did you see that openstacksdk for the task wait thing merged?17:50
clarkb(I think next step is making a bug fix release?)17:50
*** jiapei has quit IRC17:51
*** dtantsur is now known as dtantsur|afk17:53
clarkblooks like limestone has gone into hibernation mode again. I'm going to restart the nodepool launcher on nl02 to get it going again17:57
mordredclarkb: yes - I pushed up a backport of it to stable/rocky and that is winding through the gate right now. I'll push up a release patch as son as it lands17:57
*** yamamoto has quit IRC17:57
clarkblauncher is restarted17:58
clarkbthe git sha1 has not changed17:59
eumel8corvus: just to mention that are only 15 periodic jobs for propose-translation-update are running. Since 07/18 and especially the last week: http://status.openstack.org/openstack-health/#/?searchProject=propose-translation-update&duration=P1M&end=2018-08-06T18:00:00.000Z&groupKey=build_name18:03
eumel8looks like a limit18:03
*** e0ne has quit IRC18:05
*** jcoufal_ has quit IRC18:05
*** trown is now known as trown|lunch18:06
clarkbfollowup on the zuul executor kernel updates: they all seem to run a similar number of jobs now, probably a reasonable assumption to infer the kernel did improve the performance18:08
*** roman_g has quit IRC18:09
mordredclarkb: ++18:09
clarkbeumel8: I don't think we added any pipeline limits like that. Did we happen to remove jobs from that pipeline maybe?18:13
clarkbAJaeger: ^ you may know18:13
*** e0ne has joined #openstack-infra18:14
AJaegerclarkb: we have a limit - let me grab it...18:14
AJaegerclarkb: a semaphore to run only 10 jobs, see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n121618:15
clarkboh interesting18:15
AJaegerclarkb: but that is supposed to limit the number of *simulantaneous* running jobs18:15
AJaegereumel8: that health report is broken, count http://zuul.openstack.org/builds.html?job_name=propose-translation-update18:16
AJaegereumel8: ok, 16 not 15 for today ;)18:17
AJaegerand 16 yesterday18:17
AJaegercorvus: ^18:17
eumel8AJaeger: nice shot, but there are  more then 10 in parallel :)18:18
AJaegereumel8: not in parallel, check starting times and lenght of job18:19
AJaegerlength is around 5 minutes, and we have starting times between 06:00 and 06:1018:19
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Adeed endpoint  https://review.openstack.org/58925418:20
AJaegereumel8: https://review.openstack.org/589184 merged to debug this - now corvus needs to restart zuul to use the change...18:20
eumel8AJaeger: saw it. my expectition is, it works after restart :)18:22
*** dhill_ has quit IRC18:22
AJaeger,)18:22
*** gfidente is now known as gfidente|afk18:23
*** bobh has quit IRC18:23
*** bobh has joined #openstack-infra18:23
*** nicolasbock has quit IRC18:23
*** nicolasbock has joined #openstack-infra18:24
mordredclarkb, corvus: btw - I'm focused on an infinite-recursion bug I found in sdk over the weekend - I've finally got it isolated down to a single line reproduction, now I just have to fix it18:24
mordredbut I'm going to finish fixing it before I push further on bridge18:24
clarkbmordred: if that gets us fix for limestone releaesd then I'm happy :)18:25
mordredah - those are unrelated- limestone fix is just waiting on the stable patch to land - I'll get that cut as soon as it does18:25
clarkbah thought you might want to bundle bug fix relases18:25
mordrednah. justhave mental state on the recursion problem that I don't want to lose - because, well - recursion problem18:29
mordredcorvus: also - my original reply on your comment for https://review.openstack.org/#/c/588546 was bogus (I was reading poorly) - I have left a followup comment18:34
* mordred goes to find sandwich18:34
corvusmordred: ok thanks.  i looked in the docs for first_found and didn't see a mention of skip.18:48
*** numans has quit IRC18:54
clarkbfungi: not sure if you noticed but ianw wrote a letsencrypt spec for us, you probably want to take a look given prior interest https://review.openstack.org/#/c/587283/18:58
*** dhill_ has joined #openstack-infra18:59
fungiyep, it's on my reading list19:05
fungithanks!19:06
*** yamamoto has joined #openstack-infra19:17
*** trown|lunch is now known as trown19:17
openstackgerritMerged openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route  https://review.openstack.org/55097819:27
*** yamamoto has quit IRC19:31
*** yamamoto has joined #openstack-infra19:36
*** yamamoto has quit IRC19:47
*** jcoufal_ has joined #openstack-infra19:49
*** jcoufal_ has joined #openstack-infra19:51
*** jcoufal_ has quit IRC19:51
*** dbecker has joined #openstack-infra20:00
openstackgerritColleen Murphy proposed openstack-infra/puppet-subunit2sql master: Fix subunit2sql install guard for idempotency  https://review.openstack.org/58133520:05
openstackgerritColleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests  https://review.openstack.org/58111220:05
openstackgerritColleen Murphy proposed openstack-infra/puppet-subunit2sql master: Update Gemfile for Zuulv3  https://review.openstack.org/58134720:07
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Update Gemfile for Zuulv3  https://review.openstack.org/57752120:08
*** e0ne has quit IRC20:11
openstackgerritColleen Murphy proposed openstack-infra/puppet-log_processor master: Add beaker tests  https://review.openstack.org/58169720:12
*** Guest74859 has joined #openstack-infra20:19
*** hamerins has quit IRC20:23
*** rtjure has joined #openstack-infra20:31
*** kgiusti has left #openstack-infra20:31
*** eharney has quit IRC20:39
*** trown is now known as trown|outtypewww20:49
*** bobh has quit IRC20:55
clarkbcmurphy: https://review.openstack.org/#/c/581112/7 still seems to depends on itself20:59
cmurphybah21:00
cmurphygit sucks21:00
openstackgerritColleen Murphy proposed openstack-infra/puppet-subunit2sql master: Add beaker tests  https://review.openstack.org/58111221:01
*** rockyg has joined #openstack-infra21:06
openstackgerritMerged openstack-infra/openstackid-resources master: Adeed endpoint  https://review.openstack.org/58925421:12
*** edmondsw has quit IRC21:15
openstackgerritMerged openstack-infra/puppet-subunit2sql master: Update Gemfile for Zuulv3  https://review.openstack.org/58134721:20
*** harlowja has joined #openstack-infra21:21
openstackgerritMerged openstack-infra/openstackid master: Fixed typo on logout from IDP  https://review.openstack.org/58840221:22
*** hwoarang has quit IRC21:24
*** bobh has joined #openstack-infra21:28
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added missing permission to endpoint v2  https://review.openstack.org/58929121:31
openstackgerritMerged openstack-infra/openstackid-resources master: Added missing permission to endpoint v2  https://review.openstack.org/58929121:32
*** bobh has quit IRC21:35
*** rockyg has quit IRC21:36
*** yamamoto has joined #openstack-infra21:41
*** bobh has joined #openstack-infra21:47
openstackgerritMerged openstack-infra/project-config master: Remove Ansible function jobs from cloud-launcher  https://review.openstack.org/58859921:49
openstackgerritMerged openstack-infra/project-config master: gerrit-dash-creator: revert to readthedocs template  https://review.openstack.org/58841021:49
*** yamamoto has quit IRC21:51
*** slagle has quit IRC21:53
openstackgerritMerged openstack-infra/project-config master: Switch to publish-openstack-docs-pti for murano  https://review.openstack.org/58673821:54
*** boden has quit IRC21:56
*** boden has joined #openstack-infra21:57
*** boden has quit IRC21:57
*** boden has joined #openstack-infra21:57
*** boden has quit IRC21:57
*** boden has joined #openstack-infra21:57
*** boden has quit IRC21:58
*** boden has joined #openstack-infra21:58
*** boden has quit IRC21:59
clarkbjillr: bah http://logs.openstack.org/15/583415/7/check/project-config-gerrit/0785cec/job-output.txt.gz#_2018-08-06_16_17_59_747628 the linter check didn't like that change, we need to remove the acl config line entirely since the default is to use the config file named after the project21:59
*** boden has joined #openstack-infra21:59
*** boden has quit IRC21:59
*** boden has joined #openstack-infra22:00
*** boden has quit IRC22:00
*** boden has joined #openstack-infra22:00
*** boden has quit IRC22:01
*** boden has joined #openstack-infra22:01
*** boden has quit IRC22:02
*** boden has joined #openstack-infra22:02
*** bobh has quit IRC22:02
*** boden has quit IRC22:02
*** bobh has joined #openstack-infra22:03
*** boden has joined #openstack-infra22:03
*** boden has quit IRC22:03
*** bobh has quit IRC22:07
jillrclarkb: ack, will do22:10
openstackgerritMerged openstack-infra/project-config master: Add publish-to-pypi for cinder-tempest-plugin  https://review.openstack.org/58762322:12
openstackgerritJill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo  https://review.openstack.org/58341522:14
*** rcernin has joined #openstack-infra22:15
openstackgerritMerged openstack-infra/project-config master: Enforce Py3 for `charm-glance`  https://review.openstack.org/58906722:17
openstackgerritMerged openstack-infra/project-config master: Setup two additional translation jobs for Rocky  https://review.openstack.org/58905222:17
corvusclarkb, fungi: any idea how to get an http password for a gerrit system account (eg, "zuul")?22:20
clarkbcorvus: it is in the account external id's database table, though I'm not sure if we need to trigger generating that somehow (or if there is a more human friendly way to get that22:20
corvusapparently you can now set an "--http-password" on the "gerrit create-account" command....22:21
corvusooh!22:21
corvushttps://review.openstack.org/Documentation/cmd-set-account.html22:21
corvusset-account has an --http-password option22:21
clarkbTIL22:22
corvusso i'm going to run "gerrit set-account zuul --http-password $SOMETHING"  sound good?22:22
clarkbyup22:22
clarkbassuming zuul is the username22:22
corvuspretty sure it is22:22
corvusdone22:24
corvusadding to hiera now22:24
*** nicolasbock has quit IRC22:25
corvuscommitted to puppetmaster and bridge22:26
fungiyep, that's what i was going to suggest22:26
fungieither set-account --http-password or i think you can do something similar via the rest api22:27
fungihttps://review.openstack.org/Documentation/rest-api-accounts.html#set-http-password22:28
fungiyou can either provide a pregenerated password or ask it to generate and return one in the response22:29
corvusthis is a bit of a one-off, so i find the ssh api much easier to use22:29
corvusso i just used pwgen -s22:29
fungiyeah, i totally agree22:29
corvus(with some suitable number following the -s :)22:29
fungii tend to use the ssh api for most stuff because it's easier than all the setup needed to make rest api calls manually22:30
corvushrm, the zuul docs don't seem to be up to date: https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#connection-configuration22:30
corvusthat should have an entry for 'password'22:30
fungii used the gerrit ssh api just this morning to add and then remove myself from project bootstrappers so i could push content into the resurrected openstack-chef repo22:31
corvusfor some reason, the failure is only visible in ara: http://logs.openstack.org/91/91c775a1aabf53116da534c27058ea6c62ddeb86/post/publish-zuul-docs/1230b25/ara-report/result/fc2abd92-01f2-4675-b534-3c938a37c0bb/22:32
fungiyeah, the builds status suggests publish-zuul-docs has been failing at least as far back as 2018-07-2622:33
corvusthat does not happen in the check/gate docs builds22:33
corvusnor does it happen locally22:34
clarkbare check and gate using a non warning as error invocation?22:34
fungiWARNING: Unexpected return code 1 from command 'zuul --help'22:34
fungineat22:35
corvusyeah, same command in build-sphinx-docs22:35
corvushttp://logs.openstack.org/27/577027/10/gate/build-sphinx-docs/1848b8f/ara-report/result/86e74885-a5bd-42e1-ba13-76ec560f3677/22:36
clarkbhrm what python version is it using I wonder22:36
fungiModified: 15 July 2018, 18.12.41 UTC22:36
fungiso it changed at some point after that i guess22:37
corvus7-15 is last success; 7-17 appears to be the first failure22:37
clarkbcorvus: http://logs.openstack.org/91/91c775a1aabf53116da534c27058ea6c62ddeb86/post/publish-zuul-docs/1230b25/job-output.txt.gz#_2018-08-06_19_31_01_76471322:38
clarkbits failing to build libre2 before running sphinx22:38
fungisphinx 1.7.6 was uploaded july 16, between the last success and the first failure22:39
corvusbroken:22:40
corvus2018-07-17 01:12:17.176601 | ubuntu-xenial | libre2-1v522:40
corvusworking:22:40
corvus2018-07-15 18:10:05.119819 | ubuntu-xenial | libre2-dev22:40
corvusin the bindep install task22:40
corvuschanged in this commit: https://review.openstack.org/58015922:41
clarkblibre2-dev is installed for profiles compile and test22:42
clarkbwe must not apply those profiles to the publish job bindep run?22:42
*** yamamoto has joined #openstack-infra22:43
fungiaha, yep that change added those profiles to it, which becomes a sieve preventing it from running when neither is requested22:43
corvus      bindep_profile: doc22:43
corvusthat's in the publish job22:43
*** agopi has quit IRC22:43
fungilooks like we need to add doc to that line22:43
corvus      bindep_profile: compile doc22:44
corvusthat's in the build-sphinx-docs job22:44
fungior include compile in the publish job, that would work22:44
corvushopefully after the artifact fetching rework is done, we can refactor that job to inherit from build-sphinx-docs22:47
corvusatm the post-playbook isn't really compatible22:47
*** yamamoto has quit IRC22:47
openstackgerritJames E. Blair proposed openstack-infra/project-config master: Add compile bindep profile to publish-zuul-docs  https://review.openstack.org/58930122:48
corvusfungi, clarkb: ^22:48
corvusfungi, clarkb: this is zuul's connection information: http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/group/zuul-scheduler.yaml22:51
corvusis there a way to add in a secret value in there?22:51
fungiit gets merged with whatever you put in the groups/zuul-scheduler.yaml file in private hiera22:52
corvusi'm going to need to pull that out of hiera and put it in a puppet-language-datastructure22:52
clarkbcorvus: ya I think you read in the value to a variable, then update a subkey of the dict then use that22:52
fungior at least it's supposed to be22:52
clarkbcurrently we just pass it striaght through but you can modify subkeys iirc22:52
corvusoh?  a subkey is what i need22:52
corvushow does that work?22:52
corvus(i need to set zuul_connections[1]['password'])22:53
clarkbI think we need to use puppet stdlib, I'm looking22:53
fungiaha, you need to merge parts of a key... okay that one's definitely beyond me22:53
fungiaside from tracking them in separate keys and then doing some smashing together via puppet or an exec22:54
clarkbcorvus: https://github.com/puppetlabs/puppetlabs-stdlib#deep_merge should work, I'm reading more to see if there is a better options22:55
corvusclarkb: since there's a list in there, that's probably not going to work as-is (i would overwrite the entire gerrit connection)22:56
corvusoh i guess i could just merge that hash22:56
fungisort of defeatist but we could simply stick zuul_connections in private hiera22:57
corvusstill, i'd have to manipulate the list too22:57
corvusfungi: yeah, i think the option to beat is "put it in puppet dsl"22:57
clarkbthere is also a merge22:57
clarkbI think either merge or deep_merge are our best bets22:58
openstackgerritMerged openstack-infra/project-config master: Add compile bindep profile to publish-zuul-docs  https://review.openstack.org/58930122:59
corvusnone of these methods is compatible with the reason this stuff is laid out this way -- to allow direct reuse of our site manifest by others23:00
corvushuh, there's a connection_secrets value that puppet-zuul takes23:02
*** tosky has quit IRC23:02
corvusoh that's clever23:02
corvushttps://review.openstack.org/48294123:03
corvusthat's apparently how we do this :)23:03
fungihah23:05
fungithose crafty puppet-zuul authors23:06
corvusokay, private hiera updated on puppetmaster and bridge to move the gerrit http password value to zuul_connection_secrets23:07
corvuskicking the scheduler23:11
corvusokay, password is in place23:11
corvusi'd like to restart zuul now to pick up the debug info i added and also test out the http reporting23:12
*** bobh has joined #openstack-infra23:12
clarkbzuul seems quiet23:13
clarkbI'm around for about another 45 minutes to help23:13
clarkbthen I am making burger23:13
clarkb*burgers23:13
* mordred sticks his head up from the longest day of debugging in the history of mankind23:16
mordredthe end result of which was the addition of the line "setattr(instance, self.name, None)"23:17
mordredthe whole day23:17
corvusmordred: you've been helpful here recently -- a patch you wrote a year ago is just what i needed 10 minutes ago :)23:17
mordredcorvus: yay!23:17
corvusor some combination of you and pabelanger23:17
corvusdoes anyone feel like http://zuul.openstack.org/ should have more stuff?23:18
clarkbcorvus: yes23:18
corvushttp://zuul.openstack.org/api/info23:18
ianwhttp://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 seems to confirm 0 node requests?23:19
*** hongbin has quit IRC23:19
clarkbit would've just happened bceuase I checked the status page a few minutes ago to see things were relatively idle23:19
mordredcorvus: that doesn't look great23:19
corvuszuul-web is not running23:19
clarkbbut not that idle23:19
corvusi attempted to restart it, maybe it failed23:19
corvusthe pidfile was left behind23:21
openstackgerritJames E. Blair proposed openstack-infra/zuul master: DNM: test gerrit http reporting  https://review.openstack.org/58930523:22
corvusoh hrm, where are my re-enqueue commands?23:23
corvusthere they are :)23:23
clarkbjust the pidfile then? did we merge the change that stopped removing the pidfiles in the init scripts?23:24
corvusclarkb: i think we did.  should be easy enough to start and stop zuul-web a few times and see what happens23:24
corvusmuch easier to fix than zuul-scheduler :)23:24
clarkbindeed23:24
corvushuzzah!  zuul's first gerrit http report -- with inline comments!  https://review.openstack.org/58930523:24
clarkbnice23:25
clarkball reporting is over https now?23:25
corvusyes, so we should keep an eye out for how that behaves23:25
corvusthe ssh reporting has been *very* reliable (i believe it uses a semi-persistent connection).  i worry about https.  but it should attempt 3 times before giving up.23:26
corvus#status log zuul now reports to gerrit over HTTPS rather than ssh; please keep an eye out for any issues23:27
openstackstatuscorvus: finished logging23:27
corvusinfra-root: ^23:27
mordredcorvus: that is the coolest thing I've seen23:27
clarkbya for all the concerns people have had about gerrit's ssh api not being REST it worked really well23:27
corvusclarkb: the range highlight seems to have worked23:29
mordredcorvus: it did!23:29
clarkbcorvus: yup23:29
clarkbit renders a little weird with the diff coloring23:29
clarkbthere is the random line that ends in green23:29
clarkbbut that is gerrit's fault not zuul's23:29
corvusyeah, with a slight green underline23:29
mordredthe second comment also renders slightly weird, also gerrit's fault not zuul's23:30
*** tpsilva has quit IRC23:30
mordredbecause it chooses to elide some of the lines that are marked with a comment23:30
corvusyeah, you have to expand up a lot to actually find zuul-stream-functional23:30
mordredyah23:30
fungiwowie zowie!23:30
mordredbut I think that's fine23:30
corvuswould be nice if it included the whole range23:30
mordredbecause it's just super cool23:30
mordredmaybe we could submit that as a bug upstream - it might be that having a comment on a line-range that spans outside of the diff section is just a case that hasn't come up23:31
clarkbmordred: I'm sure some of it is the size of the highlighted range23:32
clarkbI'm guessing most human reviewers pick one a single word or line23:32
corvusw00t: a successful merge: https://review.openstack.org/#/c/586888/23:32
openstackgerritFatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI  https://review.openstack.org/57787023:32
corvusverified and submit are two separate POST calls23:33
corvusso it's good to see that works :)23:33
mordred\o/23:33
corvusthat's what that looks like: http://paste.openstack.org/show/727449/23:34
ianwcorvus / mordred : thoughts before i send out reviews -- would you rather see a "kerberos-client" role and a "openafs-client" role, with the later depending on the prior; or keep it more contained in a openafs-client role?23:36
*** hemna_ has quit IRC23:36
*** rpioso is now known as rpioso|afk23:37
corvusianw: my instinct is 2 roles, mostly because of how well segregated they are elsewhere23:37
*** agopi has joined #openstack-infra23:38
ianwcorvus: ok, so in terms of laying that out -- does the openafs-client role take kerberos arguments, or we just include the kerberos-client role, and document "you need to setup these parameters"?23:38
mordredI think 2 roles is fine as well - although I'm not sure I'd have them depend on each other to start- just having both in the playbook seems easy enough for today23:38
ianwor more document "you need to setup extra parameters, see the kerberos-client role"23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!