Friday, 2018-04-13

openstackgerritMerged openstack-infra/zuul-jobs master: document that the build-reno-releasenotes job accepts sphinx_python  https://review.openstack.org/56104100:39
openstackgerritMerged openstack-infra/zuul-jobs master: Fix typo in build-sphinx-docs docstring  https://review.openstack.org/56104000:39
mnaserso either we did something terribly wrong in puppet-openstack00:42
mnaseror for some reason, a certain job is running as non voting when its not00:42
mnaserhttps://github.com/openstack/puppet-openstack-integration/blob/master/.zuul.yaml .. yet notice .. https://review.openstack.org/#/c/560628/ puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial is non-voting00:43
pabelangermnaser: did you back port change to stable branches?00:45
mnasermaybe not, im thinking that might be it00:45
pabelangerhttp://logs.openstack.org/28/560628/1/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ade36fd/zuul-info/inventory.yaml is the branch matchers that were applied00:45
EmilienMyeah but zuul should check master no?00:45
mnaserwe did not for the set to voting00:45
pabelangerit will check all branches00:46
EmilienMhttps://github.com/openstack/puppet-openstack-integration/blob/stable/queens/.zuul.yaml#L13900:46
EmilienMthis is in queens, non voting00:46
mnaserits interesting that only scenario4 is the one going non voting00:46
EmilienMbut puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial as well00:46
EmilienMyeah00:46
mnaserhttps://github.com/openstack/puppet-openstack-integration/blob/stable/queens/.zuul.yaml#L8800:46
mnaseryeah except the rest are okay and not this one00:46
EmilienMyeah00:46
EmilienMthere is a nice debug option00:46
EmilienMwe can use it00:46
pabelangerthe inventory file I linked shows how the branch matcher was applied00:47
EmilienMmnaser: do that: https://review.openstack.org/#/c/529846/1/.zuul.yaml00:47
EmilienMso we can debug00:47
*** harlowja has quit IRC00:52
*** JasonCL has joined #zuul00:56
*** JasonCL has quit IRC00:57
*** gouthamr has quit IRC00:59
*** gouthamr has joined #zuul01:05
*** gouthamr has quit IRC01:37
*** gouthamr_ is now known as gouthamr01:37
*** rlandy|afk has quit IRC01:53
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter  https://review.openstack.org/53554301:54
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter  https://review.openstack.org/53554301:56
*** JasonCL has joined #zuul01:57
*** JasonCL has quit IRC01:58
*** JasonCL has joined #zuul02:01
*** JasonCL has quit IRC02:03
*** Wei_Liu has joined #zuul02:08
*** JasonCL has joined #zuul02:13
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter  https://review.openstack.org/53554302:30
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object  https://review.openstack.org/56070002:31
*** xinliang has quit IRC02:47
*** xinliang has joined #zuul02:58
*** xinliang has quit IRC02:58
*** xinliang has joined #zuul02:58
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object  https://review.openstack.org/56070003:40
*** riex7HX4KJ has joined #zuul03:45
*** riex7HX4KJ has quit IRC03:47
*** Dagger44EKP1 has joined #zuul04:02
*** Dagger44EKP1 has quit IRC04:04
*** gouthamr_ has joined #zuul04:09
*** gouthamr has quit IRC04:11
*** gouthamr_ is now known as gouthamr04:12
*** gouthamr_ has joined #zuul04:12
*** gouthamr has quit IRC04:20
*** gouthamr_ has quit IRC04:20
*** gouthamr has joined #zuul04:21
*** harlowja has joined #zuul04:45
*** mordred has quit IRC05:29
*** harlowja has quit IRC05:39
*** mordred has joined #zuul05:41
*** hashar has joined #zuul06:53
*** jpena|off is now known as jpena07:49
*** electrofelix has joined #zuul07:58
*** sshnaidm|afk is now known as sshnaidm|off09:10
*** sshnaidm|off has quit IRC09:14
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Align github status tests with real behavior  https://review.openstack.org/56117609:33
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support regex matching of github status  https://review.openstack.org/56117709:33
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Don't create ci test admin login unencrypted in postgres  https://review.openstack.org/56117909:45
*** xinliang has quit IRC09:45
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support regex matching of github status  https://review.openstack.org/56117709:55
*** xinliang has joined #zuul09:57
*** xinliang has quit IRC09:57
*** xinliang has joined #zuul09:57
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support regex matching of github status  https://review.openstack.org/56117710:08
*** JasonCL has quit IRC10:19
*** JasonCL has joined #zuul10:31
*** tflink has quit IRC10:34
*** tflink has joined #zuul10:35
*** snapiri has quit IRC10:43
*** xinliang has quit IRC10:51
*** xinliang has joined #zuul11:05
*** xinliang has quit IRC11:05
*** xinliang has joined #zuul11:05
*** jpena is now known as jpena|lunch11:41
*** tflink has quit IRC11:56
*** tflink has joined #zuul11:58
*** tflink has quit IRC12:05
*** tflink has joined #zuul12:19
mrhillsmanare there any known reasons for an executor to stop polling12:26
*** jpena|lunch is now known as jpena12:29
*** rlandy has joined #zuul12:31
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support regex matching of github status  https://review.openstack.org/56117712:33
*** xinliang has quit IRC12:49
*** xinliang has joined #zuul12:50
*** xinliang has quit IRC12:50
*** xinliang has joined #zuul12:50
*** tflink has quit IRC12:54
*** pwhalen has quit IRC13:03
*** pwhalen has joined #zuul13:08
*** pwhalen has joined #zuul13:08
*** sshnaidm|off has joined #zuul13:32
*** dkranz has joined #zuul13:38
corvusmrhillsman: stop polling for what?  (i'm not sure the executor polls anything)13:45
mrhillsmanjobs from gearman13:46
mrhillsman thought i was seeing it 'Polling 1 connection' or 'Polling for 1 connection' in debug before but no longer13:46
mrhillsmani pushed a change and see it just sit for +10 minutes and not pick up the job13:47
mrhillsmanthen i restart it and it does its thing13:48
mrhillsmanafter some time it does it, just sits and i don't see it checking for jobs when iirc it was before13:48
corvusmrhillsman: ah yep, that's a gear log message.  it's slightly misleading in that it's right before a call to the "poll" syscall, which is why it says "polling", but it's not a traditional "poll after interval" sort of operation.  it actually just sits there indefinitely waiting for a packet over the gearman connection.  once it gets a packet, it exits the loop, does work, then enters again and emits the13:52
corvus"polling" message.13:52
pabelangermrhillsman: zuul-executor has a governor for load and memory, when hit the limits of one, it will stop launching jobs13:52
corvusmrhillsman: so, all that said, that means that: it's normal for it to not emit that message if nothing's happening13:53
corvusmrhillsman: but it's not normal for it to not do any work if it's otherwise idle and there's a job waiting.13:53
mrhillsmanyep13:53
corvusmrhillsman: the most likely cause is a firewall between the executor and the scheduler which has decided to drop the connection between the two13:53
mrhillsmanforgive me if i get the seq wrong13:54
corvusmrhillsman: not at all -- i'm probably overexplaining as is my habit :)13:54
mrhillsmanbut i recall seeing some gearman variables or whatever like NOOP, PRE_SLEEP, etc13:54
corvusmrhillsman: we had some ideas about tcp keepalives... let me see if they made it into the code13:54
fbo_hi, is there any plan to externalize tools/encrypt_secret.py in its own repo ? It could be convenient for a zuul user to just pip install zuul-secret (whatever the name)13:55
mrhillsmanthat firewall could be the problem13:55
corvusmrhillsman: yep, executor says "PRE_SLEEP", then goes to idle.  scheduler says "NOOP" when there's a job to wake the executor up.  executor then says "GRAB_JOB_UNIQ"13:55
mrhillsmanyeah13:55
corvusfbo_: not at the moment, but it's intended to work with standard python lib so you can just "curl" it out of git.zuul-ci.org13:56
mrhillsmani think it could be firewall -_-13:56
mrhillsmani increased the traffic quite a bit over the last week and maybe it triggered something or someone13:56
fbo_corvus: ok13:56
mrhillsmanto start killing the connection after some time13:56
corvusmrhillsman: oh, does this happen when things are busy, or when it's idle?13:57
corvusmrhillsman: if it is the firewall/network, and it happens when it's idle, then adding keepalive support may prevent it from happening.   if it happens when it's busy, then it may not prevent it, but it could allow zuul to recover automatically.13:58
corvusmrhillsman: this is the change we need: https://review.openstack.org/42524813:58
mrhillsmani have seen it happen when i am starting and stopping jobs trying to troubleshoot something via pr updates13:59
mrhillsmanwhich i thought may be just me being the problem13:59
mrhillsmanbut when i see it sit for quite a while after i know it ran a job before and it does not is what triggered my question14:00
mrhillsmanthe former did not really concern me as i thought i may have hit some github ratelimit or something else14:01
mrhillsmanand just restarted the executor14:01
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Tell geard to use keepalives  https://review.openstack.org/42524814:02
mrhillsmani'll pull that in and see if it helps, thx corvus14:02
corvusmrhillsman: ^ you'll need that updated version14:02
*** elyezer has quit IRC14:05
*** elyezer has joined #zuul14:08
corvusjlk: i +2d but did not approve https://review.openstack.org/561177 in case you want to take a look14:11
corvusSpamapS: ^14:11
*** tflink has joined #zuul14:12
openstackgerritMerged openstack-infra/zuul master: Don't create ci test admin login unencrypted in postgres  https://review.openstack.org/56117914:30
corvustobiash, tristanC, mordred: if you have time today to update https://etherpad.openstack.org/p/zuul-update-email with what you did this week and/or plan to next week that would be great.14:36
tobiashcorvus: on my way home, will do later this evening14:37
corvusanyone else too -- i just distinctly remember reviewing changes from those folks :)14:38
*** acozine1 has joined #zuul14:41
*** openstackgerrit has quit IRC14:50
mordredcorvus, Shrews, tobiash: in reviewing Shrews docs patches, I realized that the static driver is written to be ssh-specific14:57
mordredfeels like we need a connection-type config option (defaulting to ssh) and to rename 'ssh-port' to 'port', right?14:58
tobiashmordred: there is a change in review about connection type15:00
mordredawesome15:00
tobiashmordred: ah, that's probably openstack specific15:01
tobiashOr was that merged already? Don't remember15:02
mordredtobiash: there's one for diskimages15:03
mordredtobiash: I'll put up one for static15:03
dmsimardThe superuser post about Zuul, OpenStack and ARA is live on superuser: http://superuser.openstack.org/articles/scaling-ara-ansible/15:14
dmsimards/superuser//15:15
Shrewsthis seemingly random zuul test failure might be concerning, yes? http://logs.openstack.org/00/558600/4/check/zuul-tox-py35/5307042/job-output.txt.gz#_2018-04-12_19_11_20_80080315:20
*** openstackgerrit has joined #zuul15:20
openstackgerritMerged openstack-infra/nodepool master: Don't gather host keys for non ssh connections  https://review.openstack.org/53568515:20
openstackgerritMerged openstack-infra/nodepool master: Add connection-port to provider diskimage  https://review.openstack.org/53568615:21
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Support winrm hosts in static driver  https://review.openstack.org/56125615:21
mordredtobiash: ^^15:21
rcarrillocruzmordred: would it make sense to do 'if node['connetion-type'] != ssh': return15:26
rcarrillocruzlike i'm thinking15:26
rcarrillocruzif we had nodes with exotic transports that are non ssh15:26
rcarrillocruzalong with winrm15:26
mordredrcarrillocruz: yah- tat's probably better15:27
rcarrillocruzwe have in-house eapi/nxapi (http based connections), and i think netconf can be run on non-ssh15:28
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Support winrm hosts in static driver  https://review.openstack.org/56125615:28
mordredrcarrillocruz: done!15:28
rcarrillocruzsweet, thx15:28
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: Add static driver doc to Zuul From Scratch  https://review.openstack.org/55880215:30
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files.  https://review.openstack.org/55883015:30
tobiashmordred: just found a small issue15:34
tobiashmordred: re https://review.openstack.org/56125615:34
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Support winrm hosts in static driver  https://review.openstack.org/56125615:45
mordredtobiash: good catch15:45
tobiashmordred: actually just a nit but could lead to confusion if someone looks at the tests as a reference15:46
mordredyah - and there's no need to have that possible confusion there15:49
mordred:)15:49
mordredrcarrillocruz: ^^ patch updated15:49
tobiashcorvus: is a zuul-sphinx release needed to pick up 559963?15:50
corvustobiash: yes!15:50
corvusoh, that reminds me, we should send out the release announcement for 30115:50
corvusyay!  https://zuul-ci.org/docs/zuul/releasenotes.html#release-notes15:51
corvusoh, hrm15:51
corvusthe anchor is "id1"15:51
corvusi kind of worry that when we release 3.0.2, *that* anchor will be id1, and 3.0.1 will be id2.15:52
corvusi guess we'll see, and if so, maybe we can update reno to produce stable anchors15:52
*** sshnaidm|off has quit IRC15:52
corvusanyway, that's what we've got for now, so i'll write up an announcement15:52
corvusdo we want just a short announcement with download/release note links, or do we want to copy the relnotes into the email?15:53
corvushttps://etherpad.openstack.org/p/51gI38SIyu15:54
mordredcorvus: yah - I tink we should update reno to produce stable anchors - id_3_0_1 would be a nicer anchor - and id_unreleased for the unreleased bits15:54
corvuszuul-sphinx 0.2.2 pushed15:55
corvustobiash: ^15:55
mordredcorvus: I think copying the relnotes into the email would be nice ... it tends to irk me when I get an email and the only thing in it is a link15:56
tobiashcorvus: thanks :)15:56
mordredcorvus: but - if we're going to do that, we might want to automate (or partially automated) it in some way - like, it would be neat if we could generate a plain-text form of those release notes with the links in text-email footnote form15:57
corvusmordred: yeah, we may want to look into the release-announce emails15:58
corvusmordred: in the interim, i updated the etherpad15:58
mordredcorvus: ++16:00
corvusokay, i'll send that now16:01
SpamapScorvus: tobiash I really love the regex status matching. I wonder if we could get it in behind the re2 switch, and never support backrefs in those regexes?16:02
SpamapSLike you could even modify the patch to pull in re2 and just use it for this.16:03
tobiashSpamapS: ok, sounds good16:03
SpamapSI will say this is less dangerous than the file matcher.16:04
SpamapSSince only write users can make statuses.16:04
tobiashSpamapS, corvus: re2 is decided for the future?16:04
corvustobiash: i think so -- i haven't heard anything against it -- we're just going to need a transition period once the branch matcher patches are fixed up and ready to go16:06
tobiashcorvus: so it's logical to start right with re2 for any new regex feature16:06
corvustobiash: ++  but it's worth thinking about whether lack of negative matchers will be a problem here, and if we need to solve it by adding more options (like "not-status:")16:09
corvusalso, "not-branches" and "not-files" is really growing on me, i have to say.16:09
tobiashcorvus: at least for the status we have already reject which would enable probably most of these not-usecases for status16:10
corvusmakes sense16:10
tobiashcorvus: so should I change it to re2?16:11
SpamapStobiash: https://review.openstack.org/#/c/536389/ has the details16:11
SpamapSbut yeah, the interface for matchers is also not quite right yet16:11
corvustobiash: that sounds like a good idea to me16:11
SpamapScorvus: I like not-* too.16:11
tobiashSpamapS: thanks, have it open already16:11
SpamapSmakes it read like natural language.16:12
corvusSpamapS: and still doesn't violate the descriptive property we like.16:13
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Support regex matching of github status  https://review.openstack.org/56117716:16
tobiashSpamapS, corvus: ^16:16
tobiashmordred: do you have time for an easy review? https://review.openstack.org/55370216:23
*** hashar is now known as hasharAway16:24
mordredtobiash: done16:25
tobiash:)16:25
*** JasonCL has quit IRC16:26
tobiashcorvus: shall I also change 535713 to re2?16:27
*** JasonCL has joined #zuul16:28
*** sshnaidm|off has joined #zuul16:33
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Support winrm hosts in static driver  https://review.openstack.org/56125616:37
openstackgerritMerged openstack-infra/zuul master: Align github status tests with real behavior  https://review.openstack.org/56117617:00
*** jpena is now known as jpena|off17:07
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list  https://review.openstack.org/55370217:26
tobiashcorvus, mordred: this re-adds the json import in test_web.py which was missing now on master and broke gate ^17:27
*** electrofelix has quit IRC17:38
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list  https://review.openstack.org/55370217:48
openstackgerritMerged openstack-infra/zuul master: Remove zuul-tox-py35  https://review.openstack.org/56098318:21
*** smyers has quit IRC18:30
*** smyers has joined #zuul18:33
openstackgerritMerged openstack-infra/nodepool master: Use AFS mirrors for debian-stretch testing  https://review.openstack.org/55863318:36
pabelangerShrews: left review on 56099318:46
mordredtobiash: +218:50
*** harlowja has joined #zuul19:10
mrhillsmandid something change re accessing project public key for encryption?19:19
mrhillsmanwith the changes to zuul-web?19:19
clarkbmrhillsman: I think the api paths may have changed but other than that I don't know of any changes19:20
*** acozine1 has quit IRC19:20
clarkbmrhillsman: its now /api/tenent/stuff iirc. Unsure if the encryption stuff goes through that api19:20
mrhillsmannah, the tool is not using /api19:21
mrhillsmanreq = Request("%s/key/%s.pub" % (args.url.rstrip('/'), args.project))19:21
mrhillsmanyou provide url and project19:22
mrhillsmanbut manually it is no bueno19:22
mrhillsmani am probably typing it wrong19:22
clarkbright I'm not sure if that moved under /api19:22
Shrewspabelanger: the default IS actually nodepool.pid19:24
Shrewspabelanger: that threw me too at first19:24
Shrewspabelanger: we can change the default, but that should probably be a separate change19:25
pabelangerShrews: Ah, I see now. nodepool.pid and nodepool-builder.pid19:27
Shrewspabelanger: yup19:28
pabelangeryah, maybe a follow change it to nodepool-launcher makes sense19:28
pabelangerI'll update my vote19:28
Shrewstobiash: ^^^19:28
clarkbmrhillsman: ('GET', '/api/tenant/{tenant}/key/{project:.*}.pub' is the route in zuul web19:29
mrhillsmanugh19:29
mrhillsmanthx19:29
clarkbmrhillsman: so ya should be /api/tenant/yourtenanthere/key/projectname.pub19:29
mrhillsmandocumentation needs some updating ;)19:29
clarkbmrhillsman: and the script too19:29
mrhillsman++19:29
clarkbmrhillsman: areyou going to test that and push up an update?19:29
mrhillsmansure can ;)19:29
Shrewspabelanger: tobiash: one of you care to add the +A to that?19:31
tobiashmrhillsman, clarkb: ya one of my customers also had problems with that19:31
pabelangerShrews: no issue with nodepool starting using it?  I haven't tested myself19:32
tobiashEncryptsecret works if you put api/tenant into the url before your tenant name19:33
Shrewspabelanger: i didn't test these, just pulled them from what was already merged19:33
tobiashBut that's awkward19:33
pabelangerShrews: yah, i think we might want to consider a test before +A. I can give a try here in a bit and update to +A if working, unless somebody else beats me19:34
clarkbmrhillsman: tobiash cool happy to review it since I've already done the investigating of how it should work now19:35
clarkb*review the fixes19:35
mrhillsman++19:35
mrhillsmanfor some reason mine is taking forever to respond19:35
mrhillsmanonce i confirm it is working will push up docs and tool patch unless someone beats me to it19:35
tobiashmrhillsman: zuul-web might be hanging19:36
mrhillsmani'm thinking that may be it with the changes19:36
mrhillsmanwhich brought me to ask initially with the tool not working out the box19:36
tobiashNow that the url is more complex we might want to set the tenant via an argument instead of encoding it into the url on the commandline19:37
mrhillsmanAttributeError: 'NoneType' object has no attribute 'public_key'19:38
tobiashclarkb, mordred what do you think about a javascript based variant of the encrypt script?19:38
openstackgerritMerged openstack-infra/nodepool master: Add example nodepool-launcher systemd service file  https://review.openstack.org/56099319:39
mrhillsmanscheduler crying19:39
mrhillsmanhttps://www.irccloud.com/pastebin/YDgHVccF/19:39
tobiashI think that would greatly increase usability for non zuul devs19:39
clarkbtobiash: with a text box that you can drop the secret into and have it spit out yaml to copy and paste?19:39
clarkbtobiash: probably not a bad idea especially if we can keep all of that in the browser19:40
clarkb(so we never see the secret unencrypted)19:40
tobiashYes, like zerobin19:40
tobiashmrhillsman: in that case the project is not found19:41
tobiashMaybe missing tenant information19:41
mrhillsmanhrm, let me try again19:41
mrhillsmanzuul-web went away as well19:42
mrhillsmanlet me just restart all the things real quick19:42
mrhillsmani was working on some other stuff19:42
mrhillsmanstupid internet went down20:27
mrhillsmanor rather my shit isp did20:28
openstackgerritMerged openstack-infra/zuul master: Don't count non-live queue items in tenant list  https://review.openstack.org/55370220:37
*** JasonCL has quit IRC20:38
*** JasonCL has joined #zuul20:57
*** evrardjp has quit IRC21:40
*** rlandy has quit IRC22:09
*** hasharAway has quit IRC22:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!