Wednesday, 2017-03-01

jeblairmordred, pabelanger: careful -- our current subprocess invocation is the result of much work.00:00
jeblairespecially bits around ttys00:01
mordred++00:01
clarkbpabelanger: re running python straight out of venv and ansible-playbook the issue is likely that when you don't source the venv the only things that are modified are your python paths, but ansible is subprocessed so its looked up in your shell path00:02
clarkbpabelanger: if yo usource the venv then shell path is also updated to include eg venv/bin00:02
jeblairpabelanger, clarkb: mordred is making a patch to ttrun to help with that00:02
pabelangermordred: jeblair: ya, it will have to wait until tomorrow, atleast for pause task. But I'm sure if I do use that, I'll need some help00:03
jeblairmordred, pabelanger: thinking and looking further, most of our tty issues for 2.5 are on the remote host00:03
pabelangerotherwise, I have no other thoughts how to block a job to get aborted00:03
jeblairmordred, pabelanger: so we have ansible pipelining set to true to aid in that00:03
clarkbre pause why not just use command: sleep 30 ?00:04
jeblairmordred, pabelanger: but we never dealt with ansible local stuff00:04
clarkbis expectation that remote end isn't reachable during that period? for something like a reboot?00:04
pabelangerclarkb: our zuul action plugins block it00:04
jeblairmordred, pabelanger: so we may want to do something like what mordred suggests to make that more robust.00:05
clarkbpabelanger: they don't block it on the remote and that has mostly the same net effect of waiting for 30 seconds00:05
pabelangerclarkb: all our testing right now is localhost connection, so we'd have to build that out for remote things00:05
clarkboh this is test suite specific?00:06
pabelangerYa, sorry, I might have left that piece out00:06
mordredpabelanger, clarkb, jeblair: ttrun updated - I _think_ the path thing should work now00:06
mordredif you upgrade to 1.0.500:06
mordredk. I've been running solid for 12 hours - I will now unplug from the laptop brain for a bit00:07
clarkbjeblair: perhaps use local ssh connection rather than fork?00:08
pabelangerYa, going to do the same00:08
jeblairclarkb: hopefully closing stdin will be enough for us; if we need a pty, that's still probably easier to work with than ssh.00:10
clarkbspeaking of I wonder if ansible with paramiko (which is not default) will clena up these threads or will it leak like nodepool00:10
*** jamielennox is now known as jamielennox|away00:36
*** jamielennox|away is now known as jamielennox00:44
openstackgerritMerged openstack-infra/nodepool master: Add destructor to SSHClient  https://review.openstack.org/43924301:04
*** saneax is now known as saneax-_-|AFK01:14
*** saneax-_-|AFK is now known as saneax04:55
*** saneax is now known as saneax-_-|AFK05:16
*** saneax-_-|AFK is now known as saneax05:21
*** abregman has joined #zuul06:07
*** adamw has joined #zuul07:30
*** abregman has quit IRC08:26
*** abregman has joined #zuul08:28
*** isaacb has joined #zuul08:40
*** abregman is now known as abregman|mtg09:01
*** abregman|mtg has quit IRC09:22
*** saneax is now known as saneax-_-|AFK09:22
*** abregman has joined #zuul09:23
*** saneax-_-|AFK is now known as saneax09:53
*** isaacb has quit IRC10:13
*** bhavik1 has joined #zuul10:24
*** bhavik1 has quit IRC10:45
*** hashar has joined #zuul10:49
*** isaacb has joined #zuul10:59
*** isaacb has quit IRC11:24
openstackgerritJan Kundrát proposed openstack-infra/zuul master: Prepare correct refspec on new Gerrit  https://review.openstack.org/43905711:56
*** saneax is now known as saneax-_-|AFK13:02
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add timeout variable to jobs  https://review.openstack.org/43963714:57
pabelangerjeblair: clarkb: ^So, we might need a way to override the default WATCHDOG_TIMEOUT_GRACE, currently it is 5mins.  Maybe we should expose that some how, otherwise our unit test will be long.  Also left a few comments for feedback15:00
*** isaacb has joined #zuul15:02
*** isaacb_ has joined #zuul15:02
*** isaacb has quit IRC15:03
*** isaacb_ has quit IRC15:03
*** isaacb has joined #zuul15:04
jeblairpabelanger: we may just want to drop the grace period; that was from zuulv2.5 where we expected another timeout to hit first.  with the other timeout gone, this is the primary one.15:08
pabelangerjeblair: sure, we can do that oo15:09
pabelangertoo*15:09
*** isaacb has quit IRC15:22
*** isaacb has joined #zuul15:22
jeblairpabelanger: i responded to your comments/questions on 439637.15:32
jeblairSpamapS: comment on 43056815:35
pabelangerjeblair: thanks15:35
openstackgerritPhilip Marc Schwartz proposed openstack-infra/zuul feature/zuulv3: Add Directed Acylic Graph for usage as a JobGraph  https://review.openstack.org/43966315:42
openstackgerritPhilip Marc Schwartz proposed openstack-infra/zuul feature/zuulv3: WIP Add Directed Acylic Graph for usage as a JobGraph  https://review.openstack.org/43966315:44
phschwartzjeblair: SpamapS: ^^ added you both as reviewers so you can start looking at the first part.15:47
phschwartzhmm, looks like the linters failed but no valid link to logs for it15:48
pabelangerjeblair: do you want the default timeout to be None(run forever) or support the ability to set no timeout15:50
*** abregman has quit IRC15:53
openstackgerritPhilip Marc Schwartz proposed openstack-infra/zuul feature/zuulv3: WIP Add Directed Acylic Graph for usage as a JobGraph  https://review.openstack.org/43966315:53
jeblairpabelanger: i'm inclined to "None" but am willing to be persuaded that we should have a default timeout.  I think in either case, we should support no timeout (even if that means users set an explicit 0)15:53
pabelangerjeblair: okay, for now, I'll default to None. And if we want to update in the future we could15:54
jeblairsounds like a plan15:55
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add timeout variable to jobs  https://review.openstack.org/43963715:55
*** abregman has joined #zuul15:56
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add timeout variable to jobs  https://review.openstack.org/43963716:04
*** abregman is now known as abregman|mtg16:18
*** abregman|mtg has quit IRC16:44
*** isaacb has quit IRC16:47
*** abregman has joined #zuul16:52
*** abregman has quit IRC17:06
*** Gravious has joined #zuul17:10
rbergeronhi peeps: I have a draft of the Very First exciting zuul project update for folks to eyeball. https://etherpad.openstack.org/p/zuul-v3-first-status-update17:23
rbergeron^^ mordred, jeblair, pabelanger, spamaps, anyone else who is around17:23
jeblaircool, i will read s soon as i finish my firefox memory purge :)17:24
rbergeronit is... longish. I think in future editions that we can trim a significant portion of the first part, but i think leaving the stuff at the end (about zuul, community info), at the bottom, could be helpful.17:24
rbergeronoh god17:24
rbergeronsorry17:24
SpamapSdunno what you people do to Firefox to make it so hungry. Mine's only using 1.2G with two slacks and a verse open. :-P17:27
rbergeronfor those reading: i am not opposed to cliipping a bunch of the top and just saying "go read monty's blog" but i (clearly) like to fall on the side of more information, provided the things ppl really need to know are quickly findable / apparent17:30
rbergeronand i think the only technical-ish question i had (not to say that i may or may not have screwed stuff up without even questioning those items) -- the terminology around the 'not openstack clouds' in the future and if that includes "random hardware that isn't even a cloud"17:31
jeblairrbergeron: cool; i added a suggestion near the botom about irc channels.17:31
rbergeronand fwiw: did notice in my "where's a generic description of zuul" that there is the one in docs; monty referring to it as anelastic ci/cd engine in slides; jeblair referring to it as "system to drive project automation" in a blog17:32
jeblairrbergeron: the part about other clouds and not-clouds read okay to me.17:33
rbergeronthat might be a thing to more concretely cement in the future, since we sometimes call it gating, sometimes CI / CD, etc.17:33
rbergeronjeblair: awesome17:33
jeblairrbergeron: it's a rorschach test?17:33
rbergeronjeblair: totally :)17:33
rbergeronjeblair: and yeah, the irc thing. I think i mentally had that in my squirrelbrain and um, yeah, squirrels or something17:34
pabelangerlooks like zuul is complaining about pbr 2.0.0 this morning17:46
pabelangeror a dependency rather17:46
*** hashar has quit IRC17:47
rbergeronpabelanger, jeblair: i think you're the ones who eyeballed / had comments - does it look kosher to send out?17:51
rbergeron(and before i do: do we want to have something like [zuul-status] or etc. in the subject (now or in the future))?17:51
pabelangerrbergeron: I left a URL in one part, but seems straightforward to me17:51
pabelangerI am setup for [infra] personally17:52
rbergeronpabelanger: i caught that and added it17:52
rbergeronwas just thinking of the folks who may explicitly just want zuul updates and filter out hte rest of openstack-dev, but maybe we're not worried about that yet or expect that they're on the infra list anyway17:53
rbergeroni just don't want to be the evil person who doesn't follow rules about [infra] or something like that :)17:53
jeblairrbergeron: lgtm!17:56
rbergeronjeblair, pabelanger: i sent you a, um, test version just to make sure it doesn't come out on the other side with me looking moronic... gmail ate something of mine this week and digested it in awful fashion18:01
rbergeronif you have uno moment to just say "oh yes" or "or god no", no need to reread the whole thing18:02
rbergeron(and do i do to: openstack-dev and openstack-infra in the same mail, or two separate things?18:02
rbergeron)18:02
rbergeron(sorry, questioning robyn)18:02
pabelangerI got a plain text version18:04
jeblairrbergeron: a double to, or a to and cc should be fine.  i think the lists set reply-to.18:04
rbergeronpabelanger: excellent, and ty, jeblair18:06
rbergeronokay, it is sent; i do believe this is my first post to at least -dev, and likely -infra as well, so if there is a "first post needs to be approved by someone" thing going on, then those buttons will need pushin'18:11
jeblairrbergeron: i think you only need to be subscribed to them18:11
rbergeronjeblair: well, then we should see things soonishly if not already18:16
jeblairrbergeron: i have received it!18:18
rbergeronjeblair: excellentay!18:22
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Update hacking dependency  https://review.openstack.org/43975618:23
Gravioushi, zuul code seems to be python3 compatible but the setup.cfg only mentions python2, should i read anything into this?18:30
clarkbGravious: it doesn't currently get python3 testing and I don't know that all the deps like gear are fully compatible. Though it is being worked on18:31
Graviousclarkb, ah ok, cheers :)18:33
jeblairyes, hopefully around the v3 release (maybe a little before or after depending on how things go).18:41
rbergeronansible itself is not yet v3 compatible (workin' on that), but unclear if that's related (I think not, but who knows)18:42
*** harlowja has quit IRC18:43
clarkbrbergeron: in current design it shouldn't matter since ansible is run as a subprocess. It does mean you would have to have both versions of python handy though18:44
pabelangerlast I checked, shade is not 100% python3 too18:44
pabelangerOh18:44
pabelangermaybe that is just for RPM packaging18:44
*** harlowja has joined #zuul18:46
rbergeronclarkb: that was my thought, but :)18:50
pabelangerjeblair: mind a +3 on 43975618:52
pabelangerfixes our pbr-2.0.0 dependency issue18:52
mordredpabelanger: if shade isn't 100% python3 I'm going to be very sad because we do gate on python319:01
pabelangermordred: Ya, this is for RPMs. I've been trying to package it for python3, but some dependencies are missing python3 packaging support19:03
mordredpabelanger: ahhhh. fun19:03
jlkWhat's preferred in a layout file? manager: independent, or manager: IndependentPipelineManager ?19:20
jlklooks like the former started being used around december of '1519:20
jeblairjlk: independent.  it's a zuulv3 change19:26
pabelangerneat, tests.unit.test_v3.TestInRepoConfig.test_in_repo_branch failure in the gate pipeline19:26
pabelangerfirst I have seen that19:26
pabelangerhttp://logs.openstack.org/56/439756/1/gate/gate-zuul-python27-ubuntu-xenial/aec5d24/console.html19:26
jlkAnother question. I'm looking at tests/fixtures/zuul.conf which references a tenant_config of main.yaml, but I can't find that file anywhere. Looks like it was removed in be706e1c6d57d96c85e8992041e15184f250ac63 -- what's the intent here?19:27
jlkare the tests finding that file in-repo somewhere?19:28
jeblairjlk: now each test class specifies its main tenant file.  eg: unit/test_v3.py:    tenant_config_file = 'config/ansible/main.yaml'19:29
jlkoh maybe it's buried in a /tests/fixtures/config/ path19:29
jlkokay19:29
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Update hacking dependency  https://review.openstack.org/43975619:38
pabelangerdoh19:40
pabelangerfatal: write error: No space left on device19:40
pabelangerthat is zuulv3-dev.o.o19:40
pabelangerpurging the /tmp dir19:41
pabelangerwe might want to consider putting jobdir in /opt for now19:42
jesusaurpabelanger: doesn't the jobdir get automatically deleted after the job runs?19:47
pabelangermordred: clarkb: mind +3 on 437938?19:47
pabelangerjesusaur: usually yes, but we are keeping them alive for debugging19:47
jesusaurahh19:47
jeblairpabelanger: maybe we can turn that off now?19:47
pabelangerhowever, maybe it is time to expose keepjob setting, per job.19:47
jeblairpabelanger: no, we should not expose that19:48
pabelangerjeblair: sure19:48
jeblairpabelanger: that's an advanced debugging thing for a sysadmin to enable19:48
pabelangerokay, I thought we had discussed having it setup per job19:48
pabelangerI am fine with that too19:48
pabelangerlauncher restarted19:49
jeblairpabelanger: thanks19:49
pabelangerjeblair: would mind a review on 439637, left a TODOv3 also, if you are okay with that19:52
jeblairpabelanger: cool, one small thing inline.  rest lgtm.20:00
openstackgerritClark Boylan proposed openstack-infra/zuul master: Update trigger script to handle periodic jobs  https://review.openstack.org/43982120:01
*** openstackgerrit has quit IRC20:03
pabelangerjeblair: actually, does it make sense to do the evaluation for your reply?  The more I think about it, that check is only valid for jobs that are 9 seconds long. Other wise, we'll still wait up to 10 secs after the timeout.20:08
*** harlowja has quit IRC20:08
jeblairpabelanger: i'm not sure what you're asking.20:11
pabelangerjeblair: updating time.sleep(), I am not sure it is valid on my patchset 439637. It would only be invoked if a job timeout is less then 10.  Otherwise, job timeout could be timeout + 10s.  I only added it for our unit tests, so we didn't have to wait 10 secs for it to pass. But not sure if there is real world value20:13
*** openstackgerrit has joined #zuul20:15
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add timeout variable to jobs  https://review.openstack.org/43963720:15
*** hashar has joined #zuul20:27
*** jroll has joined #zuul20:27
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow using webapp from connections  https://review.openstack.org/43983120:29
jeblairpabelanger: right, it's only really valuable for the unit tests, so i didn't want to incur the cost of doing that 10 times per second (which is what we would be doing on a launcher with 100 jobs running).  if you want to remove it entirely (so the unit test job takes 10 seconds), that's fine too.20:29
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks  https://review.openstack.org/43983420:30
pabelangerjeblair: ya, let me remove for now.20:30
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add timeout variable to jobs  https://review.openstack.org/43963720:31
jlkfun times, I can't get tox to work locally on v3 branch :(20:37
jlksifting through output20:38
jlkkazoo timeout  error20:39
pabelangerare you running zookeeper?20:40
jlkDoes tox not set that up for me?20:40
pabelangernot any more20:40
pabelangeryou need to manually start it20:40
jlk..... okay.20:40
pabelangerbut, once running. zuul does the rest20:40
tflinkpabelanger: I remember you talking about a sprint for zuulv3 this spring but I don't remember when that was. Can you refresh my memory on that?20:40
pabelangertflink: Yes, likely our PTG, which happened last week20:41
jlkis this documented somewhere, starting ZK?20:41
pabelangertflink: rbergeron just sent out ML post bout our latest updates.20:41
pabelangerhttp://lists.openstack.org/pipermail/openstack-infra/2017-March/005202.html20:42
tflinka zuul mailing list?20:42
* tflink is slow20:42
pabelangerjlk: we likely should update the README.rst20:42
pabelangeror TESTING.rst20:42
jlkor at least TESTING.md20:42
pabelangerya20:42
jlkyeah that word20:42
pabelangertflink: np, would have been posted to openstack-dev and openstack-infra mailing lists20:43
tflinkthanks20:44
tflinkpabelanger: may I PM?20:46
pabelangertflink: sure20:46
tflinkis v3 still in a state that it's likely not worth looking at in terms of how well it might work for other uses in the longer term future?20:50
tflinkIIRC, some of the stuff at the PTG was making some openstack-specific bits more generic20:50
*** tobiash_ has joined #zuul20:50
*** mmedvede_ has joined #zuul20:52
*** mmedvede has quit IRC20:53
*** tobiash has quit IRC20:53
*** mmedvede_ is now known as mmedvede20:53
pabelangerHmm, I think you could start poking around at it. Even start uploading some patches to openstack-infra/zuul to see how it actually works. Things like how jobs are defined are still in flux, but in general I think we are well into the right path20:54
tflinkok, thanks. I'll put that back on my list of stuff to do20:56
pabelangertflink: I do have a fedmsg patch in flight, https://review.openstack.org/#/c/426861/ that I am looking forward to testing20:58
*** harlowja has joined #zuul21:01
*** harlowja has quit IRC21:01
*** harlowja has joined #zuul21:01
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Document needing zookeeper to run tests  https://review.openstack.org/43985721:24
openstackgerritCullen Taylor proposed openstack-infra/zuul feature/zuulv3: Re-enable test_tags  https://review.openstack.org/43985821:25
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Document needing zookeeper to run tests  https://review.openstack.org/43985721:37
openstackgerritCullen Taylor proposed openstack-infra/zuul feature/zuulv3: Re-enable test_tags  https://review.openstack.org/43985821:40
jheskethMorning21:58
jlkDid y'all already fix something related to pbr 2.0.0 today?21:58
pabelangerjlk: ya, hacking should have been updated22:00
jlkgot it, needed to rebase22:00
pabelangerjhesketh: morning22:00
pabelangerjhesketh: mind reviewing 437938 and 43963722:01
openstackgerritCullen Taylor proposed openstack-infra/zuul feature/zuulv3: Re-enable test_tags  https://review.openstack.org/43985822:03
* Shrews returns to Zuultopia and sees many good things happening in his absence22:15
Shrewsjeblair: oopsie on the quota bug. will fix it in the morn22:16
jeblairShrews: cool, np and thanks.  we have something working well enough for the ptg and small scale testing.  :)22:26
Shrewsjeblair: that's good. i imagine i'll spend most of the morning catching up on scrollback then22:29
Shrewsoh so much scrollback22:29
openstackgerritCullen Taylor proposed openstack-infra/zuul feature/zuulv3: Re-enable test_tags  https://review.openstack.org/43985822:36
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Report dynamic layout config errors  https://review.openstack.org/43988722:43
jeblairpabelanger, SpamapS, mordred: ^ i think we will be very pleased to have that as we start throwing actual configs at zuul.22:44
pabelangernice22:45
jeblairi pasted a copy of the error message into a gerrit comment22:45
SpamapSoh yes indeed22:45
SpamapSjeblair: I did see your comments on reviews earlier today. Just been in all the meetings. :-P22:46
SpamapSBTW, finally showed Zuul v3 to CNCF TOC, and received a warm reception.22:46
SpamapSAnd was invited to show it to the Kubernetes testing SIG22:47
pabelangerBTW: we should land https://review.openstack.org/#/c/437938/ if at all possible. It is the last piece needed to review the proper URL on review.o.o. I know a few people today were looking for logs22:47
SpamapSI think they understand the scale problems that it solves.22:47
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Report dynamic layout config errors  https://review.openstack.org/43988722:48
jeblairSpamapS: cool, thanks for being in all the meetings :)  that sounds like great progress22:52
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Report dynamic layout config errors  https://review.openstack.org/43988723:02
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Publish python-linters jobs to apache  https://review.openstack.org/43793823:10
SpamapSjeblair: https://github.com/kubernetes/test-infra/tree/master/prow <-- this is our competition in Kubernetes btw23:14
SpamapShook is ~= to zuul-scheduler23:15
SpamapSand they're wondering if they can put Zuul's algorithm into it for gating.23:15
jeblairSpamapS: it far surpasses zuul in its ability to tell jokes.23:17
jamielennoxpabelanger, jeblair: for zuul 2.5 do we need to maintain the jjb 1.6.1 pin?23:22
jeblairSpamapS: if they do adopt the algorithm, i hope they cite us.  but i'd rather they came and joined us.  :)23:23
jeblairjamielennox: i have no idea; we use non-public jjb apis, so we'd have to experiment.23:23
pabelangerI think above that broke 2.5 IIRC23:23
SpamapSjeblair: Me too. Trying to enlighten them, but I don't think standing in their way and trying to stop them will work. We'll have to catch them when they're tired of pushing things through the mud. ;)23:24
jamielennoxso jjb 1.6.1 is the last of the 1.X series, but it specifies: pbr>=1.0.0,<2.0, which is breaking some things now that 2.0 has been released23:24
jamielennoxand pip version dependency resolution being what it is23:24
jamielennoxis the gerrit set up in such a way we could do a 1.6.2 that just removes that 2.0 cap?23:25
jeblairSpamapS: yes.  also, we are unlikely to convince them until we run it ourselves.  :)23:25
jeblairjamielennox: i don't know; this is probably a better #openstack-infra question23:26
jamielennoxjeblair: makes sense23:26
jlkWith v3, is it assumed that anything hitting the webapp will have a tenant name prefix?23:31
pabelangercool, gerrit URLs for zuul is working again: https://review.openstack.org/#/c/439637/23:34
pabelangersee python-linters in table23:34
*** hashar has quit IRC23:34
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow using webapp from connections  https://review.openstack.org/43983123:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!