Friday, 2017-10-27

*** bhavik1 has joined #zuul03:15
*** bhavik1 has quit IRC03:27
*** bstinson has quit IRC04:10
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Silence ansible-lint  https://review.openstack.org/51552004:11
*** bstinson has joined #zuul04:14
*** pleia2 has quit IRC04:21
*** pleia2 has joined #zuul04:22
*** logan- has quit IRC04:38
*** logan- has joined #zuul04:42
*** maxamillion has quit IRC04:51
*** pleia2 has quit IRC04:51
*** bstinson has quit IRC04:51
*** bhavik1 has joined #zuul05:02
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging  https://review.openstack.org/51547005:04
*** bstinson has joined #zuul05:04
*** maxamillion has joined #zuul05:05
*** pleia2 has joined #zuul05:05
openstackgerritMerged openstack-infra/zuul-jobs master: Silence ansible-lint  https://review.openstack.org/51552005:10
*** bstinson has quit IRC05:19
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Run ansible-lint on repo  https://review.openstack.org/51559305:19
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Run ansible-lint on repo  https://review.openstack.org/51559305:19
*** bstinson has joined #zuul05:22
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Run ansible-lint on repo  https://review.openstack.org/51559305:39
*** yolanda has joined #zuul05:42
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Run ansible-lint on repo  https://review.openstack.org/51559305:44
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Run ansible-lint on repo  https://review.openstack.org/51559305:49
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: test ansible crash logging  https://review.openstack.org/51547005:49
*** yolanda__ has joined #zuul06:02
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Do late decoding of log stream buffer  https://review.openstack.org/51504306:04
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correctly stream the remaining buffer  https://review.openstack.org/51547006:04
*** bhavik1 has quit IRC06:18
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correctly stream the remaining buffer  https://review.openstack.org/51547006:45
*** yolanda__ has quit IRC07:27
*** hashar has joined #zuul08:09
*** electrofelix has joined #zuul08:48
jktSpamapS: that's that part which I like about submodules -- it's one standard implementation of that checking out script, so that I don't have to maintain my owm09:32
jktSpamapS: a colleague just lost two days of work getting mad at his non-working shell scripts for this purpose :)09:32
*** hashar is now known as hasharLunch10:13
*** robcresswell has quit IRC11:03
*** sambetts|afk is now known as sambetts11:41
*** electrofelix has quit IRC11:43
*** electrofelix has joined #zuul11:43
*** hasharLunch is now known as hashar11:48
ShrewsMy new phone dies yesterday after only 24 hours of use. I wake up this morning (coldest morning yet of the winter) to no heat. What a wonderful way to end the week.12:41
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move test_model.test_job_inheritance_configloader  https://review.openstack.org/50949512:51
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove test_job_inheritance  https://review.openstack.org/50950912:51
* dmsimard is sad he can't release a new version of ara until monday12:51
*** fdegir has quit IRC13:19
*** fdegir has joined #zuul13:19
dmsimardShrews: do you have the URL to that etherpad we had for zuul with ansible 2.4 ?13:41
Shrewsdmsimard: there was an etherpad??13:42
Shrewsi guess that's a "no" answer  :)13:42
dmsimardShrews: there was :/13:45
* dmsimard searches logs13:45
dmsimardShrews: oh it was with jlk https://etherpad.openstack.org/p/zuul-ansible-2.413:46
dmsimardmixed you two you13:46
dmsimardmixed you two up*13:46
openstackgerritMerged openstack-infra/zuul-jobs master: Run ansible-lint on repo  https://review.openstack.org/51559313:59
jeblairtobiash: responded on 51553614:58
tobiashjeblair: thanks, now I understand14:59
tobiash:)14:59
tobiashjeblair: I think I'm through your stack15:00
SpamapSjkt: understood, but it's also extremely inflexible, and makes cross-repository testing nearly impossible. I wouldn't say that Zuul will never support them, but I would say you are likely going to need to find submodule fans to design and write the feature, because none of us are submodule fans. ;)15:53
SpamapSjkt: you _might_ be able to get it all done in zuul-jobs actually15:54
*** hashar is now known as hasharAway16:18
*** robcresswell has joined #zuul16:19
mnaserjeblair: i was wondering under what is the issue we discussed (inheritance stuff) is listed under in zuulv3 issues to track it so i dont ping you and i can look at the reviews/updates :)16:23
SpamapSso, today's challenge.. FYI.. is feeding secrets into my zuul job so that it can spin up LBaaS resources that point at the nodes that nodepool spun up. This.. could be tricky. ;)16:39
*** sambetts is now known as sambetts|afk16:41
*** hasharAway has quit IRC16:41
SpamapSalso trying to debug why my zuul is dropping the comment config item from the githubreporter16:48
jktSpamapS: actually, I'm using that (sort of) for "cross-repo testing" -- the submodules are 3rd-party repos that I cannot CI-gate, so I explicitly tie their versions with my code, and I gate that combination16:48
jktSpamapS: so if I want to update a dep, I update a submodule (and possibly also my code) in one commit, and let Zuul test that combination16:49
SpamapSjkt: yeah if they're public repos and your nodes have good access to them, that is a simple solution.16:50
fungiheads up, the zuul v3 migration thread on the openstack-dev ml is getting hijacked with questions about deploying zuul v3, if anyone here has time to answer: http://lists.openstack.org/pipermail/openstack-dev/2017-October/124048.html16:56
*** electrofelix has quit IRC16:56
clarkbSpamapS: you might be able to just run a task on the executor that looped through each of the nodepool nodes adding them to some load balancer pool. I think because ansible has shade things you can do it all natively too17:05
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Ansible linting fixes  https://review.openstack.org/51577817:05
SpamapSclarkb: the problem is I have to add them to a pool in the same tenant as nodepool spun the vm up in17:12
SpamapSbut that's not so bad17:12
SpamapSand I already have that ansible written.17:12
clarkbwait really?17:12
SpamapSI just have to separate it out from other things, because it has to run trusted.17:12
clarkbits not valid to load balance arbitrary IPs?17:13
clarkbactually I seem to recall this is why lbaas testing is so difficult17:13
clarkbthey need actual backends running in a cloud (not great to do with qemu)17:13
SpamapSclarkb: oh actually it is17:13
SpamapSbut17:14
SpamapSI have a funky cloud17:14
SpamapSactually no, this might work17:14
SpamapSclarkb: thanks, I can at least try it with a distinct project just for the load balancers. :)17:15
ShrewsSpamapS: a funky cloud, with a funky funky style?17:16
jeblairmnaser: i was bad and did not add to etherpad.  will do so now.17:17
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Ansible linting fixes  https://review.openstack.org/51577817:18
jeblairmnaser: the stack starts at https://review.openstack.org/51445917:19
SpamapSShrews: this cloud drops beats on the daily17:27
SpamapShrm17:44
SpamapSI think my theory that 3s is enough for github is actually wrong17:45
SpamapSjust hit it with that patch installed17:45
SpamapSlooks like Github is just as broken as Gerrit in that regard..17:45
SpamapSso.. this is weird17:49
SpamapShttp://paste.openstack.org/show/624842/17:49
SpamapSI have a change, which depends on another, which is merged. And the dependent change is failing to go into the gate.17:49
SpamapS2017-10-27 10:47:33,534 DEBUG zuul.Pipeline.GoDaddy.gate:   Change <Change 0x7f64fc0cce80 5,f8a6d40db753e30f0e270ce79dedba17fc906640> in project cloudplatform/godaddy-zuul-jobs does not share a change queue with <Change 0x7f64fc160cf8 144,4c0378812b0d9b2e0b7bbd08e1bbebc1d3d48e93> in project cloudplatform/k8s-ansible17:49
SpamapS2017-10-27 10:47:33,534 DEBUG zuul.Pipeline.GoDaddy.gate: Failed to enqueue changes ahead of <Change 0x7f64fc160cf8 144,4c0378812b0d9b2e0b7bbd08e1bbebc1d3d48e93>17:49
SpamapSNot sure why it think it needs to enqueue it, when 5,f8a6d40db753e30f0e270ce79dedba17fc906640 is already merged.17:50
tobiashSpamapS: was the dependee merged by zuul itself?17:58
tobiashmaybe zuul doesn't know that it's merged17:58
SpamapStobiash: it was, but zuul should still check with github, since it may have been restarted since then.17:59
tobiashhm, maybe a bug in the cache handling18:00
SpamapSno, the logic I'm reading says that if you depend on something you don't have a shared queue with, you can't ever merge18:05
SpamapSjeblair: ^^ is that intentional or am I reading it wrong?18:05
*** harlowja has quit IRC18:05
*** harlowja has joined #zuul18:05
tobiashSpamapS: that logic sounds broken and needs to be fixed ;)18:06
SpamapSagreed18:06
SpamapSbut what I see is that dependent calls enqueueChangesAhead, which checks for needed changes using checkForNeededChanges, which returns false if any needs_change changes don't share a queue with the change being considered.18:08
jeblairSpamapS: a merged change shouldn't show up in the needed changes18:15
jeblair(that's the cutoff so we don't pull in a linked list of every git commit ever :)18:15
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Create fetch-puppet-module-output role  https://review.openstack.org/51579818:16
SpamapSjeblair: hrm, so github driver maybe isn't seeing that it is merged.18:19
SpamapSjeblair: and yeah, in reading the tests, I was thinking the logic that is there is the right logic, and was wondering how this change was still being considered.18:20
mnaserjeblair: thanks for the info18:22
SpamapSso github driver checks the is_merged attribute of the PR18:23
SpamapSI think we may have a cache invalidation issue18:27
SpamapStobiash: good call18:27
tobiash:)18:27
SpamapSoh whoa18:48
SpamapSmaintainConnectionCache is never called?18:48
SpamapSThat means we never actually refresh PRs in github.18:50
tobiashSpamapS: I think maintainConnectionCache was initially to clean up the cache18:59
tobiashNot to refresh it18:59
tobiashSince v3 it isn't called anymore19:00
jeblairyeah, it's only in there as a stub until we figure out something better; my weakref change removed it, but that didn't work out.19:00
jeblair(i'd like to look further into that but haven't been able to)19:01
SpamapSJust trying to figure out how things get refreshed.19:02
jeblairSpamapS: but PRs should refresh when any event arrives for them, like the event we (presumably?) get when they are merged19:02
jeblairSpamapS: (see line 190 in githubconnection)19:02
SpamapSjeblair: so maybe I didn't get a webhook when the parent merged.19:02
* SpamapS looks for it in logs19:03
jeblairSpamapS: yeah, if it missed that, it may not recover :/19:03
SpamapSactually, zuul merged it19:03
jeblairSpamapS: it still wouldn't update the change in memory unless it got an event after merging though19:03
SpamapSkk19:03
jeblair(it doesn't assume it's state change actually took until it can confirm it)19:04
SpamapShm, zuul also left the status as pending:  GoDaddy/gate gate status: pending19:04
SpamapSme thinks there were issues19:04
jeblairSpamapS: posting a comment on the PR might be enough to get it to refresh19:04
* SpamapS digs19:04
jeblair++19:04
openstackgerritMerged openstack-infra/zuul-jobs master: Create fetch-puppet-module-output role  https://review.openstack.org/51579819:05
SpamapSjeblair: I think this is all the "lags behind events" problem19:09
SpamapSjeblair: the PR merges.. webhook fires... API responds is_merged:False for a small window19:09
SpamapS2017-10-26 16:04:38,486 DEBUG zuul.GithubConnection: Set commit status to success for sha f8a6d40db753e30f0e270ce79dedba17fc906640 on cloudplatform/godaddy-zuul-jobs19:10
SpamapSsuper weird because it's showing as pending still19:10
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add multi-branch support for project-templates  https://review.openstack.org/51580519:12
jeblairtobiash: ^ if you're still around19:13
jeblairShrews: ^ you too19:15
Shrewslooking19:16
SpamapSOH!19:19
SpamapSthe gate is left pending in the PR19:19
SpamapSbecause the commit that is marked as 'success' is the child19:19
SpamapShttps://photos.app.goo.gl/4zYNLu72ytQpEnAf219:20
SpamapSjlk: ^19:20
SpamapSI wonder if we should try and set the status on the merge commit too19:20
Shrewsjeblair: wow, that hurts my head. If a project has 2 branches (A and B), each defining the same template, and the 'project' definition in B uses that template, what is supposed to happen? The templates are merged and the jobs from each are run?19:22
jeblairShrews: the templates are merged and the jobs from each are added to the list of jobs to *possibly* run.  however, all of those jobs will now have implied branch matchers, so only the jobs on B will run for changes to B19:23
jeblairoh hrm19:24
Shrewsjeblair: then what's the point of merging?19:24
jeblairi think there's a case i haven't thought through yet -- and that's that even if a template and all of its jobs were defined in zuul-jobs and therefore have no branch matchers, if the template is applied only in a stable branch, we need to add implied branch matchers there.  so i need to put back some of the project-template branch matcher logic i remove in that change.  (and add a test).19:25
Shrewsa test might help me see the end result more betterer19:26
jeblairShrews: the merging is just the method of reconciling multiple config objects with the same name.  we have to build the project pipeline definitions at config time, and we lose the templates and their context at that point.19:26
jeblairShrews: basically, by the time configuration is done, there aren't any templates any more.  just a single project definitions with all the jobs that project is supposed to run in the various pipelines.19:27
Shrews*nod*19:28
jeblairi'll wip it until i address the issue above19:28
ShrewsSpamapS: i find the "cbyrum committed 2 days ago" text very reassuring19:29
Shrewsall locked up, safe and sound19:29
jeblairthat explains the facial hair too19:29
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add implied branch matchers on 'master'  https://review.openstack.org/51445919:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Switch to late-binding inheritance  https://review.openstack.org/51135219:33
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move test_job_auth_inheritance to test_v3  https://review.openstack.org/51524919:33
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add pragma directive  https://review.openstack.org/51548319:33
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Support file extension in playbook path  https://review.openstack.org/51553519:33
SpamapSShrews: the point of the picture is to show the green check mark next to the commit by me, but the PR cares about the merge commit's status.19:33
SpamapSwhich is a little weird19:33
SpamapSand likely represents another github mismatch with gating in general19:34
ShrewsSpamapS: yep. i was just failing at a bit of humor there.  :)19:36
SpamapSShrews: wasn't sure it was clear enough is all. Glad you value my commitment.19:38
SpamapSlooking at the logic of _getChange19:38
SpamapSI think it may be broken19:38
SpamapShttps://github.com/openstack-infra/zuul/blob/feature/zuulv3/zuul/driver/github/githubconnection.py#L613-L63119:39
SpamapSHrm no, we do run _updateChange which should grab it from the API and update it.19:42
SpamapSow, so looks like my assumption that 3s was enough for github was wrong19:46
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Increase github delay to 10 seconds  https://review.openstack.org/51581219:49
SpamapSI wonder if we could start tracking the etags and using If-None-Match. Maybe GitHub will actually invalidate its cache if you get a miss there where without one it just returns whatever it has in its cache19:58
SpamapSThey also don't count 304's against rate limiting so that seems like a win.19:58
* SpamapS is going to try that next19:59
*** hashar has joined #zuul20:22
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add multi-branch support for project-templates  https://review.openstack.org/51580520:41
jeblairShrews, clarkb: ^ that's updated to fix the thing i was concerned about, and a test case to exercise it20:42
jeblairanyone around to review that ^?  i'd love to get it in with the others and restart21:27
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Update test fixtures to use explicit run  https://review.openstack.org/51553621:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove implied run  https://review.openstack.org/51548721:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Validate that a job has a run playbook on freeze  https://review.openstack.org/51553721:30
SpamapSwow... the github3 lib makes this.. hard21:31
*** hashar has quit IRC21:39
SpamapSoh poo, no, we're already doing it21:43
SpamapS wow, Github.. your API.. sucks then.21:43
* SpamapS seeing delays > 10s21:43
jeblairwow, that's pretty messed up21:49
jeblairi mean, we put in the 5s delay for gerrit because, you know, sometimes it was like 1 second behind so 5s seemed really generous.21:50
jeblair(tbh, i don't know how they both ended up with systems that emit events before they are durable, but hey)21:51
SpamapSjeblair: I think they're durable, but there are caches fronting the retrieval that don't get invalidated.21:54
SpamapSwhich you'd think would, you know, be keyed off the etag.21:55
jeblairi'm going to self-approve 515805 so i can get working on this restart.  retro-reviews are welcome.22:01
SpamapSwell that's quite a commit message. :-D22:09
jeblairfungi: i replied to the msg on openstack-dev, directing my reply to openstack-infra22:18
SpamapSjeblair: so, secrets usage question. I have a job which runs some ansible against nodes. I'd like to ask the cloud those nodes run in to allocate a loadbalancer using octavia, and point at those nodes..22:45
SpamapSjeblair: 1) Can I even do that anywhere in a check pipeline safely?22:45
SpamapSjeblair: 2) If so, how? if not.. do you have suggestions how we might test the loadbalancer interactions before gating?22:46
SpamapSto be more clear, I planned on putting secrets in my job definition and having the ansible for the job set up the loadbalancer.22:51
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add multi-branch support for project-templates  https://review.openstack.org/51580523:09
jeblairSpamapS: i think you can make a pre-run playbook to set up the load balancer (and a post-run to destroy it).  if you put those playbooks and the job that uses them in a trusted project, it's safe for those playbooks to use secrets.  you can then inherit from *that* job and put that child anywhere and have it do anything in any pipeline.23:11
jeblairSpamapS: like this http://paste.openstack.org/show/624857/23:14
jeblairSpamapS: with more actual correct zuul syntax: http://paste.openstack.org/show/624858/23:15
SpamapSjeblair: ok, and I can just have the pre leave the details of the LB somewhere on disk for the run to find.23:24
SpamapS(mostly just need the vip of the lb so I can configure things properly and test against the vip)23:25
jeblairSpamapS: ah yep.  can drop them in the executor.... i wonder if zuul_return would work... lemme check23:25
SpamapSjeblair: thanks. I was reading the manual and just not sure. It kind of feels, when reading, like you just can't use a post-review job in a pre-review pipeline.23:25
jeblairSpamapS: that is true -- you can't use a post-review job in pre-review.  however, a job is only post-review if either a) you set it explicitly, or b) it has secrets attached to it in an untrusted repo.  but we're attaching secrets in a *trusted* repo, so it's okay.23:27
SpamapSjeblair: I think we're going to need pictures in the manuals, especially with the inheritance changes (which are awesome btw, I'm about to start running distinct tests per branch here!)23:30
SpamapSlike, I ingested all of that at one point23:30
SpamapSbut it didn't really sink in until I tried to put it in a zuul.yaml23:30
SpamapSjeblair: thanks for explaining slowly and patiently for me. :)23:31
jeblairSpamapS: no problem23:31
jeblairSpamapS: i don't think you can use zuul_return to ship the data between playbooks... i'll think about whether we could adapt it for that, but i suspect it may be dangerous.  so in the mean time, probably just handle that yourself with a private file on the executor.  you can stick it in the jobdir/trusted directory if you want to make sure the child jobs can't manipulate it (eg, make the post playbook destroy the wrong load balancer)23:33
jeblairSpamapS: but obviously you'll need at least a copy of the info in jobdir/work for the child job to be able to pick it up23:33
SpamapSjeblair: right, zuul_return seems more like a way to ship things from job -> job not so much from pre->run.23:34
jeblairyep23:34
SpamapSjeblair: and my plan for not letting the run playbook mess with it is that the name of the lbaas is the build UUID which should allow me to find and delete it no matter what happens before pre tries to run.23:38
SpamapSall I really need is that vip23:38
jeblairSpamapS: cool, that sound solid.23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!