Friday, 2017-04-28

*** dkranz has joined #zuul01:00
*** SotK has quit IRC01:06
*** SotK has joined #zuul01:06
*** yolanda has quit IRC01:55
*** SotK has quit IRC02:22
*** SotK has joined #zuul02:25
SpamapSweird... I can't get gear's test suite to run03:06
*** bhavik1 has joined #zuul04:48
*** Cibo has joined #zuul09:05
*** Cibo_ has quit IRC09:07
*** Cibo_ has joined #zuul09:10
*** Cibo has quit IRC09:11
*** Cibo_ has quit IRC10:07
*** bhavik1 has quit IRC10:17
*** jkilpatr has quit IRC10:42
*** jkilpatr has joined #zuul11:10
Shrewsmorning12:20
tobiashevening ;)12:33
tobiash(almost)12:33
*** Cibo_ has joined #zuul12:34
tobiashare there any objections adding a max-ready-age setting to the label config and cleaning up nodes which are longer than that in ready state?12:35
tobiashI have two use cases for this, first ensure that the slaves are not older than e.g. one hour (for e.g. forcing a recent cache)12:37
tobiashsecond we try to scale down to a minimum during non-office times12:37
tobiashso we have a config for office hours with a high min-ready and a config for non-office hours with min-ready of 112:37
tobiashthis would also allow for nodepool initiated scale down of unneeded nodes12:38
*** dkranz_ has joined #zuul12:40
*** dkranz has quit IRC12:40
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix CleanupWorker exception messages  https://review.openstack.org/46101912:44
tobiashsomething like http://paste.openstack.org/show/608324/12:47
tobiashthe work could then be done by one of the existing cleanup worker threads or a separate one12:55
Shrewstobiash: so you're really talking about 2 things: scaling down when min-ready changes, AND having a max-ready-age13:10
tobiashShrews: yes, both could be done by max-ready-age (where scale down would be delayed by max-ready-age)13:20
Shrewstobiash: true13:21
Shrewsi can't see any immediate problem with implementing that, but pabelanger and jeblair should probably weigh in from an ops perspective13:23
mordredShrews, tobiash: sounds like a good addition to me - but I also agree, i'd lke pabelanger and jeblair to chime in13:29
jeblairyeah, max-ready-age sounds good.13:33
tobiash:)13:34
tobiashthen I'll try that13:35
tobiashthx13:35
tobiashpatch proposal will follow probably next week13:35
*** Cibo_ has quit IRC13:49
*** dmsimard is now known as dmsimard|cavern14:44
pabelangermax-ready-age seems fine14:49
clarkbrefular cleanup should already do that I think?14:58
clarkbset the max age and cleanup frequency and that should do it?14:58
jeblairclarkb: we only have a max age for images, not nodes14:59
pabelangerindeed! When you have to pay for cloud resources out of pocket, it is nice to have a way to limit nodes for off hours :)15:12
clarkbjeblair: ah15:15
*** openstackgerrit has quit IRC15:17
*** Cibo_ has joined #zuul15:22
*** Cibo_ has quit IRC15:42
*** yolanda has joined #zuul15:43
eventingmonkeyHey SpamapS15:56
SpamapSeventingmonkey: so, you were thinking of tackling https://storyboard.openstack.org/#!/story/2000879 yeah?15:56
eventingmonkeyYes, I am.15:56
SpamapSeventingmonkey: so what we need is a thread running in the zuul executor that walks the job dirs and if there is a job that has exceeded the defined limit per-job of bytes on disk, kill the job.15:56
SpamapSand kill it softly enough that it cleans up its job dir ;)15:57
SpamapSeventingmonkey: so you'll need to understand a) how to kill jobs (there's a way) and b) how to find out all the running jobs and their job dirs (I believe that may only be held in memory)15:58
eventingmonkeyKilling it softly with words. Got it.15:58
SpamapSor with your song15:58
pabelangercouldn't we use tmpreaper for that? I was actually considering adding http://git.openstack.org/cgit/openstack-infra/puppet-tmpreaper to our zuul-executors to help15:58
SpamapSpabelanger: that's the suspenders that helps with wayward executors15:58
pabelangerI guess this is to protect against a single jobs from filling HDD?15:58
SpamapSpabelanger: but this is for a perfectly healthy executor that knows all the jobs that are running, and yet, one of the jobs has exceeded its allocated storage.15:59
SpamapSNote that we had a pretty good discussion about this in Atlanta15:59
SpamapSand we probably should have written it down as a spec15:59
eventingmonkeylol15:59
pabelangerYa, remembering now15:59
SpamapSbut basically, like SELinux, the best system ways to do this are hard for operators to deploy16:00
SpamapSLike ideally, we'd just have an LVM volume group and mount each job dir as an LV with a defined size.16:00
SpamapSa thin LV too, so that we can overcommit16:00
pabelangerdoes bubblewrap profile file limits for the chroot too?16:00
pabelangers/profile/provide16:00
SpamapSno, bubblewrap doesn't really know how to do that.16:00
pabelangerack16:01
SpamapSand there's no ulimit for bytes on disk16:01
SpamapSand quotas are per-user (except in XFS, which has per-project, which dirs can be marked with xattrs, but.. XFS)16:01
pabelangerI thought clarkb or fungi also mentioned lvm might do something too?16:02
SpamapSpabelanger: before I take eventingmonkey monkey too far down this road.. isn't there a pile of ansible to write still?16:02
eventingmonkeySpamapS: these things are all in memory...care to help by putting them in words?16:02
SpamapSpabelanger: LVM is perfect for this. But setting it up requires quite a bit of work.16:02
clarkbzfs too16:02
clarkbbut thats an even bigger mess16:02
SpamapSyyyyyyyyyyeeeeeeeeeeeeeeeaaaaaah16:02
pabelangercoolio!16:02
SpamapSSo one alternative method we could explore..16:03
clarkbbut zfs totally has the dir quota set command16:03
SpamapSis having Zuul throw a big loopback on disk if you don't give it a block device.16:03
SpamapSand just carve out job dirs from a vg created on that loop16:03
SpamapSclarkb: yeah, I forgot zfs, because.... I forgot zfs. ;)16:03
clarkbSpamapS: you need more freebsd in your life :P16:03
* fungi bets butterfs does too. kitchen sink and all16:03
* eventingmonkey is still surprised btrfs didn't catch on more16:04
pabelangerSpamapS: ya, that's the discussion I am remembering about loopback on disk16:04
SpamapSone problem with doing it that way16:05
SpamapSis it requires root16:05
eventingmonkeySpamapS and pabelanger: So...should I dive into this one, or start poking at ansible?16:05
*** openstackgerrit has joined #zuul16:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the security context of the playbook when checking out roles  https://review.openstack.org/46109216:05
SpamapSup until now, we haven't required root to do executor's job16:05
SpamapSeventingmonkey: as we're talking this out, I'm wondering if this is a bit more complicated than I'd originally thought16:06
eventingmonkeySpamapS: Yeah, I'm getting that impression...16:07
pabelangerjeblair: we need to land https://review.openstack.org/#/c/459728/ before we can start up zuulv3-dev again right?16:07
jlkjeblair: mordred: SpamapS: ya'll around to chat naming issues for github events?16:11
jeblairpabelanger: i don't think so, but we should land it soon.  i will probably rip out that code in the not too distant future.16:12
jeblairjlk: yes!16:12
* jlk pulls up the review tree16:13
SpamapSjlk: I'm here.16:13
jeblairmordred, clarkb, SpamapS, jlk: before i forget, https://review.openstack.org/461092 can use some good security brain thought when you have a minute.  i think that is correct now.  (of course, it could also use tests of each of the cases)16:14
SpamapSand I think I'm in the "pass through unless it makes users go o_O" camp at this point.16:14
jlk(opened that tab for later)16:14
* SpamapS will try and find a good security brain somewhere ;)16:14
jlkokay https://review.openstack.org/#/c/439834/ is where I think the first feedback came in16:15
jlkwhere we're considering what to do with the events.16:15
jlkI'm pulling up my code, one sec16:15
pabelangerjeblair: okay, I'm not sure where we stand about bring zuulv3-dev.o.o back online. Is there anything stopping us from doing so?16:16
jeblairyep, briefly, that's the one where it looks really simple: "hey just pass them through, dude".  then later changes make the question stickier.  :)16:16
jeblairpabelanger: i'm not aware of any16:16
jlkhttps://github.com/j2sol/zuul/blob/github-v3/zuul/driver/github/githubconnection.py  (yes github, but I couldn't find a gitweb link easily)16:16
jeblairpabelanger: i think we landed the immediately necessary fixes16:16
jlkthat's the current state of the file.16:16
pabelangerjeblair: great, I'll pip install latest version of zuul and bring it back online16:17
jlkhttps://developer.github.com/webhooks/#events is the set of events Github could send us16:17
jeblairand here's an etherpad: https://etherpad.openstack.org/p/7HFyMRf51916:17
jlkoh cool16:18
jlkright so lots are buried behind a "pull_request" event, namely (un)label, (re)opened, closed, syncronized (content changes).16:19
SpamapSI'm missing context16:19
SpamapSwhat makes mechanically just passing them through with .'s between object.action complicated later?16:20
jeblairi think the mechanical translation is pretty straightforward for pull_request16:20
jlkbut the ohter events we care about is status (this is like a CI vote), pull_request_review (this is like a human vote), issue_comment (this is somebody commenting on the pull request), and maybe create (for tags being created, but that's out of scope for today)16:20
jlkso pull_reqeust.label pull_request.closed etc..16:20
jlkand really, what we're talking about is what we expose in the layout file, right?16:21
jlkthe thing that consumers will interact with?16:21
jeblairjlk: yes16:21
jeblairyep.  and honestly, i thought i had a pretty good handle on what this would look like until i got to "issue_comment"16:22
jlkyeah...16:22
jlkso pull requests are "special" issues16:22
jeblairthat's the one where i throw up my hands and curse at github16:22
jlkevery PR Is an issue, but not every issue is a PR16:22
clarkband in fact to close a PR you ahve to go through the issue api not PR api16:22
clarkbits really :(16:22
jlknow, we could mask that16:22
jlkand fake a pull_request.comment16:22
jlkinternally we'd trap the issue_comment, see if it's a pull request, and present it up the stack as a pull_request.comment16:23
jlkjeblair: how do you see this looking in the layout, to capture the sub pull-request elements? Could you toss something quickly in the etherpad?16:26
jeblairyep.  i think the pr/issue thing is confusing enough for new users that it's worth seriously considering that.  i guess the question is, which is easier to read: the one that says 'pr-comment' and we have an asterisk in the docs that says "this is really the issue_comment github hook" or the one that says "issue_comment" with an asterisk in the zuul yaml that says "pull requests are issues".16:26
jeblairjlk: yeah, i'll type things16:26
jeblairha, habitual typing of 'gerrit' :)16:27
SpamapS+1 for the *16:27
jlkeven as a user of Github for years, it was surprising to me that PRs are issues. I think pr-comment  would  be easier for github users to grasp. We could asterisk it somewhere that we're injesting issue-comment and translating it, but it's really just for the pedants who know the API who would otherwise get upset.16:27
clarkbyou'll also notice that issues and PRs share the same id sequence16:29
jeblairokay, style (A) and (B) are, i think, fairly reasonable examples of what that would look like.  they both need a separate event stanza so that the 'comment' comparison is only applied to the comment event.16:31
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role  https://review.openstack.org/45906616:31
jeblairso it's really only the one line that would change.16:31
jlkYeah, I think I'm still in favor of B16:31
jlkif I see A, I'm going to immediately ask "what issue?"16:32
clarkbone thing to keep in mind is that users are likely to configure pipelinse very rarely. So either option is likely workable16:32
jeblairso it sounds like we're all in favor of saying issue_comment is bonghits enough that it crosses the line into something we need to mask.  i can dig it.16:32
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role  https://review.openstack.org/45906616:32
jeblairclarkb: yes, and probably quite a bit from copy pasta16:32
pabelangerokay, zuulv3-dev.o.o is back online16:32
jeblairespecially if we do a good job writing the manual :)16:33
jlkpabelanger: \-/16:33
jeblairjlk: is that geordi raising his arms?16:33
jlkI think it was supposed to be a pint glass of beer or something.16:33
jlkbut I'm not so good with the ASCII16:33
jeblairoh, i can see a whiskey shot16:34
jeblairokay, so the other icky thing is the push/tag issue16:34
jlk|~|  <---  rocks glass.16:34
jlkright.16:34
jlkSo, we _could_ have zuul trigger whenever code is pushed to a specific branch16:35
jlklike, push to master, trigger some kind of deployment thingy16:35
jlkor regression tests16:35
jlkthat's outside of a "pull request" realm, and more into just "repository actions" realm16:35
jeblairjlk: yep.  and this is a surprisingly direct mapping to gerrit's ref-updated event.16:36
jeblairwhich we use for several pipelines in openstack16:36
clarkbjeblair: re the security related change you posted. Would it be simpler to just not allow trusted repos to dep on untrusted repos?16:36
jlkcool. We could even keep the same name yeah? It's not super user friendly, but... probably okay if it's used elsewhere in zuul16:36
* mordred reads scrollback16:38
pabelangerjust noticed a new exception in zuul_stream.py16:40
jeblairjlk: well, i'm of two minds here... first, i kind of want to stick to the mechanical passthrough when possible.  but two, the push/tag split that the github patches do is potentially useful.16:40
pabelangerhttp://paste.openstack.org/show/608361/16:40
jeblairclarkb: lemme get back to you on that16:40
pabelangerlooks like possible decode issue16:40
jlkjeblair: oh I think I see. Because ref-updated on gerrit catches tags too?16:40
jeblairjlk: yes, just like github push, actually.16:41
jeblair(the github patches take the github "push" event and split it out into either a "push" or "tag" event depending on what the target ref is)16:41
jlkyeah, makes regexes harder16:41
jlkI bet github originally just sent push, and then later broke out the tag event, but kept tags going through push for backwards compat16:42
jeblairjlk: no that's the thing -- github doesn't have a tag event16:42
jeblairthat's a construction of, i guess, Jan's16:42
jlkwell16:43
mordredyah - github is the same as gerrit here - it sends an event called "push" (which gerrit sends as "ref-updated")16:43
jlkit has a "create" event16:43
jlkwhen a branch or tag is created16:43
jlknot the same as a tag getting updated tho :/16:43
jeblairif you look at 443947 githubconnection.py lines 88-91, that's where it conditionally creates a push or tag event16:43
mordreda tag should never be updated :)16:43
jeblairare we sure those things aren't all just push events as well?16:44
jlkhttps://github.com/j2sol/zuul/blob/github-v3/zuul/driver/github/githubconnection.py#L9116:44
jlkhttps://developer.github.com/v3/activity/events/types/#createevent16:44
jeblairlike, i wonder if, when you create a branch, you get a "create" and a "push" event?16:44
jeblairor do you just get "create", and "push" only happens on update?16:45
jlkit looks remarkably like push event16:45
jlkbut with fewer fields16:45
jlkjeblair: good question, I can test that real quick16:45
jeblaircool, i'll type more in etherpad while you do that16:45
jlkpusing a new branch did get a "create" event16:47
jlkand a push event16:47
jlkyou get both16:48
jeblairwe're rich!16:48
jlksame for tags16:48
jeblairsetting aside create for the moment, i added two new stanzas to the bottom of the etherpad16:49
jeblairthe configuration for a 'post-merge' pipeline (even though you might choose to use pull_request.closed on github for this, this pretty much looks the same as what you'd do for 'change landed on master')16:49
jlkyeah that could work. I don't see a strong need to split ref update / tag into separate events16:49
jeblairand then the configuration for a 'tag' pipeline16:49
jlkpr closed might be something different16:50
jeblairbasically you just always have to use a regex in there.  and with github, the regex is pretty straightforward.16:50
jlkkeying off of "when new code hits this branch" is, I think, a better mind set.16:50
jeblairthough, you do have to at least understand a little bit about git.16:50
mordredwe have found that people understand git _way_ less than the zeitgeist would have us believe16:51
jeblairthe regex is pretty ugly for gerrit, because it does something weird to the ref if it's a branch (it drops 'refs/heads/')16:51
jeblairmordred: yeah16:51
jlkyeah this is pretty straight forward, and fairly easy to document in our documentation.16:51
jlka branch is in refs/heads  and a tag is in refs/tags16:51
jlkand if somebody wants to do crazy ass things with the git driver and be able to trigger on OTHER things in refs/, continuing to use a regex seems best16:52
jeblairso i think there's a compelling case for "make some simple synthetic events for 'branch head was updated' and 'tag was updated'".  but i think that needs to happen for both drivers, and i think we need to make sure we don't accidentally make it hard to trigger on refs/notes/foo.16:52
* jlk imagines somebody keying off of a git note in refs/notes/16:52
mordredyes. this is sort of what I was thinking in my brainhole16:53
clarkbyou could even use it against refs/changes16:53
jeblairso i'm thinking we should limit ourselves to passthrough the event as-is for now, effectively requiring regexes, document them so people can copy-pasta them, then later, maybe add synthetic events to both drivers16:53
mordredmaybe we make three synthetic events: push, tag and ref - and we put any ref-updated/push events that don't match head or tag into ref?16:53
jlkI'm getting twitchy on "push"16:54
jlkbecause you push a tag16:54
jlkthese actions are hard to classify16:54
mordredjlk: head, tag and ref?16:54
jlkbranch, tag, ref ?16:54
mordredyah - I hearyou16:54
jeblairjlk: yeah, i think it makes sense to have the passthrough event called 'push'.  it really is all of them.16:54
jlknod16:54
mordredbranch, tag and push?16:54
jlkthat could be it.16:55
jlkbranch: master, tag: release-*, push: regexmebaby16:55
mordredthe reason I like synthetic events is that the associated operations on them are git operations, not necessarily gerrit or github operations16:55
jeblairmordred: my variant on your proposal is that we keep the passthrough events (ref-updated, and push, respectively).  then we add "branch-updated" and "tag" or whatever as additional synthetic events later.  i think we would end up firing both events in zuul in that case.16:55
mordredjeblair: I think that could work nicely if we're ok firing both events16:56
jeblair(er, both events in the case of someone pushing a tag.  we'd get 'ref-updated'/'push' as well as 'tag')16:56
jeblair(obvs only one event in the case of someone pushing /refs/notes/foo)16:56
mordredit's mostly that things like responding to "branch-created" vs. "branch-updated" all involve some ugly git foo - whereas the other tihngs "pr-created" make sense to a normal user16:57
jeblairmordred: yeah, i think the cost of that is small, and we can avoid backwards compatability snafus that way.  we don't have to say "in zuul v3.7, pushes to refs/notes now emit a 'note' event *instead of* a 'push' event)16:57
mordredso it's just mostly  my twitchy "exposing the guts is powerful, but potentially hard for new users to undestand"16:57
jeblaircost == cost of sending 2 events.16:57
mordredjeblair: ++16:57
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Validate flavor specification in config  https://review.openstack.org/45187516:58
*** Cibo_ has joined #zuul16:59
jeblairmordred: yeah.  i want cake and eating cake.  :)  [to seriousify that, we've benefited greatly from making things like this easy, but we also created 4 kinds of pipelines we never dreamed of without having to write any python code by exposing these things.  so i like both.]16:59
jlkso to recap this, if I understand it right.17:00
jlkFor Github, keep the event as "push" for github, and within push handle both tags/branches by way of regex, instead of synthesizing it.17:00
jlkAt a later time, we'll do a proposal to synthesize some branch vs tag type events, that will cover all the drivers.17:01
jeblairjlk: yep (and that mostly means the github driver doesn't have to do anything smart with push right now)17:01
jlkA+17:02
jeblairone more borderline bikeshed:17:02
jeblairwe've been going with 'pull_request.opened' as the mechanical translation of a "pull_request" event with action="opened".17:03
jeblairstrictly speaking, the zuul config language can handle "{event: pull_request, action: opened}" just fine.17:03
jlkhrm17:03
jeblairi wrote up style(C) in the etherpad to show what that would look like17:04
jlkright17:04
jlkDO we use dot notation anywhere else within layouts?17:05
jlkbecause if not, style C seems more "natural"17:05
jeblairnope, the dot notation is new17:05
clarkb+1 to verbose layout17:05
jeblairif you were to take a really literal view of how the gerrit trigger is configured (basically arbitrary matching of paremeters), it would actually look more like C.17:05
*** harlowja has quit IRC17:06
jlkyup, I lean toward C17:06
jeblairi'm okay with dot notation if folks want it and it feels more human-friendly.17:06
jlkYou also had input about preventing comment on reporting17:07
jeblairbut i'm also okay with C.  C probably makes our "pull_request comment" lie slightly more of a lie (and maybe more confusing).17:07
jlkand I'd like to go through that with you if you're ready.17:07
jeblairmordred: ^ you proposed the dot notation -- i don't want ^ to slip by without you having a chance to scream.17:08
jeblairjlk: oh yes17:08
jeblairjlk: 444060, right?17:08
jeblairjlk: do you have any links to what a zuul comment in github looks like?17:09
jlkyes17:09
jlkoh I was saying ues to 44406017:09
jlkI'll get to that in a moment, but I want to explain the reasoning.17:09
jeblairheh, figured :)17:09
jeblairok17:09
jlkIn Github, there is the concept of a "status"17:09
jlkhttps://github.com/BonnyCI/zuul/pull/4717:10
jlkhas such a status17:10
jlkcheck_github has failed, and we set the status to failure.17:10
jlkGithub has traditionally used status on a commit hash to hold CI responses17:11
jlkand in fact, "branch protection" is a set of conditions that must be met before somebody can push the "merge" button on a PR. One such protection is that an appropriate status is set.17:11
jlkWhen a status is set, it does not cause an email from github to go out17:12
jlkwhereas comments get emailed to everybody who's attached to the ... issue.17:12
jlkthat gets noisy in a hurry, and people seem to not like it.17:12
jlk(one sec, dog needs out)17:12
jeblairso does the status show up as the little red X next to b6b5f9a above the comment?17:14
jlkIn general, robot actions on github get treated different than human actions.17:14
jlkjeblair: yes, it shows up as the x. And for people with merge rights, it shows up in a summary too17:14
jeblairwhen you set a status, you also set a url; where is that used?17:15
SpamapSthat's for the log output of the job17:15
jlkthat red X is clickable17:15
SpamapS"Details" in the UI17:15
jlkthat's where the URL comes into play17:15
jeblairha, HCI affordance failure.  :)17:15
jeblairit moused-over as "Failure" so i did not assume it was a link.  :)17:16
jlkgotcha17:16
jlkI don't know if you see a summary down at the bottom of the PR of the reviews and statuses17:16
jeblair(i see the link at the bottom now though)17:16
jlkat least there it says "Details"17:16
jeblairno summary for anonymous me17:16
mordredjeblair: I'm perfectly fine with C17:16
jlkk17:16
jlkGithub has 3 different ways to provide input17:17
jlkthere's the status we're talking about, where CI robots put pending, success, fail.17:17
jlkthere's commentary on the PR directly17:17
jlkthere's commentary in the changes view of the PR (yes, it's different)17:17
jlkand there's a new system called "Reviews"  (I lied there's 4)17:17
jlkWe still need to gather data from users about where they'd like BonnyCI to report things. whether a comment on failure is appropriate, or if the status is enough. We're pretty sure they don't want a comment on start, and probably don't want a comment on success17:18
jlkhttps://github.com/BonnyCI/zuul/pull/40 is an example where a job failed17:19
mordredare the different places one can report things that could be exposed at pipeline config time? so that a user could say "please report statuses only to robot status" or "please also to make comments" ?17:19
jlkwe comment on failure, providing the link17:20
jeblairjlk: https://github.com/BonnyCI/zuul/pull/47#issuecomment-296492291 looks like a failure comment, right?17:20
jlkmordred: as it stands in the patch set, yes.  pipelines can influence whether you get status or comment or both or neither17:20
mordredso the strategy is currently "put pending, sucess and failure status in the robot place, and also on failure make a comment with links"?17:20
jlkjeblair: oh right, yes there was a failure there too.17:21
jlkmordred: that's BonnyCI's strategy17:21
jlkmostly because a "status" is attached to a given commit hash17:21
jeblairthe only way to get an itemized report of the run is still with an actual comment though, right?17:21
jlkif somebody updates the PR with a wholly new commit hash, that failing status is lost17:21
mordredyah. so if you update the PR, the status would no longer be valid, right?17:21
mordredah - that too17:21
jlkjeblair: that's correct.17:21
*** Cibo has joined #zuul17:21
mordredso you'd also lose job history without the failure comments17:22
jlkjeblair: unless17:22
jlkjeblair: unless we make the URL not go directly to the logs, but instead go to a summary page17:22
jlkmordred: correct.  This is another github bonghit17:22
mordredfor myself, I really like being able to get the content directly in a change/pr without having to go exit to another system17:23
jlkthat summary page thing could be entirely a BonnyCI thing, since we can control the URL passed to the status.17:23
jeblairjlk: right.  we don't have a way to generate a summary page at the moment (the closest thing we have is the link to the log index page, which it looks like you're doing.17:23
jlkmordred: indeed, and this is definitely an area where we need to gather input from a wide swath of users17:23
*** Cibo_ has quit IRC17:23
mordredjlk: ++17:24
jlkI could totally see comment on success/failure being a way users decide they want to go17:24
jlkmy HUNCH is that less people care about a summary of success and more care about a summary on failure17:24
mordredyup17:24
jeblairjlk: i think summary page generation would be useful for zuul in general, so if y'all write that, please upstream it.  :)17:24
mordredthat's my hunch too - and I'd argue is a great 'default'17:24
jlkjeblair: totes!17:24
mordredut I could see a project deciding they did not want the failure comments17:25
mordredI think it's unlikely anyone would want the statuses not posted to the robot status thing17:25
mordred(well, except for thing-party-ci's where they don't have access to do so)17:25
jlkwell...17:25
jlkstatus reporting is pretty wide open I think.17:25
jeblairso i do now see the need to disable comments.  so how about we keep that feature, but default it to 'leave comments' so that novice zuul users get useful output.  but it's easy to turn off.17:26
jlkoh I take that back17:26
mordredso the status just needs _someone_ to post success? or can you say for protected branches "please only trust statuses from XXX"17:26
jlkjeblair: I'm totally fine with that.17:26
jlkmordred: I was wrong. YOu can grant specifically teh right to set status17:27
mordredok. cool17:27
jlkand you can also narrow down _which_ status is required for merging17:27
jeblairmordred: i'm 95% with you, except i can concieve of wanting to do 'pending' followed by 'success' even on failure (it closes out the action, but doesn't splat a red x on things.  think of "nonvoting pipeline")17:27
mordredjeblair: yah. I could see that too17:27
jlkyeah it gets ugly because an _overall_ status is used for the UI display17:27
jlkall statuses pass, you get green. ANY status fails, you get red17:28
SpamapSstatus reporting requires write access17:28
* SpamapS is a bit lagged out17:28
mordredeven if it's a non-binding status I guess?17:28
jlkwhether or not that red will BLOCK the merge is a different storry17:28
mordredfantastic17:28
jlkmordred: I believe so, but I can't easily set up a test of that17:28
jeblairthat's what prompted my other suggestion, which was change status to an enum rather than bool in zuul config.  are there any problems with that?17:28
mordredno worries - mostly just learning - the answer isn't specifically germane17:28
jlkSpamapS: by default yes, but you can explicitly grant status set rights without merge rights.17:28
SpamapSjlk: oh that's handy17:29
jlkjeblair: can you sketch out your enum suggestion on the etherpad?17:29
jeblairyep17:30
*** persia has quit IRC17:30
jeblairjlk: at bottom17:31
jlkOkay, so in that case, the end user has to know which status states are allowed17:31
jlkbut that's fine17:31
jeblairjlk: i think the current version of the patch just has it hard-coded anyway17:32
jlkit does17:32
jlkit takes the decision away, but it also means consumers don't have to learn the restrictions.17:32
jlkbut, I'm not strongly tied to that.17:33
jeblairjlk: what restrictions?17:33
jlkI'm okay with an enum17:33
jeblairi mean, i know they may not have perms for a given status17:33
jlkjeblair: the Github API will only accept pending, success, or fail.17:33
jlkyou can't do status: bonkers17:33
jeblairbut the current version of the patch would fail if you said "status: true" but couldn't set success17:33
jeblairjlk: oh yeah17:33
jeblairjlk: we can validate the enums in zuul17:33
jlkcool17:34
jeblair(we just can't (easily) validate that they have permission)17:34
*** Cibo_ has joined #zuul17:34
jlkyeah the permissions don't come into play. Status perms are all or nothing.17:34
jeblairbut that's true for many, many things in the zuul conf.17:34
jlkAnybody else have objections to using a (validated) enum for status?17:35
jlkthe code for this is kind of wonky, because it all falls under "report", and the report action in github driver has to tease out what all reports should be done, calling out to other sub-functions.  It's do-able, but only slightly awkward.17:36
*** Cibo has quit IRC17:36
jeblairjlk: yay abstraction layers!17:37
jlkalright, was there any other big things to discuss?17:38
jeblairjlk: i *think* that's all open ended questions i left masquerading as review comments..., yeah?17:38
jlkI think so17:38
jlkI have one thing that's not directly related, that I wanted to wait until after17:38
jeblairafter like now, or after like after now? :)17:39
jlkafter like now would work17:39
jeblairok17:39
jlkSo you've done work in a way that a pipeline could service for multiple drivers17:39
jlkor sources?17:39
jlkyou took source out of the top pipeline stanza, and let there be a source reference for trigger/success/fail17:40
jlk(is it a configured source name, or is it a driver name?)17:40
jeblairjlk: yeah, pipelines don't have sources anymore.  so a single pipeline can be triggered by different drivers, and changes from both drivers can end up in that pipeline and they should be happy.17:41
jeblairjlk: sources are attached to projects directly17:41
jlkI get confused because in the examples and tests we tend to name the source after the driver17:41
jeblair(so those examples on the etherpad where i put gerrit and github together under trigger were legit)17:41
jlkso it's hard to see which is a named source reference and which is a driver name reference17:41
jlkeither way, the intent is mixed drivers, one pipeline17:42
jeblairyeah, and it doesn't help that this stuff is changing.  :)17:42
jeblairlemme take a crack at it:17:42
jeblaira pipeline has *triggers* which are *connections* to remote servers implemented by *drivers*.  a *trigger* may emit an *event* with an associated *change* for a *project* which is fetched from a *source* over a *connection*.17:44
jlkThe reason I bring it up, is regardless of driver or source, mixing them in a pipeline breaks down at the require/reject level: https://review.openstack.org/#/c/449390/6/tests/fixtures/layouts/requirements-github.yaml17:44
jlkUnless we try to map all drivers features into gerritisms for require/reject, I think we'll wind up with incompatibilities17:45
jlk(or we come up with zuulisms to map everything to)17:46
jeblairjlk: yep, that's going to be muddled.17:46
jlkIn the pipeline there, if I were to try to throw gerrit projects at that pipeline, it'd fail.17:46
jlkWould it make sense, in the future, to put require/reject into the same connection break down?17:46
jlkrequire: github: <something>     require: gerrit: <something_else>17:47
clarkbthat is what i was just going to sugget17:47
jeblairit's currently a fairly limited set of things you can require, and they are aspects of the Change, so it's not inconceivable that we could abstract them.17:47
clarkband then only apply the requirements to the corresponding trigger17:47
jeblairclarkb: well, these requirements should be independent of the trigger, though they may not be independent of the underlying connection/driver17:48
jlkIn github land, we would want to be able to require a status (see status discussion above), or a passing review from somebody with commit access (see discussion of inputs above), or a label being present, maybe even a comment being present.17:48
clarkbthe requirements control when a trigger applies right?17:48
jeblairclarkb: (imagine a pipeline triggered only by IRC, but still has a "status: open" pipeline requirement)17:48
jlkhttps://github.com/BonnyCI/hoist/blob/master/roles/zuul/templates/etc/zuul/config/layout.yaml#L46  is our v2.5 layout17:49
jeblairclarkb: we have two very similar things -- trigger requirements and pipeline requirements.  trigger requirements are easy, but pipeline requirements are, ideally, more independent17:49
jlkfor our "gate"17:49
jlkand there is a reason why we list these (partially) as both a trigger req _and_ a pipeline req17:50
jlkAny time you have multiple requirements, you have to accept that they'll come in unordered17:51
jlkyou're not going to get _both_ requirements in teh same event17:51
jlkso you need to allow yourself to "trigger" on any of the events that _could_ meet requirement, and then use pipeline requirements to see if all are met17:51
jlkunless, I Just had a light bulb moment17:51
jlkWe have to _trigger_ on all the potential events. A trigger requirement, will that fetch details about the change itself (not just the data in the event) and would be able to see if the other requirements are already in place?17:52
jeblairjlk: in the gerrit driver we did a weird 'change-has-approval' trigger requirement, so even if you get 2 approvals in sequence, you can have both requirements in the trigger req17:52
jlkI.E. could I move my pipeline requirements into being trigger requirements?17:52
jeblairjlk: yeah, i think you just invented that :)17:52
jlkalright, I think that would side-step the question of what to do with multi-driver pipline requirements.17:53
jeblairjlk: "required_approvals" is i think what we called it17:53
jlkthose have to be generic enough.17:53
jeblairjlk: (also spelled as 'require-approval')17:54
*** Cibo_ has quit IRC17:54
jlkjeblair: I think that's right. it was a little difficult to wrap my head around it, and I think I punted on trigger requirement at hte time. That was a mistake I think. I think these should have been implemented as trigger requirements instead of pipeline requirements.17:54
*** Cibo has joined #zuul17:54
jeblairjlk: oh, and actually, that's in the BaseFilter object, which is probably way on the wrong side of what should be an api layer.17:55
jeblairit's got some decidedly gerrit things in there17:55
jlkyeah17:55
jlkthis was an area that needed discussion to get right17:56
jlkWe plowed ahead with a brute force method, with knowledge that it'll likely get changed a lot to land in v317:56
jlkit's also where we had to start editing model.py more which felt wrong17:57
* SpamapS is caught up and nodding but has nothing to add17:57
jlkjeblair: now, if they were trigger requirements, the triggers we list in a pipline, can we have different triggers per connection using the same driver, or do all connections using the same driver have to share the same trigger configuration?17:59
jeblairin general, we should probably try to head to a point where the gerrit driver spits out a GerritChange(Change) and github driver emits GitHubPullRequest(Change).  and the Change object implements an api for abstract pipeline requirements (which they can override).  we're still going to need some kind of solution though for approvals, because that's just not going to abstract.  so we'll need to separate them by driver in the config, or extend the ...17:59
jeblair... language to boolean-or sets of pipeline requirements.17:59
jlksay I have a github and a github_enterprise connection. Both use github driver. Could I have one pipeline that requires one thing for github and something different for github_enterprise ?18:00
jeblairjlk: yes -- the word 'gerrit' and 'github' that we conventionally use in pipeline triggers is actually the connection name.  so "trigger: {github_enterprise: ..., github: ...}" is valid18:01
jlkOkay great.18:01
jlkokay I think I have enough to chew on today and into next week18:03
jeblairyay!  this has been fun.  thanks :)18:03
jlkthank you!18:03
jeblair16:36 < clarkb> jeblair: re the security related change you posted. Would it be simpler to just not allow trusted repos to dep on untrusted repos?18:03
jeblairclarkb: re that ^ -- i think it would be too constraining to say that a playbook in a trusted repo can't use certain roles.18:05
jeblairclarkb: i mean, we might set up some infra role repos which otherwise don't really need to be in a trusted repo.  but we may want to use them from project-config.18:05
* jlk goes for a bike ride18:05
jeblairclarkb: this should let us do that safely.18:06
*** yolanda has quit IRC18:07
clarkbjeblair: I'm just thinking that without any implementation details a trusted entity depending on an untrusted entity transitively makes the entire setup untrusted18:08
clarkbwe're trying to work around that here or we could possibly just avoid the issue entirely?18:08
jeblairclarkb: well, now i think we've gone and gotten ourselves in trouble with our new 'friendly' terminology of trusted/untrusted.  :)18:09
jeblairclarkb: the author of the playbook in the "trusted" repo *did* trust the maintainers of the role they chose to depend on enough to add that role as a dependency18:09
jeblairclarkb: the only thing that would be unexpected to them, is if someone convinced zuul to run their trusted playbook with un-merged code to that role.18:10
clarkbI see18:10
jeblairbecause that's trusting "random person on the internet" versus "the authors of this role" :)18:10
* SpamapS should take a break from o_O'ing at gear and how to elegantly provide a text facade :-P18:10
jeblairclarkb: so the super-short version of the change is "when we run an trusted playbook, we don't use speculative merging"18:12
jeblair* for roles  (it was already the case for the playbooks themselves)18:12
*** yolanda has joined #zuul18:16
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add support for specifying key-name per label  https://review.openstack.org/45546418:30
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move makemergeritem into model  https://review.openstack.org/46071418:38
*** harlowja has joined #zuul18:38
*** _ari_ is now known as _ari_|gone18:57
*** harlowja has quit IRC21:17
*** harlowja has joined #zuul21:17
*** dmsimard|cavern is now known as dmsimard21:20
*** harlowja has quit IRC21:36
*** cmurphy has quit IRC21:59
jlkjeblair: still around? Looking at implementing things from our meeting today. A pull request event could have one of many actions. Should we support actions as a list of things, or should we require adding a new event: pull_request block for each action?22:04
*** cmurphy has joined #zuul22:05
jeblairjlk: most things like that in zuul silently handle both.  there are some "as_list" and "to_list" helper methods that coerce single-items into lists.22:05
jeblairjlk: so maybe do that?22:05
jlkyeah trying to think of how that would look in the YAML22:05
jlktaking a stab in the etherpad22:05
jlkSee style D line 43? https://etherpad.openstack.org/p/7HFyMRf51922:06
SpamapSwe need to fix testrepository so it uses a db type that exists in python 2 and 322:08
SpamapSNo module named gdbm22:08
SpamapSsilly testr22:08
jlkgod damn bowel movement ?22:08
SpamapSjlk: we're not supposed to drink until 3:30 man. ;)22:09
jlkeh, I rode 20+ miles, I should be able to drink22:09
SpamapSjeblair: https://review.openstack.org/461172 redoes the TextJob and TextWorker to be simpler and use inheritance. Probably needs some docstrings.22:11
SpamapSjeblair: I did it as a new change becuase the old one was a huge mess.22:11
jeblairjlk: yep.  style d matches existing similar things to my recollection.22:13
jeblairSpamapS: cool!22:14
jeblairSpamapS: i'm trying to unload some gnarly git stuff from my brain right now, so will probably look at it monday22:14
clarkbSpamapS: the gdbm thing is independent of python2 and python322:15
clarkbSpamapS: also it works if you do python3 before python222:16
clarkbSpamapS: I have to install a separate package to get gdbm locally22:16
SpamapSjeblair: no rush. Thanks.22:17
SpamapSI may polish it a bit before then too.22:17
SpamapSclarkb: the 3 before 4 thing doesn't work with 3.522:17
SpamapSerr22:17
SpamapSclarkb: the 3 before 2 thing22:17
SpamapS3.5 dropped the thing that was backward compatible with 222:17
SpamapScan probably just put gdbm in test-requirements22:18
clarkbfun22:18
SpamapSbut really, seems like testr should just put gdbm in its reqs, and use that.22:19
clarkbthe problem is its part of the python build itself aiui22:19
clarkbnot something pip installable22:19
mordredyah. it's all about the python compilation22:19
jlkeww22:20
mordredya22:20
clarkbyou could put it in bindep though22:20
clarkbor maybe we just need tsetr to maintain two sets of dbs22:20
SpamapSlooks like it uses "anydbm"22:22
SpamapSbut that's kind of a flawed thing22:23
SpamapSsince it doesn't mean you'll have the same dbm across interpreters22:23
SpamapSI wonder how badly dumbdbm performs.. at least it is consistent across 2/322:24
SpamapSwhat a weird problem22:27
mordredSpamapS: yah - who would have thought?22:27
clarkbit would be fine if we just split by python major version I think22:27
SpamapSclarkb: indeed, that may be easier than figuring out how to make one that works for all22:28
SpamapSclarkb: you actually probably have to go by something unique to the build tho... as mordred said.. one could compile custom python that breaks it22:29
*** jkilpatr has quit IRC22:30
clarkbya though that seems less of a problem with tox at least22:32
clarkbsince tox is going to make a virtualenv that will stick around for some time22:32
SpamapShere's a thought22:32
SpamapSsqlite22:32
clarkbya I think mtreinish has considered a fork that does that22:33
SpamapSdo we have to fork it? AFAIK, the owners are all inactive, or lifeless22:33
clarkbmtreinish would know better than me. The impression I got was yes no one reviews code and no one else can get commit access22:33
SpamapSah yeah22:34
* SpamapS opens a window to let out the yak smell22:34
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger  https://review.openstack.org/46117622:55
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor  https://review.openstack.org/46117722:55
jeblairwhew! that's today's second fix for an 'unsuitable for production' issue.  it feels really good to exorcise that from my brain.22:57
mordredjeblair: brian exorcisms are the best22:59
mordredgah22:59
mordredbrain22:59
mordredjeblair: first patch no likey22:59
jeblairi guess that's what i get for running flake8 *before* 'git commit -p' and manually editing the resulting diff23:00
SpamapSmordred: I'm Brian, and so's my wife23:01
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger  https://review.openstack.org/46117623:02
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor  https://review.openstack.org/46117723:02
jeblairman, zuul and jenkins are just ganging up on me23:02
mordredthey do that23:03
mordredjeblair: those both actually make complete sense23:06
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks  https://review.openstack.org/43983423:24
jeblairmordred: \o/23:25
jlkegg23:25
jlkthat got the wrong topic, which I'll fix in the larger rebase23:25
jlkbut23:25
jlkjeblair: does that look like what you'd want to see?23:25
jeblairjlk: after a quick scan, yep!23:28
jlkcool. Tests pass locally. I'll await feedback in review before continuing on with the stuffs.23:28
jlkas an aside, is there an easy way to change the topic of a review, without needing to change the code of the review?23:28
jeblairjlk: i'm not sure validate_conf is used anywhere23:29
jeblairjlk: yes.23:30
jlkoops, I lifted that from the gerrit driver :)23:30
jeblairjlk: yeah, looks like it's dead code there too23:30
jlkneat!23:30
*** harlowja has joined #zuul23:30
jeblairjlk: i have changed the topic.  i'm sure there's some way to do that it the webui, but it's been years since i've used it.  it's C-t in gertty.23:31
jlkhahah23:31
jlkcheers, I'm going to go earn a beer by mowing the lawn!23:31
jeblairyou and fungi and23:31
jeblairclarkb have almost convinced me to mow my lawn23:32
jeblairbut i think i'm just going to sit and watch it grow instead23:32
clarkbI have to weed mine first23:32
clarkbthen mow it23:32
clarkbthen wonder why I got a place with a yard23:32
jeblairclarkb: kids can mow lawns starting at 6 months, rightL23:32
jeblair?23:32
clarkbjeblair: I wish23:32
clarkbthey "help" with the weeding though23:33
jlkstrap them to the remote control lawn mower and sure23:33
clarkbthen they blow the dandelion seeds everywhere23:33
jlkmy 10 yo started mowing some this past summer.  Right now he gets dog poop duty (hehehe, duty).23:33
clarkbalso all the rain makes yardwork a giant mess23:33
fungiyup, been having to time it between storms23:34
* SpamapS has a grass-free yard.23:34
fungiwith enough time to sort of dry back out23:34
jlkyeah, that's a big reason why I'm running out to do it now. This time of the year you gotta mow whenever i'ts dry, no matter the time.23:34
SpamapSsucculents and palms and moss :)23:34
jeblairfungi: don't the storms just rip your lawn away?23:34
clarkbSpamapS: I'm encouraging the succulents23:34
clarkbSpamapS: they are neat plants23:34
fungiSpamapS: soon so shall i. just need to keep the grass^H^H^H^H^Hweeds short enough to avoid town ordinance violations until i can relandscape23:35
fungijeblair: not often enough i'm afraid23:35
clarkbfungi: oh thats my other problem. I live in the county which means I basically have zero rules. Which means motivation at times is very low23:35
fungiSpamapS: but yeah, palms, ferns, bromeliads, succulents, cacti and other drought-tolerant flora23:36
SpamapSI also have some rather unruly rosemary bushes23:36
fungisoon to be rosemary trees23:36
clarkbyou just need to cook with rosemary every day23:36
clarkbI just planted rosemary with ^ in mind23:37
SpamapSI do scrape rosemary off them when I need some :)23:37
SpamapSfantastic in turkey23:37
fungiand in bread!23:38
*** harlowja has quit IRC23:55
*** jkilpatr has joined #zuul23:56
*** harlowja has joined #zuul23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!