| SpamapS | hmmmm | 00:03 |
|---|---|---|
| SpamapS | Unable to freeze job graph: Job syntax-check-ansible does not specify a run playbook | 00:03 |
| SpamapS | I got that in a PR when I didn't touch that job, or its parents. | 00:03 |
| SpamapS | did something change recently where we have to always specify a run: ? | 00:23 |
| fungi | jamielennox: as noted, i'm currently en route. you australians will have to disabuse me of prior cultural misconceptions involving paul hogan and fosters "beer" | 00:27 |
| jamielennox | well the fosters we can do, i don't even know where you can buy that here | 00:28 |
| jamielennox | unfortunately though everyone does look and sound pretty much like paul hogan | 00:29 |
| fungi | bush hats and large knives abound? | 00:31 |
| jlk | SpamapS: I didn't think so. I thought there was an assumed run: of a playbook matching the job's name | 00:34 |
| jeblair | SpamapS: yes, that's changing. always supply a run, and i'd go ahead and stick ".yaml" on it. | 00:34 |
| jeblair | see "The Bonus Change" section in http://lists.openstack.org/pipermail/openstack-infra/2017-October/005634.html for background | 00:37 |
| jeblair | (we now support extensions, and i'd like to drop implied extensions soon, or at least before release) | 00:40 |
| jeblair | (i submitted 78 changes to openstack to add .yaml extensions; i'm still waiting for 32 of them to merge) | 00:42 |
| ianw | jeblair: i can think of a quick way to get any of the important ones to merge ;) | 00:51 |
| fungi | ianw: well, we basically agreed in the last infra meeting (i think?) that we were not above using that "quick way" rather than let individual projects delay progress on such mass transitions | 01:53 |
| fungi | s/wre/we're/ | 01:53 |
| *** xinliang has quit IRC | 02:13 | |
| *** xinliang has joined #zuul | 02:29 | |
| *** xinliang has quit IRC | 02:29 | |
| *** xinliang has joined #zuul | 02:29 | |
| *** xinliang has quit IRC | 02:41 | |
| *** xinliang has joined #zuul | 02:54 | |
| *** xinliang has quit IRC | 02:54 | |
| *** xinliang has joined #zuul | 02:54 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-output role https://review.openstack.org/509233 | 07:22 |
| *** xinliang has quit IRC | 08:39 | |
| *** xinliang has joined #zuul | 08:52 | |
| *** xinliang has quit IRC | 08:52 | |
| *** xinliang has joined #zuul | 08:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add cloud quota handling https://review.openstack.org/503838 | 09:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 09:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 09:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 09:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 09:03 |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul feature/zuulv3: [WIP] Support autoholding nodes for specific changes/refs https://review.openstack.org/515169 | 09:21 |
| *** electrofelix has joined #zuul | 09:30 | |
| *** hashar has joined #zuul | 09:42 | |
| *** sambetts|afk is now known as sambetts | 10:20 | |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul feature/zuulv3: [WIP] Support autoholding nodes for specific changes/refs https://review.openstack.org/515169 | 10:32 |
| *** hashar is now known as hasharCooking | 11:12 | |
| *** hasharCooking is now known as hasharEat | 11:38 | |
| *** hasharEat is now known as hashar | 12:16 | |
| *** jkilpatr has joined #zuul | 12:16 | |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul feature/zuulv3: [WIP] Support autoholding nodes for specific changes/refs https://review.openstack.org/515169 | 12:42 |
| *** bhavik1 has joined #zuul | 13:03 | |
| kklimonda | jeblair: I've reworked my patch a bit - I'm now converting --change into ref internally and using that for autohold requests. I've not added --oldrev and --newrev yet, both because I'm not sure what they would be useful for, and I'd have to think more how to store them - definitely not as part of the key. | 13:05 |
| *** bhavik1 has quit IRC | 13:26 | |
| kklimonda | Also, with the regex approach for refs, I can use .* to scope the requests to jobs instead of explicitly handling them. | 13:47 |
| *** jkilpatr has quit IRC | 14:06 | |
| jlk | I have no idea why my change keeps timing out :/ | 14:12 |
| *** jkilpatr has joined #zuul | 14:24 | |
| *** dkranz has joined #zuul | 15:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix gerrit branch creation detection https://review.openstack.org/517363 | 16:07 |
| jeblair | that's a semi-urgent fix for an issue we're seeing in production -- it could allow folks to land untested zuul configuration changes (which would then break zuul) | 16:11 |
| * SpamapS now combing through all of his zuul.yaml's adding run:'s | 16:31 | |
| jeblair | SpamapS: do you have a lot? i have a script that could probably be adapted in about 20 minutes. | 16:38 |
| pabelanger | jeblair: +3, left comment about maybe pep8 failure | 16:38 |
| SpamapS | jeblair: no, 15 jobs right now. Inheritance reduced that to maybe 8 | 16:38 |
| pabelanger | but zuul hasn't run anything yet | 16:38 |
| SpamapS | it was mostly just that zuul stopped doing anything useful while I fell down that well ;) | 16:39 |
| jeblair | SpamapS: probably easier by hand then. i'll get the script pushed up eventually anyway -- it just needs to go through one more iteration before it's not a mess | 16:39 |
| SpamapS | Now setting up to run my hoist CI on PR's to my local zuul fork and will stop just pulling in master from git.openstack.org. | 16:39 |
| SpamapS | s,master,feature/zuulv3, | 16:40 |
| SpamapS | that was lazy anyway | 16:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix gerrit branch creation detection https://review.openstack.org/517363 | 16:40 |
| SpamapS | Should always have been testing things. | 16:40 |
| jeblair | pabelanger: ^ thanks :) | 16:40 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Unpause a declined request https://review.openstack.org/517068 | 16:42 |
| Shrews | jeblair: change lgtm | 16:44 |
| SpamapS | ok I really need to figure out why my zuul is ignoring comment: false | 16:46 |
| * SpamapS fires up pdb | 16:46 | |
| SpamapS | starting to get spammy with all the success comment emails :-P | 16:46 |
| Shrews | jeblair: fwiw, 517068 is something we should get in soon, too. i'm not sure what kind of havoc that could cause, but I have seen it in production as recently as yesterday | 16:47 |
| jeblair | Shrews: havoc?! i thought it was only chaos and agony! you should have said something sooner! :) | 16:48 |
| Shrews | jeblair: it has escalated | 16:48 |
| Shrews | plus, i discovered a thesaurus since yesterday! :) | 16:49 |
| jeblair | Shrews: lgtm. nice test. :) | 16:50 |
| SpamapS | cry havoc! | 16:53 |
| *** bhavik has joined #zuul | 16:58 | |
| *** bhavik has quit IRC | 16:59 | |
| *** hashar is now known as hasharAway | 17:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: normalize daemon process handling https://review.openstack.org/517381 | 17:25 |
| *** electrofelix has quit IRC | 17:44 | |
| mnaser | i think i found a zuul bug? | 17:48 |
| mnaser | my gertty wasnt up to date and i did a recheck on a few merged checks | 17:48 |
| mnaser | and they're sitting in the check queue right now | 17:48 |
| mnaser | shouldnt that recheck be ignored? | 17:48 |
| SpamapS | mnaser: yes | 17:51 |
| mnaser | SpamapS: ok cool, example of a merged change in check queue right now - 514985 | 17:51 |
| SpamapS | mnaser: actually I think it might be a misconfigured pipeline. | 17:53 |
| SpamapS | mnaser: --> #openstack-infra | 17:53 |
| mnaser | ok ill follow | 17:53 |
| *** sambetts is now known as sambetts|afk | 17:59 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix gerrit branch creation detection https://review.openstack.org/517363 | 18:00 |
| jlk | SpamapS: I'm super duper interested in your rpdb findings. | 18:07 |
| SpamapS | jlk: oh! so I found that I'm a moron. ;) | 18:10 |
| SpamapS | jlk: I had an uncommitted PR and I wasn't looking at project-config master. ;) | 18:10 |
| jlk | ohhh? | 18:10 |
| jlk | that's... funny | 18:11 |
| SpamapS | Saul Goodman now | 18:11 |
| jlk | however, I think you've found that we don't have adequate debug logging in that area. | 18:11 |
| SpamapS | jlk: yeah, sorry for the brain draining ;) | 18:11 |
| SpamapS | jlk: well.... I mean... I didn't set comment.. and it wasn't logging comment.. because it was falling through to later defaults. | 18:11 |
| SpamapS | so I think we're OK actually | 18:12 |
| SpamapS | I broke you | 18:12 |
| SpamapS | by saying "here's the config" | 18:12 |
| SpamapS | and me, by believing "here's the config" | 18:12 |
| SpamapS | but in fact, that was not hte config zuul saqw. | 18:12 |
| SpamapS | now... on to my current problem | 18:12 |
| SpamapS | Unable to freeze job graph: Pre-review pipeline check does not allow post-review job test-ephemeral-service-account | 18:12 |
| SpamapS | I have a job whose parent uses secrets. | 18:13 |
| jlk | SpamapS: sure, but zuul wasn't adequately displaying what config it saw, and the debug lines weren't informative enough to validate whether it was seeing the correct config | 18:13 |
| SpamapS | and I'm not sure how to make the check queue run that job. | 18:13 |
| jlk | ah | 18:13 |
| jlk | so it doesn't by default allow secrets to be used in pre-review pipelines (check) | 18:13 |
| jlk | You've got to mark the secret as usable in pre-review | 18:14 |
| SpamapS | http://paste.openstack.org/show/625378/ | 18:14 |
| jlk | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#secret | 18:15 |
| jlk | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-pipeline.post-review | 18:16 |
| jlk | looks like you have to define the pipeline itself as post-review | 18:16 |
| jlk | or rather | 18:16 |
| jlk | set hte job.post-review to false | 18:16 |
| SpamapS | I'm wondering if there's a danger there | 18:17 |
| mnaser | SpamapS: depends on your environment | 18:19 |
| mnaser | but it means i can make a child pre-review job that does "cat $secret" | 18:19 |
| SpamapS | so that sounds wrong | 18:19 |
| mnaser | and see what the value of the secret is | 18:19 |
| SpamapS | since the first paragraph claims that playbooks which override this job's playbooks don't get access to the secret | 18:20 |
| SpamapS | "Secrets are bound to the playbooks associated with the specific job definition where they were declared." | 18:20 |
| SpamapS | mnaser: I think the reason for defaulting to now allowing this is not that, but the danger that somebody will be able to publish things in a pre-check. | 18:20 |
| SpamapS | So if you've made a job that uses a secret to provide a resource for any job, you can assert that with post-review: false | 18:21 |
| SpamapS | which, in my case, is OK I believe | 18:21 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Unpause a declined request https://review.openstack.org/517068 | 18:21 |
| mnaser | SpamapS: yeah you're right | 18:21 |
| SpamapS | because I'm using the secret in the pre to make a resource, and handing it to the job | 18:22 |
| SpamapS | and then in the post, I use the secret to delete the resource I made | 18:22 |
| jlk | is the playbook that uses the secret in a trusted repo? | 18:23 |
| jlk | is there any way somebody could propose change to that playbook, which would in turn cause it to use the speculative future with the secret before a human can read it? | 18:24 |
| SpamapS | jlk: Oh yeah that's the reason the review flag is getting flipped. I need to move the job to my project-config repo. | 18:25 |
| jlk | or, does the use of the secret cost you any real money? Somebody could exhaust your funds by opening up a ton of pull requests that would cause the secret to be used. | 18:25 |
| SpamapS | actually I was going to use a semaphore to prevent it from creating too many resources. :) | 18:26 |
| jlk | those are good for "too many at once", but I don't think we have anything in place for "too many total" | 18:26 |
| SpamapS | Once set, the post-review attribute may not be unset | 18:27 |
| SpamapS | Zuul isn't letting me do the wrong thing. Good job Zuul. ;) | 18:27 |
| SpamapS | jlk: the post deletes the resource | 18:27 |
| SpamapS | The trouble I'm having here is that I kind of want to test it in check before I lock it down in project-config. :-/ | 18:28 |
| SpamapS | so I have to like, land the code in project-config... then recheck the usage in godaddy-zuul-jobs ... pretty frustrating. I'd like to be able to turn off the safeties while I develop it. | 18:28 |
| jlk | SpamapS: I think I'm still talking past you. I was thinking of the Ansible case, where each job that uses a windows VM costs them a dollar or something like that. | 18:28 |
| jlk | so they'd want both a quota (not too many at once) and a budget (not too many over a set period of time) | 18:29 |
| SpamapS | ah yeah | 18:29 |
| SpamapS | that's a different thing | 18:29 |
| jlk | "may not be unset" ? maybe not live, but certainly with a restart it can be unset :D | 18:29 |
| SpamapS | I can do bazillions of these. | 18:29 |
| SpamapS | I was hoping I could set post-review: false on the job where I declare the secret. | 18:30 |
| SpamapS | but both are in an untrusted repo | 18:30 |
| SpamapS | so I think I may have to just do the painful thing of landing the code in project-config and rechecking in the untrusted repo. | 18:31 |
| jlk | It's a valid use case, perhaps we should discuss it more with jeblair, and then document it somewhere. | 18:33 |
| SpamapS | It's the old "I really really do want to break this. Just for now. Please let me." | 18:34 |
| SpamapS | but it may be possible | 18:34 |
| SpamapS | just not obvious to my overloaded brain | 18:34 |
| jlk | right, we might be making an invalid assumption somwhere | 18:34 |
| tobiash | SpamapS: responded on 517067 | 18:53 |
| SpamapS | tobiash: thanks! I wasn't sure I understood. Sounds like something worth doing. | 18:54 |
| tobiash | :) | 18:54 |
| SpamapS | hrm | 19:08 |
| SpamapS | seems like things are working... weirdly with the secrets job as my parent | 19:08 |
| SpamapS | http://paste.openstack.org/show/625382/ | 19:09 |
| SpamapS | the parent pre/post seem to have run one after the other, with no RUN | 19:09 |
| SpamapS | n/m... another PBKAC | 19:21 |
| tobiash | jlk: responded on 517121 | 19:27 |
| tobiash | s/responded/commented/ | 19:28 |
| jeblair | SpamapS: yeah, if you think it's safe enough, you could set post-review on check. just try to remember to remove it before it causes an incident that makes the evening news. :) | 19:28 |
| SpamapS | 2017-11-02 19:27:39.300980 | testecm | the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: {{ zuul.secrets.ecm_auth.effort }}: 'dict object' has no attribute 'secrets' | 19:28 |
| jeblair | SpamapS: drop '.secrets'; they just show up as a variable under their name (which can be overridden) | 19:29 |
| SpamapS | OOOOOOOH | 19:29 |
| SpamapS | derpaderp | 19:29 |
| jeblair | SpamapS: also, i documented our procedure for changing our base job; it may be helpful: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n7 | 19:29 |
| SpamapS | jeblair: I went ahead and just landed the job in project-config, and I'm using it in an untrusted repo to iterate. | 19:30 |
| jeblair | SpamapS: cool | 19:30 |
| SpamapS | yeah looks like I got halfway to your solution | 19:30 |
| SpamapS | I like it | 19:30 |
| SpamapS | jeblair: +1 for carefulness :) | 19:31 |
| jeblair | (that's several steps, but once you get going, the "land fix to test job and recheck" iteration cycle can be pretty quick) | 19:31 |
| SpamapS | jeblair: and actually, we have a test auth domain for exactly this | 19:31 |
| jlk | oh cute, seems i Just ran into a bug in the githubconnection driver... | 19:32 |
| SpamapS | so I can have a job that uses the test auth domain without secrets in the check pipeline. | 19:32 |
| jlk | holy cow this is a pretty basic bug :9 | 19:33 |
| SpamapS | they all are once you find them | 19:34 |
| SpamapS | it's the finding that isn't always basic :) | 19:34 |
| jlk | also looks like the github docs are wrong | 19:35 |
| jlk | https://developer.github.com/v3/activity/events/types/#pushevent claims a 'pusher' object is in the payload, but it's not | 19:35 |
| jlk | also claims that the 'ref' key has a full git ref (refs/heads/whatever) but it doesn't. It seems to just be the branch name. | 19:36 |
| Shrews | jeblair: gah, I missed something on that declined pause change. We need to reset the request to REQUESTED instead of leaving it PENDING so another launcher will try to pick it up. | 19:36 |
| jlk | oh derp, I'm looking at the wrong event. DAMN | 19:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Reset state on unpaused, declined request https://review.openstack.org/517417 | 19:44 |
| Shrews | jeblair: ^^^ that's actually not urgent since our cleanup worker handles it nicely for us | 19:44 |
| Shrews | once again, the cleanup thread sweeps my mistakes under the rug :) | 19:45 |
| SpamapS | speaking of github bugs | 19:56 |
| SpamapS | http://paste.openstack.org/show/625387/ | 19:57 |
| jeblair | Shrews: lgtm | 20:04 |
| SpamapS | This github delay thing is killing me | 20:05 |
| SpamapS | 10s is ok.. sometimes | 20:05 |
| SpamapS | but some things take 40+s | 20:05 |
| SpamapS | jlk: I wonder if GraphQL would be more up to date :-P | 20:06 |
| SpamapS | hrm, debugging some problems.. | 20:24 |
| SpamapS | secrets seem like they get cleaned up even in keep mode? | 20:24 |
| pabelanger | don't we write them to tmpfs in bwarp, so they don't actually endup on disk | 20:28 |
| SpamapS | ah clever | 20:29 |
| SpamapS | I'm getting auth denials from the end point I"m pointed at, so I'm pretty sure my secret content is somehow wrong | 20:30 |
| SpamapS | struggling to figure out how to debug that | 20:30 |
| SpamapS | I guess I can land a secret that isn't important, debug print it and see what it is, and then adjust accordingly. | 20:31 |
| pabelanger | yah, had a few issues properly testing secrets, most of it was land, test, iterate. | 20:32 |
| pabelanger | biggest one was using binary data in ansible varibles, took a little bit to figure out that wasn't supported | 20:33 |
| SpamapS | hrm, I wonder if I need to trim it | 20:39 |
| SpamapS | pabelanger: I remember the binary fight ;) | 20:39 |
| SpamapS | I think I may have put the \n on the end of a password | 20:40 |
| *** kmalloc has quit IRC | 20:40 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Switch to threading model of socketserver https://review.openstack.org/517437 | 21:00 |
| *** dkranz has quit IRC | 21:12 | |
| *** hasharAway is now known as hashar | 21:13 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a generic stage-output role https://review.openstack.org/509233 | 21:49 |
| jlk | SpamapS: I wonder if we're suffering from cache invalidation problems. | 21:55 |
| SpamapS | jlk: yeah I have a todo to trace requests usage of the Etag controller to see if it is actually being used | 21:56 |
| *** hashar has quit IRC | 22:00 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Prime github app install map on connection load https://review.openstack.org/517121 | 22:14 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!