| *** smyers has quit IRC | 00:01 | |
| *** smyers has joined #zuul | 00:01 | |
| *** olaph1 has joined #zuul | 00:38 | |
| *** olaph has quit IRC | 00:38 | |
| *** olaph1 is now known as olaf | 00:45 | |
| *** olaf is now known as olaph | 00:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create zuul_executor_dest variable for fetch-sphinx-output https://review.openstack.org/498624 | 00:57 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix errors in mirro-workspace-git-repos https://review.openstack.org/498997 | 00:58 |
| *** timrc has quit IRC | 01:12 | |
| *** timrc has joined #zuul | 01:13 | |
| *** yolanda has quit IRC | 05:21 | |
| *** hashar has joined #zuul | 06:23 | |
| *** yolanda has joined #zuul | 06:57 | |
| *** jamielennox has quit IRC | 07:09 | |
| *** jamielennox has joined #zuul | 07:14 | |
| *** MaciekW has joined #zuul | 07:40 | |
| MaciekW | Hi, I have problem with zuul 2.5.2 - zuul is not triggering new jobs, on zuul's status page Queue lengths: reached over 600 events. I don't see any usefull information in any of the logs. Do you have any idea what could be wrong? | 07:44 |
| MaciekW | server where zuul is launch is not overloaded - load is less than 1 and there is huge amount of free memory | 07:44 |
| MaciekW | we have already restarted zuul few times but after a while problem happens again | 07:45 |
| MaciekW | could you suggest me how we can investigate the root cause of the problem? | 07:47 |
| MaciekW | Can anyone can help me ? | 08:05 |
| *** electrofelix has joined #zuul | 09:23 | |
| *** hashar has quit IRC | 09:39 | |
| *** hashar has joined #zuul | 09:39 | |
| *** hashar has quit IRC | 09:42 | |
| *** hashar has joined #zuul | 09:44 | |
| *** hashar has quit IRC | 10:07 | |
| *** hashar has joined #zuul | 10:07 | |
| *** hashar_ has joined #zuul | 10:13 | |
| *** hashar has quit IRC | 10:13 | |
| *** jkilpatr has quit IRC | 10:36 | |
| *** jkilpatr has joined #zuul | 11:09 | |
| *** hashar_ is now known as hashar | 11:36 | |
| *** hashar has quit IRC | 12:07 | |
| *** olaph has quit IRC | 12:07 | |
| *** Shrews has quit IRC | 12:07 | |
| *** lennyb has quit IRC | 12:07 | |
| *** jeblair has quit IRC | 12:07 | |
| *** jeblair_ has joined #zuul | 12:07 | |
| *** hashar has joined #zuul | 12:20 | |
| *** olaph has joined #zuul | 12:22 | |
| *** Shrews has joined #zuul | 12:22 | |
| *** lennyb has joined #zuul | 12:22 | |
| mordred | MaciekW: hi! I'm not the world's best at diagnosing such things - but it seems like perhaps you don't have build nodes registered with zuul | 12:51 |
| mordred | pabelanger: ^^ thoughts? | 12:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add comment to top of zuul.yaml pointing to other files https://review.openstack.org/499147 | 13:16 |
| *** dkranz has joined #zuul | 13:19 | |
| Shrews | on line 62 of https://review.openstack.org/#/c/498623/5/zuul.yaml , can you set dependencies there, or do they need to be under the job definition itself? | 13:27 |
| Shrews | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.dependencies implies it's part of the job def | 13:28 |
| Shrews | oops, mordred: ^^^^ | 13:30 |
| mordred | Shrews: they can be in either place | 13:31 |
| mordred | Shrews: so in this case I did not put them on the job def themselves because announce-release could also be (potentially) be used after a javascript or go release | 13:31 |
| mordred | Shrews: (although Ido not think the scripts it uses actually support that today) | 13:31 |
| Shrews | hrm. that should be in the documentation somewhere (unless it already is and i missed it) | 13:32 |
| mordred | Shrews: agree. it may be in the job variant section - or it may be implied in the job variant section and need to be spelled out more explicitly | 13:32 |
| Shrews | ah, it is. "Each item of this list may be a string, in which case it is treated as a job name, or it may be a dictionary, in which case it is treated as a job variant local to this project and pipeline." | 13:33 |
| mordred | Shrews: (essentially what this is doing is creating a pipeline variant of that job that adds the dependency) | 13:33 |
| mordred | yes | 13:33 |
| Shrews | lots of hidden little gems | 13:34 |
| mordred | yah - it all hangs together pretty nicely as you're writing stuff, but it's also a *very* rich system so communicating up-front all the things you can do it going to be a really fun challenge for us | 13:34 |
| pabelanger | mordred: MaciekW: possible there are no ready nodes online? | 13:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add comment to top of zuul.yaml pointing to other files https://review.openstack.org/499147 | 14:05 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix errors in mirro-workspace-git-repos https://review.openstack.org/498997 | 14:08 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create zuul_executor_dest variable for fetch-sphinx-output https://review.openstack.org/498624 | 14:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Revert "Add comment to top of zuul.yaml pointing to other files" https://review.openstack.org/499165 | 14:13 |
| jeblair_ | mordred: let's discuss that further please ^ :) | 14:16 |
| *** jeblair_ is now known as jeblair | 14:17 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a role to remove an ssh private key https://review.openstack.org/498530 | 14:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Delete .pypirc file at end of task https://review.openstack.org/498843 | 14:29 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move .pypirc into tmpfs https://review.openstack.org/498844 | 14:30 |
| *** olaph1 has joined #zuul | 14:31 | |
| *** olaph has quit IRC | 14:32 | |
| mordred | jeblair: I'm cool with that - maybe we leave the first two lines of the top comment then remove the ENTIRE comment that's already there about tox-docs not being the OpenStack docs job? | 14:36 |
| jeblair | mordred: that works for me. | 14:38 |
| mordred | jeblair: I'll push that up real quick | 14:38 |
| jeblair | kk | 14:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Partially Revert "Add comment to top of zuul.yaml pointing to other files" https://review.openstack.org/499165 | 14:39 |
| Shrews | oh, well ^^^ has an interesting -2 from zuul | 14:50 |
| jeblair | Shrews: probably this bug: https://review.openstack.org/498872 | 14:50 |
| jeblair | looks like it got lost in a restart, ironically :) | 14:51 |
| jeblair | i'll see if i can confirm that with logs and if so, restart | 14:51 |
| jeblair | yep -- there was a github api ratelimit error during reconfiguration | 14:52 |
| jeblair | the other question is -- were we running with the patches that should have prevented that error | 14:53 |
| mordred | jeblair: 2.5.3.dev1117 is what my zuul status page shows | 14:54 |
| mordred | but that may not be accurate, as locally my repo shows 2.5.4.dev191 | 14:55 |
| jeblair | mordred: how'd you get that local version number? | 14:55 |
| mordred | jeblair: python setup.py --version | 14:55 |
| fungi | is there a do you have a local 2.5.3 tag maybe? | 14:56 |
| mordred | oh - maybe I do - I think I was testing something | 14:56 |
| jeblair | 2.5.3.dev1117 is my local setup.py output | 14:56 |
| fungi | looks like 2.5.2 is the latest on origin anyway | 14:56 |
| mordred | jeblair: yes - I now get that | 14:57 |
| jeblair | mordred: hrm -- why would the running version be so new? | 14:58 |
| jeblair | oh, last restart seems to have been at 12:46 | 15:00 |
| mordred | jeblair: I restarted this morning to pick up main.yaml changes - although I totally should have just done a service reload instead | 15:00 |
| jeblair | mordred: np, it's convenient in this case anyway :) [but a reload should work] | 15:00 |
| jeblair | so we do know we're running the github fixes | 15:00 |
| mordred | also really need to get puppet to trigger the reload on file changes patch done | 15:01 |
| mordred | jeblair: the gh fixes and also your fix to not bail on reconfig issue too, right? | 15:01 |
| jeblair | mordred: no that hasn't landed yet | 15:01 |
| jeblair | only the gh fixes have | 15:01 |
| mordred | ah | 15:02 |
| mordred | jeblair: https://review.openstack.org/#/c/498872/ it's got 2x+2 - shall we push it in? | 15:03 |
| fungi | i don't suppose we're using a common review topic for the zuul v3 job migration work... i know it's mostly unique to a pretty limited set of repos plus a set of files in project-config... does anyone have a gerrit query handy they've been using for all that maybe? | 15:03 |
| jeblair | fungi: 'zuulv3' | 15:03 |
| fungi | ahh, okay so some patches are using that just not all | 15:03 |
| mordred | yah - trying to remember - sometimes forget | 15:04 |
| jeblair | fungi: yeah, it's most important on project-config | 15:04 |
| fungi | k, i'll put together some string of project names plus that topic | 15:05 |
| jeblair | so something like (project:openstack-infra/openstack-zuul-jobs OR project:openstack-infra/openstack-zuul-roles OR project:openstack-infra/zuul-jos OR (project:openstack-infra/project-config AND topic:zuulv3)) | 15:05 |
| jeblair | so something like (project:openstack-infra/openstack-zuul-jobs OR project:openstack-infra/openstack-zuul-roles OR project:openstack-infra/zuul-jobs OR (project:openstack-infra/project-config AND topic:zuulv3)) | 15:05 |
| jeblair | (typo) | 15:05 |
| clarkb | jeblair: dmsimard: ianw left a good comment and suggestion on the fix disk revert revert change. Would be good if you could take a look at that (and so we can hopefully just get that behind us) | 15:05 |
| jeblair | clarkb: cool, will look in a few mins | 15:05 |
| jeblair | mordred, jlk: here's a recent api rate limit bust: http://paste.openstack.org/show/619907/ | 15:06 |
| dmsimard | clarkb: yes, I talked with ianw last night | 15:06 |
| dmsimard | clarkb: Basically udevadm settle --timeout 10 --exitsomething is equivalent to our while loop | 15:06 |
| jeblair | mordred, jlk: and here's the error that tanked the reconfig: http://paste.openstack.org/show/619908/ | 15:06 |
| clarkb | dmsimard: ya he was suggseting that we try it with timeout 0 though to confirm that is the problem (udev events queued and unprocessed) | 15:07 |
| fungi | thanks jeblair! | 15:07 |
| clarkb | and then maybe we switch to udevadm with a nice long comment explaining what is going on (just to make it easier to understand in the future) | 15:07 |
| dmsimard | clarkb: I'll get to it after the openstack/RDO release crunch :) | 15:08 |
| clarkb | dmsimard: sounds good, thanks | 15:08 |
| clarkb | also tripleo must've gotten the overlay thing sorted out? | 15:08 |
| pabelanger | Ya, I fixed it for them | 15:08 |
| mordred | pabelanger: woot | 15:10 |
| jeblair | mordred, jlk: looking at that first paste -- http://paste.openstack.org/show/619907/ -- do we need some more debug lines in there? we're getting api rate limit log lines, but nothing else -- does that mean we're performing api calls but not logging what they are? | 15:11 |
| jeblair | mordred, jlk: ie, does that mean we performed 3 api ops, 2 authenticated, one not? | 15:11 |
| jeblair | or was the third a search api which is lower? | 15:12 |
| *** weshay is now known as weshay_interview | 15:17 | |
| dmsimard | clarkb: yeah, pabelanger copied the entire function in their tree | 15:18 |
| dmsimard | ಠ_ಠ | 15:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only add changes to status page with jobs https://review.openstack.org/499191 | 15:18 |
| dmsimard | I guess that extinguished the fire | 15:18 |
| mordred | jeblair: how about that ^^? | 15:18 |
| mordred | jeblair: looking at paste | 15:18 |
| dmsimard | clarkb, pabelanger: I'll send a patch after the release crunch to add a reliable and non-voting multinode job (standard job, doesn't run in the tripleo cloud) to try and catch if there's any other shenanigans like that | 15:19 |
| mordred | jeblair: yah - I think a smidge more logging would be helpful - I was looking at the code yesterday wanting the debug logging to be able to tell us what org/user the connection is for | 15:19 |
| *** olaph has joined #zuul | 15:26 | |
| *** olaph1 has quit IRC | 15:27 | |
| *** MaciekW has quit IRC | 15:28 | |
| clarkb | reviewing a thing for ironic has poitned out to me how complicated ironic's jjb is. Their jobs may make a good tester of the conversion stuff | 15:30 |
| clarkb | mordred: ^ fyi | 15:30 |
| mordred | clarkb: ++ | 15:31 |
| mordred | clarkb, jeblair: also, once we think devstack-legacy is ready - we can do a manual migration of the nodepool dsvm job and add a nodepool-dsvm-legacy job since nodepool is already in v3 | 15:31 |
| SpamapS | Curious.. has anyone ever suggested IRC or Slack triggers for Zuul? | 15:35 |
| SpamapS | It would be rather useful for me to be able to go "hey zuulbot, deploy region4" and have that trigger a CD job. | 15:35 |
| SpamapS | and report, obviously, via IRC/Slack too. | 15:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Added more debug logging for github requests https://review.openstack.org/499207 | 15:41 |
| mordred | jeblair, jlk, SpamapS: ^^ | 15:41 |
| mordred | SpamapS: yah - I think the thinking was that once we add the fedmsg driver, we can use the tools already written for fedmsg to get triggers and reporters | 15:42 |
| mordred | SpamapS: without having to write triggers/reporters for each of the slack/irc/hipchat/jabber/gnome/etc. | 15:42 |
| SpamapS | mordred: yay middleware ;) | 15:45 |
| pabelanger | +1 for fedmsg | 15:46 |
| clarkb | mordred: SpamapS do easy buttons support fedmsg? | 15:47 |
| SpamapS | I do wonder if fedmsg can be trusted to pass along things like whether or not a user was authenticated though. | 15:47 |
| SpamapS | zomg I want an easy button that publishes to fedmsg now ;) | 15:47 |
| SpamapS | clarkb: is that a joke or is there a library called easybuttons ;-) | 15:47 |
| clarkb | SpamapS: a joke but I agree it should be a thing | 15:48 |
| SpamapS | Like that's the thing. I was pretty sure it was a joke.. but... | 15:48 |
| clarkb | "that was easy" and you are deployed | 15:48 |
| SpamapS | I would love to have a team of people who all have actual big red buttons on their desks. | 15:48 |
| SpamapS | well it doesn't say "that was easy" until the deploy succeeds ;) | 15:48 |
| pabelanger | ah, github rate limit again | 15:56 |
| *** weshay_interview is now known as weshay | 16:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Added more debug logging for github requests https://review.openstack.org/499207 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Try getting project branches authenticated https://review.openstack.org/499222 | 16:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Cache branches for a project and use it on rate limit errors https://review.openstack.org/499223 | 16:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Keep a cache of branches for projects and update on events https://review.openstack.org/499224 | 16:23 |
| mordred | jeblair, SpamapS, jlk: ^^ there are some more thoughts on followups we can do | 16:24 |
| mordred | jeblair: check me on my comments on the last of them - I'm pretty sure that wants to be an event, but I wanted to get the idea up to discuss it real quick before doing that | 16:24 |
| jeblair | mordred: i fixed your logging change so you'll need to rebase that stack | 16:24 |
| mordred | jeblair: okie - and thanks | 16:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Cache branches for a project and use it on rate limit errors https://review.openstack.org/499223 | 16:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Try getting project branches authenticated https://review.openstack.org/499222 | 16:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Keep a cache of branches for projects and update on events https://review.openstack.org/499224 | 16:25 |
| mordred | jeblair, jlk, SpamapS: I have verified that github does not emit an event when branch protection is added :( | 16:29 |
| jeblair | mordred: reviewed all 3 | 16:32 |
| jeblair | mordred: i also just realized a subtle aspect of this situation that makes it worse than i would normally expect... i will explain with a patch | 16:33 |
| mordred | jeblair: cool | 16:34 |
| mordred | SpamapS: y'all have chatted with the github folks before - what's the best way to do that? | 16:35 |
| jeblair | mordred: nevermind, i was wrong. :) | 16:39 |
| jeblair | mordred: what we might consider doing with your third change is to just use the cache (ie, no api calls) when we perform a tenant-reconfiguration (which is the thing we do when a config change lands). but use the api-call-with-cache-fallback behavior on full system-reconfiguration (ie, sighup) | 16:41 |
| jeblair | mordred: (i mean that as a followup to your 3rd change) | 16:41 |
| jeblair | that may strike the right balance of "as long as we're watching for branch creation events, the cache should usually be correct" but still give the system a way to resync if something goes awry. | 16:42 |
| jeblair | mordred: another problem in 499207; you want to fix it this time? | 16:44 |
| fungi | now that i have an awesome gertty dashboard for zuul v3 migration, i'm crawling through open changes oldest first and finding a bunch which should probably be abandoned. i've been setting them to wip with comments explaining why, in order to help focus reviewing | 16:45 |
| fungi | (just a heads up) | 16:45 |
| jeblair | fungi: thank you | 16:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Cache branches for a project and use it on rate limit errors https://review.openstack.org/499223 | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Try getting project branches authenticated https://review.openstack.org/499222 | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Added more debug logging for github requests https://review.openstack.org/499207 | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Keep a cache of branches for projects and update on events https://review.openstack.org/499224 | 16:48 |
| jeblair | mordred: nevermind i did it :) | 16:48 |
| mordred | jeblair: sorry - was verifying something for https://review.openstack.org/499222 | 16:53 |
| mordred | jeblair: I have verified that it is possible to get the branches list authenticated | 16:53 |
| mordred | jeblair: and I like your suggestion for subhup vs. config change | 16:54 |
| mordred | jeblair: EXCEPT - that we don't get events when protection is added to or removed from a branch | 16:54 |
| jeblair | mordred: yeah. i think 207 222 and 223 are good to go now. 224 should be harmless (though see my comment on that one). | 16:55 |
| jeblair | i'd like to at least get up through 223 in asap and then restart | 16:55 |
| jeblair | anyone else want to review some github debug changes + fixes? | 16:56 |
| mordred | jeblair: so for the "only use proteced branches" logic, there's no way to track it | 16:57 |
| mordred | jeblair: another option -looking at the github3.py code, is that these calls take an optional etag parameter | 16:57 |
| mordred | jeblair: so I think we can cache the etag we get back from the call, then supply that to the next call and hopefully that will cause things to work properly and not count as an extra api call? | 16:58 |
| mordred | jeblair: lemme do that patch real quick | 16:59 |
| pabelanger | +3 on github-ratelimit stack,minus 224 | 16:59 |
| *** hashar is now known as hasharAway | 17:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Added more debug logging for github requests https://review.openstack.org/499207 | 17:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Try getting project branches authenticated https://review.openstack.org/499222 | 17:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Cache branches for a project and use it on rate limit errors https://review.openstack.org/499223 | 17:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update config cache only after all cat jobs complete https://review.openstack.org/498872 | 17:18 |
| jeblair | i made a small fix to that change; i'm going to self-re-approve | 17:29 |
| pabelanger | wfm | 17:32 |
| mordred | jeblair: ok. SO - I have learned a couple of things - and then I have also realized I'm somewhat dumb | 17:38 |
| mordred | jeblair: things I've learned: | 17:38 |
| mordred | a) github3.py does not expose an etag parameter for github.repository() which does a GET on 'https://api.github.com/repos/{org}/{repo}' in order to get a repository object on which you can call .branches() | 17:39 |
| mordred | b) the github REST API DOES properly work with an etag on that call (yay for once again python wrapper libraries not being as good as the direct REST API itself) | 17:40 |
| jeblair | mordred: re (a) -- is it possible for github.repository *not* to make that GET call? i mean, we're pretty sure that repo is there :) | 17:42 |
| mordred | c) we're using cachecontrol already, which should be automatically reading and adding etag/if-none-match headers for us on all of these calls | 17:42 |
| mordred | jeblair: well, it is if you reach in and just use REST | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update config cache only after all cat jobs complete https://review.openstack.org/498872 | 17:43 |
| mordred | jeblair: http://paste.openstack.org/show/619925/ is my sample code for poking | 17:43 |
| mordred | jeblair: (there is stuff missing from that) | 17:43 |
| fungi | mordred: sorry to interrupt, but can you confirm whether 482650 is still needed or is a dead fork in the road for bindep role work? it's failing to get tested in such a way that i'm going to need to dig into local zuul integration testing or checking the debug logs on zuulv3.o.o to work out the problem | 17:48 |
| fungi | also hasn't been touched in a month, so i'm dubious it's wanted any longer | 17:48 |
| mordred | fungi: thanks- abandoned | 17:49 |
| fungi | appreciated! | 17:49 |
| mordred | jeblair: do we re-create Connection objects when we reconfigure? | 17:55 |
| SpamapS | mordred: jlk did most of the chatting with github, so he should be able to tell you. I did open a feedback ticket about dropping approved reviews after force push, so I can look again at how I did that. | 17:56 |
| jeblair | mordred: oh, i think so (on the full system reconfig (not the tenant reconfig when a change lands)), let me double check | 17:56 |
| *** electrofelix has quit IRC | 18:01 | |
| jeblair | mordred: nerp, i'm wrong. here's what happens: on startup only, we initialize drivers+connections (so a full restart is required to change a [connection] in zuul.conf). during any kind of reload (system or tenant), we call driver.reconfigure(tenant) for every driver. github doesn't implement that, so nothing will happen in those cases. | 18:01 |
| jeblair | mordred: so both the driver and connection objects are very persistent (which i assume is where you're going with that). | 18:02 |
| jeblair | mordred: it's probably worth treating the connection object as being as persistent as the connection in zuul.conf. ie, in the future, i think the only reason we would drop/recreate a connection object is in the case that something in zuul.conf changed. that probably still matches fairly well to what we need here. | 18:03 |
| mordred | jeblair: ok. cool | 18:04 |
| jlk | o/ | 18:04 |
| jlk | reading scrollback | 18:04 |
| mordred | jeblair: then I actually expect that we should be using cache via cachecontrol | 18:04 |
| mordred | jeblair: it's in-memory so will get zeroed on a full restart | 18:04 |
| mordred | jeblair: although a followup patch to add optional external persistent cache via file or redis would not be super hard | 18:05 |
| jeblair | i'm restarting zuulv3 now with all the github changes and my reload fix | 18:05 |
| mordred | cool | 18:05 |
| jlk | so we're still rate limiting busting. :( | 18:05 |
| mordred | morning jlk ! | 18:05 |
| mordred | jlk: well, we've added a few new toys :) | 18:05 |
| jlk | it's really unfortunate we never got BonnyCI sufficiently off the ground to catch these :( | 18:05 |
| mordred | jlk: well, the upside is that I have learned a whole bunch of things this morning, which is good | 18:06 |
| jlk | that means you looked closer at my code, I'm afraid! | 18:06 |
| mordred | :) | 18:06 |
| jeblair | jlk: aside from things still in review and discussion, you'll want to be aware of these already-merged changes: https://review.openstack.org/499207 https://review.openstack.org/499222 https://review.openstack.org/499223 | 18:07 |
| jeblair | jlk: (normally we would have waited for your feedback, but i thought it better to get these in production to put out fires and collect more data) | 18:07 |
| jlk | opening | 18:07 |
| jlk | yeah | 18:07 |
| jlk | I was having a slow morning, so I approve of that direction | 18:07 |
| mordred | jlk: the only rage I've had at all has been at github3.py | 18:08 |
| mordred | jlk, jeblair: we can also add cachecontrol debug logging into our logging config so we can see what it's caching or not | 18:08 |
| jlk | yeah... | 18:08 |
| jlk | that's been on the chopping block | 18:08 |
| jlk | I had wanted to toss it with the move to graphql, but there were some roadblocks to graphql usage. | 18:09 |
| jlk | That would be good, getting that cachecontrol | 18:09 |
| jlk | We used to get some logging of etag and cache hits/misses | 18:09 |
| jlk | and timeouts | 18:09 |
| SpamapS | jlk: did we ever report the deficiencies in graphql to GH? | 18:10 |
| SpamapS | IIRC, it was a lack of certainfields. | 18:10 |
| jlk | Yeah I opened a forum thread about it (which was the official way) | 18:10 |
| jlk | I have to loop back and see if there has been any change. THere haven't been any notification emails to me about replies | 18:11 |
| jlk | other than some "me too!"s | 18:11 |
| jlk | I'll make that a priority today. | 18:11 |
| SpamapS | Should I laugh that GH has a hard time managing bugs in their own software? ;-) | 18:11 |
| fungi | and considers web forum threads the best means of providing specs for api features | 18:12 |
| fungi | seems like we do a lot more things in git than they do | 18:13 |
| jlk | mordred: looking at https://review.openstack.org/#/c/499222 the app API does allow us to get a list of branches, see https://developer.github.com/v3/repos/branches/#list-branches there is a little "i" in a circle that mouse over indicates that it's available to apps. | 18:15 |
| mordred | jlk: wow. that's how I find thatout? neat -thanks, that was going to be one of my questions for you | 18:16 |
| mordred | jlk: my solution was to make a script that authenticated as an app and geta list of branches - I should remove that comment | 18:16 |
| jlk | Looking at https://review.openstack.org/#/c/499223 , I feel like just doing a cache work around for branch data is just kicking the can down the road for the next API call. But perhaps that's fine, we just keep moving that can from API to API? | 18:17 |
| mordred | jlk: agree | 18:17 |
| mordred | jlk: I'm actually a little more concerned about why this isn't being handled via cachecontrol for us already | 18:17 |
| jlk | yeah that's a good question. | 18:18 |
| mordred | jlk: and then yah - more broadly I think not kicking the can down the road is a better idea | 18:18 |
| mordred | jlk: although sad panda that we can't get events to tell us to update our cached branch protection information | 18:18 |
| jlk | I wonder if the cache control is per-github object | 18:19 |
| mordred | jlk: ALTHOUGH - if cachecontrol does its thing I would expect just querying for branches to use cache most of the time unless protection has changed | 18:19 |
| jlk | and we make new github objects all the time. | 18:19 |
| mordred | jlk: I just checked that | 18:19 |
| mordred | jlk: it's not - we have a global one | 18:19 |
| jlk | ah okay | 18:19 |
| mordred | jlk: and I verified that we don't recreate the Connection object it lives on | 18:20 |
| mordred | jlk: pointed you at it in infra channel, but I also just made a logging config change to turn on cachecontrol logging so we can see it | 18:20 |
| jlk | yup | 18:20 |
| jlk | We should do the same in default upstream if it's not there already | 18:21 |
| jlk | IIRC when I was messing with things yesterday, I thought it was getting protected branches on repeated calls within a short amount of time and it was finding the cached data and using it | 18:22 |
| jlk | but if enough time expires then it will consider the cache stale. Unless that particular API call has an etag that we can pass along? | 18:22 |
| mordred | good. so it might just be that even with that around a restart the unauthenticated api limit is small enough that we were hitting it | 18:23 |
| jlk | something about the etags, it lets the github API see if what you ahve (even if old) is unchanged. | 18:23 |
| jeblair | mordred: http://paste.openstack.org/show/619928/ | 18:23 |
| jlk | oh yeah, unauth is pretty low limits. If there are any other unauth calls we should kill 'em | 18:23 |
| mordred | jlk: cachecontrol is suppsed to be doing | 18:23 |
| mordred | gah | 18:23 |
| mordred | jeblair: onit | 18:23 |
| jlk | yeah it does | 18:23 |
| mordred | BAH typo | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Consume the variable with the same name it's set https://review.openstack.org/499251 | 18:25 |
| mordred | jeblair: ^^ | 18:25 |
| jeblair | mordred: approved and rechecked (i manually downgraded zuul so it would start) | 18:26 |
| jeblair | once that lands, we can upgrade and re-restart | 18:26 |
| mordred | jeblair: cool. wanna update the logging config while we're at it? | 18:26 |
| jeblair | mordred: yah, if puppet hasn't done it by the time we're ready (that's also approved) | 18:27 |
| mordred | cool | 18:27 |
| jeblair | mordred: i have to afk for lunch now; i can do that when i get back or feel free yourself | 18:28 |
| mordred | okie. I'll restart it once it lands | 18:29 |
| jlk | I found a couple more places we're not using the app for calls, where I think we could. | 18:33 |
| jlk | actually | 18:39 |
| jlk | it looks like the only places we do that is a call to search (for Depends-On) and I think we learned yesterday that search has it's own API limits, and a call to get user information (which we cache internally heavily anyway). | 18:40 |
| dmsimard | clarkb: is the sync and partprobe required if we're adding in the udevadm settle ? | 18:40 |
| dmsimard | I guess it doesn't hurt to keep them there, was just curious | 18:40 |
| mordred | jlk: yah - there's that whole User object | 18:41 |
| clarkb | dmsimard: ya I don't think it hurts but probably not necessary especially since they didn't fix the problem | 18:41 |
| mordred | dmsimard: I'd say not - since I think those were just attempts to ... | 18:41 |
| mordred | yah what clarkb said | 18:41 |
| dmsimard | ok, let me try and take those out. | 18:42 |
| mordred | clarkb, dmsimard: I almost asked this yesterday and then wanted to die inside a little - but would this be helped at all by udev rules and/or systemd units? (I try to stay away from both if I can help it, but I believe we had to do a udev rule for glean didn't we?) | 18:43 |
| clarkb | mordred: I don't think it would because we still ahve to manually partition things so making swap creation automagic and mounting /opt automagically after partitioning manually is probably more effort than it is worth | 18:43 |
| dmsimard | systemd has stuff for udev yeah | 18:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add cachecontrol logging to the default config https://review.openstack.org/499255 | 18:46 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Consume the variable with the same name it's set https://review.openstack.org/499251 | 18:47 |
| mordred | clarkb: nod | 18:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Log cachecontrol info by default https://review.openstack.org/499257 | 18:49 |
| mordred | clarkb: you know what would be neat? if those fancy block_device_mapping parameters were a) documented b) worked everywhere | 18:50 |
| mordred | clarkb: since I _think_ those would allow us to actually set up swap, etc beforehand - but last I checked they're still terrifying undocumented magic | 18:50 |
| mordred | clarkb: maybe next time we get, you know, bored with no work on our plates, we can go learn how they work, test them everywhere and document them | 18:51 |
| clarkb | that would be nice though it won't fix things for systems without an ephemeral drive (so really everything but rax now) | 18:52 |
| mordred | restarted v3 scheduler | 18:57 |
| jlk | oh hrm. Github apps flat out do not support GraphQL anyway | 19:03 |
| mordred | jlk: that's awesome | 19:05 |
| mordred | jlk: so - I see thingsin the log that make me agree that we are using cachecontrol (I was stupid and restarted before the cachecontrol logging patch landed, but oh well - I see two calls one after the other with the same ratelimit number) | 19:07 |
| mordred | jlk: 2017-08-30 18:59:41,138 DEBUG zuul.GithubConnection: Got reviews for PR ansible/ansible#28833 | 19:07 |
| mordred | 2017-08-30 18:59:41,189 DEBUG zuul.GithubConnection: GitHub API rate limit remaining: 29 reset: 1504122648 | 19:07 |
| mordred | jlk: looks like getting reviews is using the unauth'd connection | 19:08 |
| jlk | remaining 29. | 19:08 |
| jlk | that's pretty low | 19:08 |
| jlk | so, it might not be using unauthed, but it might be using the non-app way | 19:08 |
| mordred | jlk: nod | 19:09 |
| mordred | jlk: well- I don't see ANY log lines with the second form I added, so I'm thinking that did not work | 19:09 |
| jlk | ah indeed | 19:09 |
| jlk | "use_app=False" | 19:09 |
| mordred | 2017-08-30 19:06:56,564 DEBUG zuul.GithubConnection: GitHub API rate limit remaining: 11991 reset: 1504121255 | 19:09 |
| jlk | there are comments in there all about why | 19:09 |
| mordred | cool | 19:10 |
| mordred | jlk: another optimization I wonder if we could make - is to not do things with projects we're not configured to care about | 19:10 |
| mordred | I'm not sure a good way to get at that information from the webhook | 19:10 |
| jlk | yeah, well | 19:10 |
| mordred | but if project not in the list of projects in main.yaml - there's really no need to do anything further | 19:11 |
| jlk | the webhook should just be getting enough info to send it through to the driver | 19:11 |
| jlk | the driver then takes that info and examines it to see if it matches a pipeline | 19:11 |
| jlk | so it's not really at the webhook side, more at the driver side | 19:11 |
| mordred | yah - totally | 19:11 |
| jlk | oh hrm! | 19:12 |
| jlk | going to test a thing here | 19:12 |
| jlk | and then I'll put my brain meat in the same space you're thinking | 19:13 |
| mordred | fun | 19:14 |
| jlk | uh, is openstack-infra/zuul-jobs broken? | 19:15 |
| mordred | jlk: so, my brainmeat is thinking about the line above - "DEBUG zuul.GithubConnection: Got reviews for PR ansible/ansible#28833" - | 19:15 |
| jlk | extra keys not allowed @ data['release'] | 19:15 |
| mordred | jlk: that's what it looks like when we get a broken config - looking | 19:15 |
| mordred | jlk: is that a new comment? | 19:15 |
| mordred | or from earlier today? | 19:15 |
| jlk | f8e746cb14646cc86885f9f7a0a817d9da864ac3 | 19:16 |
| jlk | I can't see what's wrong with it, but that's where the complaint is from | 19:17 |
| mordred | that gets me to https://review.openstack.org/#/c/498903/ | 19:17 |
| mordred | jlk: I feel we may be talking about slightly diffrent things ... | 19:18 |
| jlk | I might be missing something. But it complains about the "post" chunk there | 19:19 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Use app integration to get PR reviews https://review.openstack.org/499267 | 19:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role for adding launchpadlib credentials https://review.openstack.org/498633 | 19:26 |
| mordred | jlk: yah - I'm just confused by what you mean by f8e746cb14646cc86885f9f7a0a817d9da864ac3 and think I might not be looking at the right thing | 19:27 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Partially Revert "Add comment to top of zuul.yaml pointing to other files" https://review.openstack.org/499165 | 19:28 |
| jlk | mordred: oh well let me explain how I got there. | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role for adding launchpadlib credentials https://review.openstack.org/498633 | 19:31 |
| jeblair | mordred: oh, we should not be adding project-templates to zuul-jobs | 19:31 |
| jeblair | mordred: that should be in openstack-zuul-jobs i think | 19:31 |
| jlk | mordred: I started my local zuul off of the topic branch. My tenant.yaml includes a reference to openstack-infra/zuul-jobs as untrusted-projects | 19:32 |
| jeblair | (it imposes pipeline requirements; and while installations could use include/exclude to filter those out, i think we decided that we should restrict that repo to jobs only to make it easier to use) | 19:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets https://review.openstack.org/484000 | 19:33 |
| jlk | at start up, when zuul tries to process config for that repo, it chokes on the line I mentioned. I git logged the zuul.yaml file in that repo, foudn the commit that touched that area recently. | 19:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Revert "Make a docs-on-readthedocs project-template" https://review.openstack.org/499272 | 19:34 |
| jeblair | jlk, mordred: ^ | 19:35 |
| Shrews | mordred: i don't think there is a 'shred' module | 19:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add roles for adding and removing launchpadlib credentials https://review.openstack.org/498633 | 19:36 |
| mordred | Shrews: gah. that should be command shred | 19:37 |
| mordred | jeblair: ah. cool. gotcha (and makes sense) | 19:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Remove project stanza https://review.openstack.org/499278 | 19:38 |
| pabelanger | https://review.openstack.org/498621 and https://review.openstack.org/498267 are ready for review if people need a break from github. That is jobs for afs-docs | 19:40 |
| jeblair | 19:40 < openstackgerrit> James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add docs-on-readthedocs project template https://review.openstack.org/499279 | 19:40 |
| jeblair | mordred: ^ there's the 2nd half of move | 19:40 |
| jeblair | pabelanger: what is afs.path? | 19:42 |
| pabelanger | /afs/.openstack.org/docs from our secret | 19:43 |
| jeblair | why is that part of the secret? | 19:43 |
| pabelanger | no reason, we did that for artifact publisher, so copypasta | 19:43 |
| pabelanger | site_logs for example | 19:44 |
| mordred | jeblair: +2 to all and +3 to some | 19:45 |
| Shrews | jeblair: if the zuul-jobs project stanza is also defined in project-config, how does that work? were they merged? because the definitions are vastly different | 19:46 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Make a docs-on-readthedocs project-template" https://review.openstack.org/499272 | 19:46 |
| mordred | jeblair, pabelanger: I'm growing complex and conflicting feelings about our current pattern of bundling secret and non-secret data | 19:46 |
| jeblair | Shrews: merged, in config loading order (so project-config is first) | 19:47 |
| pabelanger | mordred: jeblair: ya, I'm happy to change it to something else if people would like | 19:47 |
| mordred | jeblair, pabelanger: on the one hand, it feels nice and encapsulated to say "add-filserver: site_logs" and have site_logs have the 4 parameters needed | 19:47 |
| pabelanger | Right, that is my feeling also. We could do that with 2 variables too | 19:48 |
| mordred | jeblair, pabelanger: on the otherhand itmeans we wind up with potentially the same private key multiple times when we want to use it for multiple purposes, when saying "add-fileserver: zuul_ssh_key, site_logs" would work too | 19:48 |
| mordred | yah | 19:48 |
| kklimonda | is there some architecture docs about nodepool? I'm looking at creating windows images, and the best way (cloudbase based) requires a windows host. I'm reading through https://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html and it seems that the architecture would support custom builders, but I'm not sure where to look next. | 19:49 |
| mordred | jeblair, pabelanger: lemme propose a quick patch to something to see what y'all think (I don't want to bikeshed too much, but it's been bothering me a little bit recently) | 19:49 |
| mordred | kklimonda: hi! so - we don't have a good windows story yet - but when tobiash gets back from vacation he's planning on working on it | 19:50 |
| pabelanger | mordred: wfm | 19:50 |
| jlk | huh weird. | 19:50 |
| mordred | kklimonda: there are a couple of things that will be needed to get it done - but I don't think it's going to be a super long thing, just work that's needed | 19:50 |
| jeblair | mordred: i agree, i am conflicted. however, bundling of *some* amount of data is the primary driver of the approach we chose, so i think it's worth exploring. i think we haven't quite decided *what* one of these secrets is -- is it just access to a server, or is it the full package. i think it's worth pushing the envelope as far as we can go in that direction, then maybe dialing it back if it's awkward. | 19:51 |
| mordred | kklimonda: which is mostly to say - I highly recommend syncing with tobiash when he's back (I think next week?) - he's got a much better sense of what's needed on the nodepool/zuul front | 19:51 |
| jlk | I made myself a test app and my API limits went down to 57 or so.. | 19:52 |
| mordred | jeblair: ++ - quick sake-of-argument coming | 19:52 |
| kklimonda | mordred: sure, is there anything I can read nodepool related that would give me a better overview of how it's working before then? This task just jumped a bit in my team's internal priority queue, and I'd like to take some time to investigate building blocks, so that I can actually talk with tobiash when he's back and not make a fool out of myself ;) | 19:53 |
| mordred | jlk: it seems like the webhook + user with oauth credentials will let you do auth'd things for all interactions - whereas we have to do unauth'd for some things becaue the api api is incomplete | 19:53 |
| jlk | mordred: yes, however, things that ARE authed as the app so it should be getting app level limits seem to be getting far less | 19:54 |
| jlk | maybe a dev app gets a lot lower to start with? | 19:54 |
| mordred | kklimonda: https://docs.openstack.org/infra/nodepool/feature/zuulv3/ are the docs so far - leifmadsen is starting work on better getting started docs for nodepool | 19:55 |
| mordred | Shrews: ^^ any thoughts on other things to point kklimonda at to read on nodepool architecture? | 19:56 |
| Shrews | kklimonda: which version of nodepool? | 19:56 |
| mordred | Shrews: the interest is in windows support, and I was mentioning that tobiash was interested in workingon adding support for that when he got back | 19:56 |
| Shrews | kklimonda: are you using the version from the features/zuulv3 branch, or from master branch? | 19:56 |
| mordred | so I think we're talking v3 | 19:56 |
| mordred | s/I think// | 19:57 |
| mordred | we will _definitely_ not be adding windows support to v2 :) | 19:57 |
| jeblair | kklimonda: if you want to read changes, https://review.openstack.org/468624 and its parent https://review.openstack.org/488384 are architecturally relevant. also their git parents, already merged, continue the story. | 19:57 |
| *** jkilpatr has quit IRC | 19:58 | |
| Shrews | kklimonda: there is some stuff under https://git.openstack.org/cgit/openstack-infra/nodepool/tree/doc/source/devguide.rst | 19:58 |
| Shrews | but it's rather incomplete. i should probably update that | 19:58 |
| *** weshay is now known as weshay_PTO | 19:58 | |
| jeblair | kklimonda, Shrews: kklimonda pointed to the image build/upload spec, which is actually implemented in both branches. however, the changes i just pointed to and their parents start to cause the implementation to diverge significantly between the two; all the more reason to only focus on the feature/zuulv3 branch. | 19:59 |
| leifmadsen | mordred: speaking of which :) | 20:00 |
| leifmadsen | when do you wanna get together and move to phase 2? | 20:00 |
| kklimonda | Shrews: we are not running nodepool currently, I've actually been thinking that windows images could be our first step towards that (as we plan on using it at some point). We are currently running zuul from master branch, not v3. | 20:01 |
| clarkb | its possible you can just change the disk image builder invocation to the windows dib (assuming that works) run the whole lot on windows and be done | 20:04 |
| clarkb | biggest thing is probably that the zookeeper lib runs on windows | 20:04 |
| mordred | clarkb: according to tobiash we need to plumb through information about username/password for remote connection withpowershell to work | 20:05 |
| clarkb | mordred: is that how windows dib works? | 20:05 |
| jlk | oh OH | 20:06 |
| jlk | hrm. | 20:06 |
| jlk | may have found a big bug. | 20:06 |
| mordred | clarkb: I think so - like, I think he's using that by hand currently to create the user account, but the informatoin on how to use that user account needs to be communicated to zuul so that the inventory can be written out properly | 20:06 |
| mordred | clarkb: we also need to set a different connection-type in the inventory if it's windows too | 20:06 |
| clarkb | oh I'm ignoring zuul for this thought experiment | 20:06 |
| mordred | jlk: yay! | 20:06 |
| clarkb | purely just nodepool making an image and uploading it somewhere | 20:07 |
| mordred | clarkb: yah- same thing - you need the admin credentials to use the windows remote connect | 20:07 |
| clarkb | mordred: because windows doesn't have chroot so it must run a vm and connect to that? | 20:07 |
| mordred | well - sure ,you could do that if all you wanted to do was upload it | 20:07 |
| mordred | clarkb: we're kind of at the limit of what I learned from tobiash | 20:07 |
| clarkb | mordred: well ya one step at a time | 20:07 |
| jlk | bug being I forgot to rebuild my container, so something was lost. | 20:08 |
| clarkb | mordred: step 0 is make image, step 1 is upload to cloud. I think that should mostly just work if you swap out the dib invocation command | 20:08 |
| mordred | clarkb: but the amount of work to add it properly actually doesn't seem too bad | 20:08 |
| kklimonda | is windows dib available somewhere? the only reference I've found was an old and retired repo from stackforge | 20:08 |
| clarkb | ya if its mostly just "this is how you connect to windows" that should be straighforward | 20:08 |
| clarkb | kklimonda: ya thats what I am not sure about. We have never used it | 20:09 |
| jlk | or not | 20:09 |
| jlk | hr . | 20:09 |
| mordred | clarkb: that seemed to be it- that and being able to mark a diskimage in nodepool config as being windows and not linux | 20:09 |
| mordred | so that that fact can be communicated to wherever that's necessary | 20:09 |
| mordred | kklimonda: tobiash is doing some amount of this locally by hand already - so he'll have the most info | 20:09 |
| kklimonda | thanks, I'll just wait for him then :) | 20:10 |
| clarkb | mordred: and maybe rather than doing windows vs linux make it connection type. ssh, telnet, remote powershell | 20:10 |
| clarkb | looks like all of cloudbase's image building tools are snapshot based | 20:11 |
| clarkb | :/ | 20:11 |
| mordred | clarkb: yah - biggest open question is whether there are also windows-specific boot params | 20:11 |
| clarkb | mordred: you mean to nova api (or similar)? | 20:12 |
| mordred | clarkb: which is an unknown for now waiting for tobiash to get back - if there aren't, then I agree, it should be connection type. if there are, it might need to be connection_type + os family - with os family == windows implying connection type == powershell | 20:12 |
| mordred | or something | 20:12 |
| mordred | clarkb: yah | 20:12 |
| pabelanger | clarkb: windows dib, crazy times | 20:13 |
| clarkb | pabelanger: it existed for a time but unsure if it is still a thing | 20:13 |
| clarkb | used powershell instead of bash etc, but design of using elements was the same aiui | 20:13 |
| *** jkilpatr has joined #zuul | 20:14 | |
| jlk | ah, pm != pem | 20:14 |
| jlk | okay, got it to work with my app now. | 20:19 |
| jlk | can confirm that an app can get a list of the reviews on a PR | 20:19 |
| * jlk out for lunch | 20:20 | |
| tobiash | Yes, I'll be back on monday | 20:23 |
| tobiash | And I finally got travel approvement for ptg :) | 20:24 |
| tobiash | So will also be there | 20:24 |
| *** hasharAway has quit IRC | 20:45 | |
| *** dkranz has quit IRC | 20:45 | |
| pabelanger | mordred: did you push up your secret changes? or did we want to keep iterating on https://review.openstack.org/498621/ and child | 20:50 |
| clarkb | tobiash: nice, don't forget to register nowish as price goes up on friday I think | 20:55 |
| * clarkb looks | 20:55 | |
| clarkb | ya saturay morning ttx says price goes up | 20:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: WIP Split add-sshkey into two roles https://review.openstack.org/499338 | 20:58 |
| tobiash | clarkb: booked already everything :) | 20:58 |
| mordred | jeblair, pabelanger: sorry- https://review.openstack.org/499339 and https://review.openstack.org/499338 | 21:00 |
| mordred | tobiash: woot! | 21:01 |
| pabelanger | mordred: ya, 499339 is bascially how I see the 2 variable approach too | 21:02 |
| pabelanger | mordred: only think I can think of an issue, is if we change the secret and forget to update other vars, or the other way around. I mean, we know fast there is an issue and just do another commit to fix it | 21:03 |
| mordred | yah - I mostly just don't know if it's 'better' or not | 21:03 |
| mordred | pabelanger: I say we just move foward as is for now | 21:04 |
| pabelanger | ya, no real preference either way for me. So will defer to what other people like :) | 21:04 |
| pabelanger | mordred: danke | 21:05 |
| mordred | pabelanger: real quick - can you push in https://review.openstack.org/#/c/499277/ and https://review.openstack.org/#/c/499278/ and https://review.openstack.org/#/c/499279 ? | 21:07 |
| pabelanger | done, done and done | 21:08 |
| mordred | pabelanger: and done on yours too | 21:08 |
| mordred | jlk, pabelanger, SpamapS: also - seems like we shoudl land https://review.openstack.org/#/c/490216/ before we forget about it (it's had 2 +2s in the past already) | 21:11 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use app integration to get PR reviews https://review.openstack.org/499267 | 21:17 |
| * jlk back | 21:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Remove the default argument to secure in nodepoolcmd https://review.openstack.org/480310 | 21:33 |
| mordred | tobiash: when you're back and ready to look at windows, there's 2 patches up https://review.openstack.org/#/q/topic:username+status:open that may have done part of what you need already | 21:35 |
| jlk | mordred: so, thinking about ignoring repos we don't care about, what would have to happen is a repo we don't care about would be configured to throw webhooks at us, signed by our webhook key (which is supposed to be private). I could see that happen in some corner situations | 21:36 |
| mordred | tobiash: I don't know if they'll be directly useful, but I thought i'd point them out | 21:36 |
| jlk | (like a repo that once installed the app we stopped caring but they keep sending) | 21:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 21:36 |
| mordred | jlk: yah - or a repo that has installed the app but that we don't quite yet care abt | 21:36 |
| jlk | maybe another scenario is finding a Depends-On: of a change we're handling that exists in a repo we don't care about. I'll look at the code for it. | 21:36 |
| mordred | jlk: it's an edge case to be sure - so if it's completely convoluted, I don't think we should lose too much sleep over it | 21:37 |
| jlk | so we'd have to maybe scrape the tenant name out of the event URL, then see if the project exists in that tenant? | 21:40 |
| jlk | jeblair: there was something related to github driver yesterday you wanted me to look into, and we put it on pause because of a eureka moment. Do you recall what that was? | 21:42 |
| jeblair | jlk: oh i *think* it was just the branch lookup stuff, iirc | 21:44 |
| jlk | oh, getting the protected branches? | 21:45 |
| jeblair | jlk: no, getting the list of branches robustly and authenticated; so i think it's taken care of now | 21:46 |
| jlk | yeah | 21:47 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove comments questioning API capabilities https://review.openstack.org/499358 | 21:48 |
| mordred | jeblair: can project-templates have descriptions to? | 21:50 |
| jeblair | mordred: yep, they actually share an implementation with projects. | 21:50 |
| mordred | jeblair: neat | 21:50 |
| jeblair | mordred: i think we may have forgotten to add them with zuul-sphinx; easy enough to fix tho. | 21:50 |
| jlk | There's one last call to a limited API, and that's searching for things that might have a Depends-On of a particular change. | 21:52 |
| jlk | currently we're caching that info when we get an event for a PR (and we refresh it for any subsequent events, and I don't think that's cachecontrol able) | 21:53 |
| jlk | I'm working through to see if we can delay fetching that info until it's actually going into a pipeline | 21:53 |
| jlk | yeah, I think I can | 21:55 |
| jlk | maybe | 21:55 |
| jeblair | mordred: ohnoes zuul said i lied! | 21:56 |
| mordred | jeblair: ohnoes! | 21:56 |
| jeblair | mordred: i will add description to projects and project-templates now | 21:56 |
| mordred | jeblair: cool - it definitely seems useful for the templates | 21:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow description fields in projects and project templates https://review.openstack.org/499365 | 21:57 |
| jlk | mordred: jeblair: there is one change object attribute that we care about, that would trigger the search, and that's the needed_by_changes attribute. Would it make sense to turn that into a property (at least in the github version of the object), and when the property is accessed, that's when we do the expensive search? | 22:21 |
| jlk | 'cause it should only happen when we're processing for a dependent pipeline | 22:21 |
| jeblair | jlk: unfortunately, that shifts network activity into the main (scheduler) thread; right now it's not slowing anything down | 22:22 |
| jlk | ah, I see. | 22:23 |
| jeblair | jlk: however, there's some rework that we need to do in order to handle cross-source dependencies | 22:23 |
| jlk | yeah we gather it in the webhook thread, it's just that it costs us an API hit every time | 22:23 |
| jeblair | jlk: it's entirely possible that there's no way to do that without doing what you suggest | 22:24 |
| jlk | every time we do a "refresh" | 22:24 |
| jeblair | jlk: so maybe we can revisit it as part of that | 22:24 |
| jlk | but if we don't look at it when we do a refresh, our cache could become stale | 22:24 |
| jlk | basically this will become our hard limit on how many webhooks we can respond to in a minute (30). | 22:26 |
| jeblair | jlk: depends on comes from the PR description right? | 22:27 |
| jlk | although in my testing it's showing 60 instead of 30, and it DOES appear that the search API call is cached. | 22:27 |
| jeblair | oh wait, this is needed by | 22:27 |
| jlk | same deal | 22:28 |
| jlk | We're searching for PRs that would depend on the PR we're dealing with | 22:28 |
| jeblair | gotcha. | 22:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Move name from block to tasks in the block https://review.openstack.org/499375 | 22:30 |
| jlk | couple things helping, looks like 60 a minute rather than 30 (or else it's giving me a 2 minute limit remainder), search is doing some caching somewhere, because a repeated search a few minutes later is not costing me an API limit | 22:35 |
| jlk | which is weird, because the cache isn't coming from cachecontrol. | 22:36 |
| jlk | anyway with the code changes made in the last couple days, and the logging changes, I'd like to see some logs and see if we're running close to API limit issues having just the ansible repo on there. | 22:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add publish-openstack-python-docs to post pipeline https://review.openstack.org/498840 | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix error in mirror-workspace-git-repos https://review.openstack.org/499382 | 22:53 |
| jeblair | mordred, pabelanger: ^ | 22:53 |
| pabelanger | +2 | 22:54 |
| jeblair | mordred: when you have a minute, the console output on http://logs.openstack.org/99/497699/26/check/devstack-legacy-tempest-dsvm-neutron-full/29bfff5/job-output.txt.gz#_2017-08-30_22_39_11_159486 is lacking some info -- it doesn't include the error(s) within the loop, so it's very hard to tell what failed (only ara and the json have that) | 22:55 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move name from block to tasks in the block https://review.openstack.org/499375 | 23:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix checkout in mirror-workspace-git-repos https://review.openstack.org/499384 | 23:00 |
| jeblair | pabelanger, jlk: want to look at that one too ^? | 23:05 |
| pabelanger | jeblair: is there an etherpad outlining the roles? I haven't really been following. And curious about delegate_to: localhost now | 23:08 |
| jeblair | pabelanger: no etherpad. what's the question? | 23:09 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix error in mirror-workspace-git-repos https://review.openstack.org/499382 | 23:09 |
| pabelanger | jeblair: so, this is to use our local git repos in /opt/git? Or am I missunderstanding the role | 23:09 |
| jeblair | pabelanger: there are at least 2 roles involved -- one clones repos from /opt/git to /home/zuul (that's in project-config). this one pushes commits from the executor to the repos in /home/zuul. | 23:11 |
| pabelanger | Okay, perfect | 23:11 |
| pabelanger | +3 | 23:11 |
| pabelanger | :) | 23:11 |
| pabelanger | thank you | 23:11 |
| clarkb | note that git push can't create remote repos so you always need something to run first | 23:17 |
| jeblair | yep; the only reason those are 2 roles right now is i don't want to deal with getting our repo cache into golang style paths yet (which i think is required in order to generalize that role) | 23:18 |
| jeblair | we can tackle that after ptg | 23:19 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix checkout in mirror-workspace-git-repos https://review.openstack.org/499384 | 23:20 |
| mordred | jeblair: I shall track down the issue with the LOOP (although it makes me sad that we have that issue) | 23:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add publish-openstack-python-docs to post pipeline https://review.openstack.org/498840 | 23:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add roles for adding and removing launchpadlib credentials https://review.openstack.org/498633 | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!