| jlk | git options do not appear to be rate limited, not git:// or ssh://. Maybe https://? Not sure. | 00:00 |
|---|---|---|
| jamielennox | right, we've known caching will fix much of this, most of the information does seem available via events, it's just an issue of timing and storing the data | 00:01 |
| jlk | nod | 00:01 |
| jlk | so I assumed with a move to graphQL and with keeping a cache, and using eTags for github side calls, we'll fly under the limits radar. | 00:02 |
| clarkb | I think github rate limits git operations by erroring when it can't fulfill them :) | 00:02 |
| jlk | lol | 00:02 |
| mordred | jlk, jamielennox: also, if there's data we need from the webhook that is causing us to need to query for it, that might also be a thing to be like "hey, gh, we're trying to do X without being bad rate limit people maybe can has another field in webhook?" | 00:03 |
| mordred | I mean, that doesn't always work - but sometimes it does | 00:03 |
| jlk | I'm building up a graphQL query right now that would get all the data we ever fetch from the individual API calls, so instead of 14 various API calls, we'd have one graphQL call | 00:03 |
| mordred | neat | 00:03 |
| jamielennox | mordred: yea, the main place where that is obvious though is an intentional github design | 00:04 |
| jlk | mordred: we tried. The two gotchas are PR comments, and commit status. | 00:04 |
| jamielennox | so statuses are set by commit hash not by PR, because the same commit can end up in multiple PRs | 00:04 |
| jamielennox | jlk: so where are you doing that call from? where are we doing more than 2 or 3 requests sequentially? | 00:06 |
| jamielennox | (ignoring the stupid github3.py library fetching the PR objects each time) | 00:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Mutex repo updates and merge ops in executor https://review.openstack.org/470571 | 00:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix reporting on changes dequeued for deps https://review.openstack.org/471143 | 00:06 |
| jlk | we're not doing any more than that sequentially, was just counting all the different functions we log API rate limits from | 00:06 |
| jamielennox | ya, we should remove that logging as well, statsd at best | 00:07 |
| jlk | fetching user info, project branches, PullReq itself, Pull file names, Pull by sha, pull reviews, repo permissions, | 00:07 |
| jlk | oh crap, I think I counted things where we _set_ stuff | 00:08 |
| jlk | getCommitStatuses was missing from above list | 00:09 |
| jlk | so 8 separate API calls down to 1 | 00:09 |
| jamielennox | jlk: yea, but those calls are not made from the same sections of code. are you thinking one big cached request? | 00:10 |
| jlk | right | 00:10 |
| jlk | make one query when an event comes in, gather all that data | 00:11 |
| jlk | and have an _update() call for getChange which pulls the PR out of our cache, re-runs the query, and updates anything that's changed. | 00:11 |
| jlk | basically how gerrit does it | 00:11 |
| jamielennox | jlk: will be interesting to see anyway, not sure atm whether it will be an advantage to just fixing how we use the rest api | 00:14 |
| jlk | so things like getting a change, we get the PR, we get the files from the PR, we get the reviews, we get the status of the head ref. | 00:16 |
| jlk | just the way the API works, that's 4 API calls | 00:16 |
| jamielennox | yea, was thinking cache might help because we get an event for each review and each status thats set | 00:20 |
| jamielennox | but it seems common to basically use those as a trigger and ask github for a real state of the world rather than assume you're constructed state is correct | 00:21 |
| jlk | yeah, and we could have missed some of those events during a restart. or network glitch | 00:22 |
| jlk | anyway, I "solved" my earlier problem, just go with updatedAt. Now I'm on to figure out how to construct a list of changed files in a given PR | 00:22 |
| jlk | oh bother, there doesn't seem to be a way to GET the files of a PR in graphQL. wtaf. | 00:35 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Status branch protection checking for github https://review.openstack.org/471175 | 00:38 |
| *** jamielennox is now known as jamielennox|away | 01:17 | |
| *** jamielennox|away is now known as jamielennox | 01:34 | |
| *** dkranz has quit IRC | 01:37 | |
| jlk | Welp. Couple things missing from GraphQL, I can't get whether or not a user has push access to a repo (to judge the "level" of their review), and I can't get a listing of the files changed in a PR. | 01:37 |
| jlk | http://paste.openstack.org/show/611447/ | 01:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 01:44 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add github reporter status-url config option https://review.openstack.org/449794 | 02:01 |
| jeblair | w00t, the 'mutex merger' change series (471161 -- 470571) is green now | 02:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add github reporter status-url config option https://review.openstack.org/449794 | 03:50 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add github reporter status-url config option https://review.openstack.org/449794 | 03:54 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose ref to the executor and url formatter https://review.openstack.org/463457 | 03:56 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 03:56 |
| *** linuxlite3 has joined #zuul | 03:56 | |
| *** linuxlite3 has quit IRC | 03:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in gerrit+github test https://review.openstack.org/471161 | 04:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 04:13 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose ref to the executor and url formatter https://review.openstack.org/463457 | 04:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 04:53 |
| *** eventingmonkey has quit IRC | 06:47 | |
| *** eventingmonkey has joined #zuul | 06:52 | |
| *** hashar has joined #zuul | 07:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 07:29 |
| *** hashar_ has joined #zuul | 07:30 | |
| *** hashar has quit IRC | 07:33 | |
| *** hashar_ is now known as hashar | 07:51 | |
| *** jamielennox is now known as jamielennox|away | 10:10 | |
| *** jamielennox|away is now known as jamielennox | 10:16 | |
| *** jkilpatr has quit IRC | 10:41 | |
| *** jkilpatr has joined #zuul | 10:59 | |
| *** smyers has quit IRC | 12:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 12:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't report start on unmanaged projects https://review.openstack.org/455711 | 12:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix reporting on changes dequeued for deps https://review.openstack.org/471143 | 12:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Mutex repo updates and merge ops in executor https://review.openstack.org/470571 | 12:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 13:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add override-branch property to jobs https://review.openstack.org/467775 | 13:08 |
| *** isaacb has joined #zuul | 13:29 | |
| *** dkranz has joined #zuul | 13:47 | |
| *** smyers has joined #zuul | 14:02 | |
| *** dougbtv_ is now known as dougbtv | 14:50 | |
| *** isaacb has quit IRC | 15:01 | |
| Shrews | holy smokes, i just saw a real-life Transformer. the new recycling truck has a yellow arm that comes out, grabs the recycling can, dumps it and sets the can back. | 15:09 |
| Shrews | this is high-tech for Durham | 15:09 |
| gundalow | Like this http://imgur.com/gallery/YiwxQKy | 15:12 |
| *** hashar has quit IRC | 15:13 | |
| Shrews | haha, a little more elegant than that one :) | 15:15 |
| Shrews | exactly like this one: https://www.youtube.com/watch?v=mE0VDZzxTR0 | 15:16 |
| jeblair | 467776 failed to merge because of this unrelated error, which i haven't seen before: http://paste.openstack.org/show/611527/ | 15:37 |
| jeblair | that's the http server failing to shutdown for some reason | 15:38 |
| jeblair | oh interesting, this is unique too: 2017-06-06 14:18:29,120 paste.httpserver.ThreadPool INFO 1 workers didn't stop properly, and 0 zombies | 15:40 |
| jeblair | i think it just didn't wait long enough for the worker to die | 15:41 |
| jeblair | it gives each worker 0.5s to shutdown gracefully, then kills the thread | 15:41 |
| jeblair | it must have taken more than 0.5s to stop, but still stopped before it got around to killing the thread | 15:41 |
| jeblair | and there's no exception handling inside of paste for that case | 15:42 |
| jeblair | i'd be tempted to just try shutting down again, but the whole process starts with a socket.close(), and that's not going to work twice | 15:44 |
| jeblair | could try calling self.thread_pool.shutdown(60) again... | 15:44 |
| mordred | jeblair: in https://review.openstack.org/#/c/467634/ I'm calling release but I'm left with jobs at the end - do I need to waitUntilSettled again after the release? | 15:51 |
| jeblair | mordred: yes | 15:51 |
| jeblair | mordred: the release will let them continue, they'll run asynchronously | 15:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 15:52 |
| jeblair | mordred: most tests end with a check that all the jobs have finished | 15:52 |
| mordred | ok. it's going to work this time :) | 15:52 |
| jeblair | i just rechecked my stack; i'm going to try to figure out how to do a hg pull request on bitbucket in the mean time | 15:52 |
| jeblair | (i barely knew before, and now they changed *everything*) | 15:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 15:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 15:53 |
| mordred | jeblair: wow, what do you need to hg for? | 15:53 |
| jeblair | mordred: i'm trying to fix the bug in paste above | 15:54 |
| mordred | jeblair: oh - so - there is a proposed openstack goal for "migrate off of paste" ... the rationale is that apparently paste is essentially a dead project | 15:55 |
| jeblair | wow, yep, that looks like it's the case. | 15:56 |
| jeblair | nice of them to mention that in the readme | 15:56 |
| *** hashar has joined #zuul | 15:57 | |
| SpamapS | so much backscroll | 15:58 |
| SpamapS | Shrews: welcome to 2005 trash collection technology! | 15:58 |
| Shrews | [WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback.zuul_stream.CallbackModule object at 0x7f768f2c2710>): 'ansible_host' | 16:00 |
| Shrews | so that's fun | 16:00 |
| * Shrews hunts down his error | 16:01 | |
| mordred | jeblair: so - maybe I really should start that thread about tristanC 's dashboard patch - because maybe it's time to have an answer to what we want out of our rest layer | 16:01 |
| jeblair | mordred: yeah, jamielennox has suggestions as well. let's pick a destination. | 16:02 |
| mordred | jeblair: ++ | 16:02 |
| jeblair | (we don't urgently need to arrive there, but i think we'll need to start the journey soon) | 16:02 |
| * jeblair is still clicking around bitbucket | 16:02 | |
| mordred | jeblair: I'm betting that picking the answer will be the hard part - we don't have a ton of http endpoints already, so I bet a migration to $whatever will be pretty simple | 16:03 |
| * jeblair prepares a d2 roll | 16:04 | |
| jeblair | https://bitbucket.org/ianb/paste/pull-requests/38/fix-error-on-httpserver-shutdown/diff | 16:04 |
| jeblair | there, i have sent that off on its long journey into the void | 16:05 |
| mordred | jeblair: well done | 16:05 |
| pabelanger | mordred: is there also an effort to move away from pecan too ? | 16:06 |
| SpamapS | Can we just wsgi, and maybe add a handy uwsgi integration for those who don't want more than that? | 16:07 |
| SpamapS | is pecan an http server? | 16:08 |
| * SpamapS has never looked at it | 16:08 | |
| pabelanger | ya, I think it started in openstack then left the project, but looks pretty dead now | 16:08 |
| SpamapS | I thought it was just a REST building framework, but that you still need a thing to serve | 16:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Return resulting commits from merger https://review.openstack.org/467776 | 16:08 |
| pabelanger | well, not may commit over the last 6 months | 16:09 |
| pabelanger | I think wsgi / uwsgi would be great. One less daemon to run | 16:10 |
| pabelanger | well, once less zuul daemon | 16:10 |
| pabelanger | one* | 16:11 |
| SpamapS | So the problem with that would be that we'd need to RPC between the scheduler and wsgi | 16:11 |
| jeblair | pabelanger: right, which means one more external dependency and configuration interaction | 16:11 |
| SpamapS | Right now the scheduler answers the web calls right? | 16:11 |
| jeblair | pabelanger: we don't just get to wash our hands of it. :) we're responsible for making sure users can actually get the whole thing up and running, one way or another. | 16:11 |
| SpamapS | uwsgi, I think, has an in-process server | 16:11 |
| pabelanger | jeblair: agree. something still needs to be configured | 16:12 |
| SpamapS | https://uwsgi-docs.readthedocs.io/en/latest/Embed.html | 16:12 |
| SpamapS | oh hrm, that's the opposite I think | 16:12 |
| jeblair | SpamapS: yeah, rpcing shouldn't be too hard though. we can throw in a gearman call for event injection. on the log streaming side, we're already moving toward a gearman function to look up some data. | 16:12 |
| SpamapS | jeblair: oh good, that might be easier than I'd thought. | 16:13 |
| SpamapS | Also, I just want to use the project because they are clearly wordsmiths; from their docs: "Now, we are ready to kick asses with uWSGI ninja awesomeness." | 16:13 |
| jeblair | mordred: don't forget the third leg of this -- the log streaming asyncio/websocket stuff | 16:14 |
| mordred | jeblair: yup | 16:14 |
| SpamapS | does asyncio have a powerful enough server implementation? | 16:14 |
| *** hashar has quit IRC | 16:16 | |
| mordred | jeblair, SpamapS, pabelanger: I think there are at least 2 questions: what, if anything, do we use for REST layer, and what do we use for http server - then as jeblair mentions, there's also the log streaming which may or may not want to be directly integrated | 16:16 |
| mordred | SpamapS: http://aiohttp.readthedocs.io/en/stable/ is the asyncio-based http framework thing | 16:19 |
| SpamapS | http://aiohttp.readthedocs.io/en/stable/web.html#aiohttp-web | 16:19 |
| SpamapS | looks pretty simple | 16:19 |
| jeblair | mordred: i think that's a fair summary. to expand a bit: our rest layer (* it's not really rest) is super simple, so we should not get too bogged down in that. and ideally, we would not ask users to run two different webservers, so being able to integrate log streaming with status page, webhooks, and dashboard would be ideal. | 16:20 |
| mordred | jeblair: agree | 16:22 |
| jeblair | SpamapS: regarding the git ssh wrapper stuff... i suspect there are problems | 16:24 |
| jeblair | SpamapS: ca5d0cac39b026bc23b74903478addd4f702173e is the most recent related change | 16:25 |
| jeblair | SpamapS: da90a50b794f18f74de0e2c7ec3210abf79dda24 is the prior one | 16:25 |
| *** jamielennox is now known as jamielennox|away | 16:26 | |
| jeblair | SpamapS: in short, i think the idea is to make sure we have the appropriate git command environment (with the right ssh key) set whenever we're running git commands on the merger which can use git+ssh | 16:26 |
| mordred | SpamapS: it does. otoh, interfacing it with any sort of wsgi-based api programming gets a little weird. HOWEVER - there is this: https://github.com/etianen/aiohttp-wsgi#design | 16:26 |
| jeblair | SpamapS: the v2 way was to write out little wrapper files, then set an env variable to tell git to use them. | 16:27 |
| mordred | SpamapS: which might allow us to have both options: "run with internal asyncio-based wsgi server" and a "run wsgi in uwsgi" | 16:27 |
| jeblair | SpamapS: jhesketh has moved us to a context manager in v3 | 16:27 |
| SpamapS | mordred: my question is why bother with wsgi based programming for this case if we can just build a REST API in the asyncio webserver directly? | 16:28 |
| SpamapS | jeblair: AHHHHH | 16:28 |
| jeblair | SpamapS: the test (test_timer_sshkey) was written because there was a path in the v2 code which would not actually invoke the ssh wrapper | 16:29 |
| jeblair | SpamapS: because of the way that the ssh wrapper was semi-persistent, i think that we actually papered over a bunch of ways that it could have otherwise been unset | 16:30 |
| jeblair | SpamapS: i suspect that now, in the v3 code with the context manager, there are *many* instances when we're running without the right environment | 16:30 |
| SpamapS | so with a context manager in place, it may actually be harder to find it? | 16:30 |
| SpamapS | Oh and it's maybe not even being used where it needs to be? | 16:31 |
| jeblair | SpamapS: in particular, if you look at the merge function (_mergeItem), the context manager is used in the second half, but not the first half. the first half performs a 'git remote update' which seems like it should need the ssh key info. | 16:31 |
| jeblair | SpamapS: i'm wondering if actually we should look into moving the context manager idea into the Repo() class (which is our wrapper around git.repo), so we can be more certain all the right commands have a context manager set up. | 16:32 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 16:32 |
| SpamapS | jeblair: makes sense to me. | 16:33 |
| jeblair | SpamapS: either that, or adopt a pattern of establishing a context manager for the duration of the high level functions in Merger (like, for the entirety of _mergeItem) | 16:33 |
| jeblair | SpamapS: i suspect we're not seeing this problem because on zuulv3-dev, the zuul user's ssh private key is what we use with gerrit, so we aren't actually exercising this. it's only an issue if the connection's ssh key is not in ~/.ssh/id_rsa. | 16:34 |
| jeblair | SpamapS: as for the test itself, i think once we establish the context manager the way we want, we shouldn't have to worry as much about "does the key get set up for this particular kind of job"; it should just be handled by the merger/repo api. it might still be valuable to have a test of whether that's working, however, the mechanism would be completely different than the ssh wrapper file check that's there now. | 16:36 |
| jeblair | SpamapS: so in short, we probably just need to drop that test, and then see if there's some alternative way we can verify the context manager. | 16:37 |
| SpamapS | jeblair: I think if we build the context manager into Repo, a simple test of whether or not Repo calls it is enough. | 16:38 |
| jeblair | mordred, clarkb: i don't understand this failure: http://logs.openstack.org/77/467777/2/check/gate-zuul-python35/0dff23e/ | 16:50 |
| jeblair | i think we've seen it a few times on the python3 tests | 16:50 |
| jeblair | there's no testrepository file | 16:50 |
| mordred | jeblair: I do not understand that either | 16:51 |
| jeblair | based on the console log (which has a whole bunch of stuff spewing out? it's not supposed to do that), it stopped doing things after 7 minutes, and then sat there for the next 38 minutes. | 16:51 |
| jeblair | what's the tmpgh4exbjp file? | 16:52 |
| clarkb | jeblair: should be the temp subunit file that get renamed when things die gracefully. We capture it when its not so graceful | 16:53 |
| jeblair | the perms seem wrong on that :( | 16:54 |
| jeblair | -rw------- 1 jenkins jenkins 85704 Jun 6 15:59 tmpgh4exbjp | 16:54 |
| jeblair | i just chmodded it | 16:55 |
| jeblair | that's not giving me any more insight | 16:58 |
| jeblair | i don't see any indication of hung tests | 16:59 |
| jeblair | it looks like all the tests passed: 265 successful tests and 18 skips. that matches the next successful py35 run. | 17:01 |
| jeblair | so why didn't testr finish? | 17:02 |
| mordred | jeblair: maybe something with a hung subprocess and testr wasn't able to properly shutdown? | 17:03 |
| mordred | jeblair: like if something had an orphaned/borked filehandle or similar? | 17:04 |
| *** jamielennox|away is now known as jamielennox | 17:05 | |
| jeblair | mordred: hrm.... maybe -- so testr runs (in this case 4) subprocesses for the tests. within each of those subprocesses, we have a hard timeout for each test. that did not trigger, so we know that each test completed. but you're thinking the subprocess itself didn't exit, even though it finished running all of its tests... | 17:06 |
| jeblair | mordred: i wonder if all 4 subprocess would have to hang like that for us to see this, or if just one would be sufficient.... | 17:06 |
| mordred | jeblair: dunno - and that'sa TOTAL stab in the dark too | 17:07 |
| jeblair | mordred: ya. so far it fits the (limited) evidence :) | 17:07 |
| jlk | o/ | 17:09 |
| jeblair | i'm going to try to catch this in action | 17:09 |
| jlk | SpamapS: I think I left you a land mine. The build of z8s that produced an image I had commented out almost all the dib elements so that it was incredibly simple (and likely non-functional). | 17:09 |
| clarkb | jeblair: I think one is sufficient as testr waits for all to exit | 17:10 |
| jlk | SpamapS: in /root/etc/nodepool/nodepool.yaml | 17:10 |
| jeblair | clarkb, mordred: so i think the debug procedure for this is: when it's hung, get a process listing, open file listing, and possibly gdb stacktraces | 17:10 |
| * jeblair writes a status json scraper | 17:11 | |
| SpamapS | jeblair: +1 that's what I was going to suggest actually | 17:13 |
| SpamapS | jlk: ACK, I got side tracked playing with kompose | 17:13 |
| SpamapS | and Helm | 17:13 |
| mordred | SpamapS: ZOMG. I finally got https://review.openstack.org/#/c/467634/ right | 17:20 |
| mordred | SpamapS: or, rathre, I finally got the tests right | 17:20 |
| mordred | SpamapS: so - can haz review? also, the one before it (https://review.openstack.org/#/c/467611) you had -1'd for tests, but the tests are in that patch and now pass | 17:20 |
| SpamapS | Yeah I'll hit that soon | 17:22 |
| SpamapS | hey we are talking about shutting down BonnyCI's v2.5 and one thing I realized is we need to translate the layout | 17:22 |
| SpamapS | pabelanger: did you already start on a translator? | 17:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 17:22 |
| SpamapS | I was thinking we could start one. | 17:22 |
| mordred | SpamapS: I do not believe we have started on a layout translator, no | 17:23 |
| mordred | SpamapS: I have some very old patches where Istarted thinking about standalone jjb translator -but those are likely bong at this point | 17:23 |
| jeblair | w00t i think i have a node with the problem | 17:24 |
| pabelanger | SpamapS: no translator yet | 17:24 |
| jeblair | https://etherpad.openstack.org/p/LXWKvYpOGt | 17:24 |
| SpamapS | jenkins 11410 0.0 0.8 3155492 67640 ? S 16:48 0:00 python -m subunit.run discover -t ./ ./tests/unit --load-list /tmp/tmp.7mpWDK7dfm/tmp73pe3n80 | 17:26 |
| SpamapS | that one looks stuck | 17:26 |
| jeblair | apt-get install lsof :) | 17:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use the executor cached repos more often https://review.openstack.org/467777 | 17:29 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Check out the appropriate branch in executor https://review.openstack.org/467778 | 17:30 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_one_branch cloner test https://review.openstack.org/467779 | 17:30 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Release executor jobs on test exit https://review.openstack.org/468553 | 17:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add LoggerAdapter for executor jobs https://review.openstack.org/468554 | 17:31 |
| SpamapS | python 11410 jenkins 28u unix 0xffff880234a2cc00 0t0 129510 /tmp/tmp.7mpWDK7dfm/tmp1aaa0lim/zuul-test/lib/executor.socket type=STREAM | 17:31 |
| SpamapS | jeblair: RPC socket still open? | 17:31 |
| SpamapS | seems like that should not be | 17:32 |
| jeblair | agreed; also there's an epoll, so geard may still be running | 17:32 |
| SpamapS | so yeah, somewhere a test didn't properly assert end state. I wonder if we have tests that don't extend ZuulTestCase but do fire some of that up. | 17:33 |
| jeblair | two open socket's actually | 17:35 |
| jlk | I believe https://review.openstack.org/464252 is ready for review, closes out a TODO | 17:35 |
| jeblair | SpamapS: unfortunately, i didn't get the nice python backtrace for that thread | 17:36 |
| jeblair | but we seem to be stuck in recv | 17:36 |
| jeblair | strace says recvfrom 8 | 17:37 |
| jeblair | python 11410 jenkins 8u IPv4 163332 0t0 TCP localhost:7900->localhost:40776 (ESTABLISHED) | 17:37 |
| jeblair | so we're reading from the tcp connection | 17:37 |
| jeblair | i don't know what that connection is | 17:39 |
| jeblair | neither of those port numbers show up in the console log | 17:40 |
| jlk | I'm picking up task 4664, making config repo updates work on github. | 17:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update the merger recent dict when saving the repo state https://review.openstack.org/469595 | 17:40 |
| SpamapS | jeblair: lsof should show both sides | 17:41 |
| jeblair | SpamapS: it does; it's the same process | 17:41 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Actually check out the branch in the jobdir https://review.openstack.org/468556 | 17:42 |
| jeblair | so it's probably the fake gearman server, or the fake zookeeper | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_multi_branch cloner/executor test https://review.openstack.org/468557 | 17:42 |
| SpamapS | jeblair: I'd guess gearman, because I feel like the ZK flaps and polls a lot | 17:43 |
| mordred | jesusaur: in 449794 - does the status get set to the status url while the job is running then to the log url when it's done? or is this only about log url when done? | 17:43 |
| jeblair | SpamapS: yeah, actually, the fake gearman would be the only thing with both connections on the same process | 17:44 |
| SpamapS | oh right, zk is real zk | 17:45 |
| SpamapS | jeblair: so perhaps it's stuck in getJob? | 17:45 |
| jeblair | SpamapS: yeah, though could also be the server | 17:47 |
| jlk | Coming up with a good verb to reflect that content in a repo/branch we care about has changed. Originally it was keyed off of "change-merged" event in gerrit, and in github it'll be off of a "push" event. I'm thinking "branch_updated" as a boolean. Any objections? | 17:48 |
| jeblair | jlk: none here | 17:49 |
| jeblair | SpamapS: i could really use a real python bt right now, but gdb isn't cooperating | 17:49 |
| SpamapS | jeblair: :( | 17:50 |
| mordred | jesusaur: nevermind. the previous patch in the stack answers my question | 17:53 |
| jlk | jeblair: unless I'm blind, I'm not seeing a test case that appears to test the functionality of a change being merged that has zuul.yaml content, leading to a reconfiguration. | 17:58 |
| mordred | jeblair: +2 on the stack up to 470442 - so pending getting the hung-tests thing figured, I think it's all good to go | 17:59 |
| jeblair | jlk: i think there's some coverage in test_v3.py in test_dynamic_config. but only for a .zuul.yaml change landing (it ensures that the dynamic config becomes the static config after reconfiguration after it lands). that should generally cover the same code, but a new test that explicitly tests what you describe would be a good addition i think. | 18:01 |
| jeblair | mordred: thx! | 18:01 |
| jeblair | i'm trying to understand how the python gdb stuff has changed with gdb7 and python3 | 18:01 |
| * mordred hands jeblair hip wading boots | 18:02 | |
| jeblair | https://docs.python.org/devguide/gdb.html | 18:02 |
| mordred | jeblair: oh - that's pretty neat - all you need now is that python-gdb.py file | 18:05 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle job or build is None in formatUrlPattern https://review.openstack.org/470415 | 18:07 |
| jlk | jeblair: what should happen if a (.)zuul.yaml file is removed? Should that cause a reconfigure too? | 18:08 |
| jeblair | jlk: erm, yes. i'm pretty sure that's untested. :) | 18:09 |
| jlk | lol! | 18:09 |
| jeblair | jlk: (who would want to do such a thing?!) | 18:09 |
| jlk | well, ¯\_(ツ)_/¯ | 18:09 |
| jlk | I'm just adding code to the github driver to expose the files associated with a push | 18:09 |
| jlk | and it breaks them down by added/modified/removed. I figured we should probably do _something_ on a file removed | 18:10 |
| jeblair | ++ | 18:10 |
| clarkb | jeblair: for that tcp connection both ends are process 11410? | 18:13 |
| jeblair | clarkb: yep | 18:13 |
| jeblair | finally! | 18:14 |
| jeblair | i have a python backtrace | 18:14 |
| jeblair | 'apt-get install python3.5-examples' and then the rest is in the etherpad | 18:14 |
| jeblair | and we were wrong, it's not gear; it's the log streamer test | 18:15 |
| jeblair | i think that's enough info to work from now | 18:16 |
| jeblair | i'm going to try to get lunch before the infra meeting | 18:16 |
| jlk | Python style help. I'm using a list comprehension to fill a set, so I'm calling f.add(blah) as the "action" in the comprehension. So this line will return something like [None, None, None...] that I just don't need. I was going to just assign that to "_" and never use it, but pep doesn't like that. Should I just let it "return" that value unassigned? | 18:45 |
| *** mordred has quit IRC | 18:47 | |
| jlk | mordred: re multiple ingestors, I was thinking on that topic the other day too, and I was going to "solve" it in Bonny with a caching load balancer with a single backend. So that if the backend went away, the LB would cache until it showed up again, or eventually timeout. | 18:47 |
| clarkb | jlk: can you paste the comprehension | 18:47 |
| jlk | [[files.add(f) for f in c.get('added') + c.get('modified') + | 18:48 |
| jlk | c.get('removed')] for c in event.get('commits').values()] | 18:48 |
| clarkb | jlk: you may just need something like [ f.add(x) for x in foo if x | 18:48 |
| *** mordred has joined #zuul | 18:48 | |
| jlk | I figured a set was more efficient at adding unique items than doing the if in a list comprehension. | 18:48 |
| clarkb | jlk: as an alternative f.add([x for x in foo if x]) | 18:48 |
| jlk | oh hrm, yeah you can add a scalar? | 18:48 |
| clarkb | oh not add() but there is a method iirc | 18:49 |
| jlk | hrm, no. | 18:49 |
| jlk | not with add() | 18:49 |
| jlk | foo.update() | 18:49 |
| jlk | hrm, maybe not | 18:52 |
| jlk | oh I see why. Because it's a double comprehension I'm getting a list of lists return | 18:53 |
| jlk | doing the f.add() inside the inner comprehension allows it to just flow, doesn't matter that the return is a list of lists. | 18:54 |
| clarkb | I think you can collapste them. [files.add(c.get()+c.get+c.get()) for c in vent.get(commits).values()] <- roughly | 18:55 |
| jlk | that's still calling add() inside the comprehension, so you'll get an unused "return" | 18:56 |
| clarkb | ya in this case just let it be c.get()+...+c.get() then do the files.update(comprehension here) | 18:57 |
| clarkb | just pointing out that you don't need two lists I don't think | 18:57 |
| jlk | oooh, maybe I'll map() this | 18:57 |
| clarkb | (dependso n how you want to represent the data structure ultimately) | 18:57 |
| jlk | or I'll just move this back to real loops and not try to be cute. | 18:58 |
| *** jkilpatr has quit IRC | 19:03 | |
| jlk | oh map() also returns none | 19:04 |
| *** hashar has joined #zuul | 19:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 19:25 |
| *** harlowja has quit IRC | 19:29 | |
| *** harlowja has joined #zuul | 19:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 19:57 |
| SpamapS | jeblair: so, for the streamer recv code.. it looks like its non-blocking.... what gives? | 20:03 |
| jeblair | SpamapS: is it? it's whatever a streamingserver does; i'm not familiar with it, and just started looking at it | 20:05 |
| jeblair | er, socketserver rather | 20:05 |
| Shrews | streamer issues? | 20:06 |
| SpamapS | Shrews: https://etherpad.openstack.org/p/LXWKvYpOGt | 20:07 |
| SpamapS | see the bottom | 20:07 |
| SpamapS | a test process is stuck in that thread | 20:07 |
| SpamapS | File "/usr/lib/python3.5/socketserver.py", line 234, in serve_forever | 20:08 |
| SpamapS | self._handle_request_noblock() | 20:08 |
| Shrews | neat | 20:08 |
| jeblair | i can reproduce by starting a LogStreamer, connecting to it without sending anything, then stopping | 20:10 |
| Shrews | if that's the serve_forever() thread, then i would expect non-blocking behavior (so we have the opportunity to stop the server) | 20:11 |
| jeblair | note that in that case the *foreground* process exits, but since we're using a forking server, the background process handling the request remains | 20:11 |
| jeblair | Shrews: no, that's the request handling subprocess | 20:12 |
| jeblair | the serve_forever thread has completed | 20:12 |
| SpamapS | I can't really see what is blocking | 20:13 |
| jeblair | i think there are two potential things of interest: a) that request handling subprocess is blocking, and b) that one of those sticking around can cause testr to hang | 20:13 |
| jeblair | SpamapS: recv() is a blocking call | 20:13 |
| mordred | jeblair: I believe we can set a sockettimeout on that socket before we make the recv call | 20:13 |
| SpamapS | jeblair: wait, are these stacktraces bottom->down? | 20:14 |
| mordred | jeblair: because we're already i na handler so, the socketserver should have already done the select bits meaning we should be expecting the changeid in pretty short order | 20:14 |
| * SpamapS was reading it as if the last thing was the place we were at | 20:14 | |
| Shrews | jeblair: ah, well the one log streamer test does just that: connects w/o sending anything then disconnects, but i've never seen it hang | 20:17 |
| SpamapS | Shrews: likely a race | 20:18 |
| jeblair | SpamapS: ah yeah, innermost at top | 20:18 |
| jeblair | mordred: i agree | 20:19 |
| jeblair | is a socket recv timeout the right approach, or should we make a select loop? | 20:19 |
| mordred | jeblair: the socketserver already has a select loop | 20:20 |
| SpamapS | I thought we were asyncio'ing and had an event loop? | 20:20 |
| SpamapS | or this is just the log streamer, not the websocket | 20:20 |
| mordred | which is what is causing this handler to be run - the issue is that something connects then never sends data | 20:20 |
| mordred | SpamapS: yah | 20:20 |
| jeblair | mordred: i mean should we put a select loop in the handler method | 20:21 |
| jeblair | (it's technically the right thing to do because there's no guarantee that our request is going to arrive in one packet) | 20:21 |
| SpamapS | yeah, 2 lines before the spot that called into our request handler in socketserver.py is where the select was done on that socket | 20:21 |
| mordred | jeblair: I'm good with select loop in the handler | 20:22 |
| jeblair | how long shall we give our clients to send their request? 10s? | 20:23 |
| Shrews | i think that's more than reasonable | 20:24 |
| Shrews | so ++ | 20:25 |
| jeblair | cool, i'll hack this up real quick like | 20:25 |
| *** dkranz has quit IRC | 20:32 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Extend in-repo config update support to github [wip] https://review.openstack.org/471477 | 20:32 |
| jlk | jeblair: ^^ that still needs the new tests we talked about, but passes existing tests. I'm going to play with the code with some real events stuffed in to see if it works right. A quick once-over would be appreciated. | 20:33 |
| *** jamielennox has quit IRC | 20:34 | |
| mordred | jlk: neat | 20:36 |
| * jlk pedals off in search of calories. | 20:36 | |
| *** jkilpatr has joined #zuul | 20:38 | |
| *** jamielennox has joined #zuul | 20:38 | |
| mordred | woot! https://review.openstack.org/#/c/467635/ stack is green | 20:40 |
| mordred | jlk: when you get back, I'd love your eyes on that ^^ - it changes how we're ansibling a bit | 20:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use a select loop on log streamer socket reads https://review.openstack.org/471480 | 20:45 |
| jeblair | mordred, Shrews, SpamapS: ^ hopefully sufficiently paraniod | 20:45 |
| mordred | jeblair: it double decodes | 20:46 |
| mordred | jeblair: but otherwise looks great | 20:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use a select loop on log streamer socket reads https://review.openstack.org/471480 | 20:48 |
| jeblair | mordred: derp, thx | 20:48 |
| mordred | jeblair: awesome. way to use gdb! | 20:49 |
| mordred | jeblair: maybe once we land that the rest of your stack can land | 20:49 |
| jeblair | mordred: i'm very happy to have cracked the py3 gdb mystery :) | 20:59 |
| jeblair | mordred: +2 on your stack | 21:04 |
| Shrews | jeblair: lgtm | 21:10 |
| jeblair | jlk: generally looks good; i left a note about a potential gerrit improvement inline | 21:12 |
| jeblair | clarkb: are you interested in looking at changes 470049--470393 ? they are the zuul-cloner tests morphing into executor git repo prep tests. | 21:17 |
| jeblair | (ignore the verified -1 or -2; they all pass tests) | 21:18 |
| clarkb | jeblair: I can, just getting back to computer post lunch now. Will take a look shortly | 21:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use a select loop on log streamer socket reads https://review.openstack.org/471480 | 21:20 |
| jlk | mordred: opened. | 21:24 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Replace equals None with is None https://review.openstack.org/471075 | 21:29 |
| clarkb | jeblair: I'm looking at https://review.openstack.org/#/c/470385/3/tests/unit/test_cloner.py and states is a list with one item in it, and self.builds should have a single build in it for integration. Why use a for loop there? | 21:34 |
| clarkb | the outer for loop specifically | 21:35 |
| jeblair | clarkb: easy copypasta between tests. we can probably refactor that into a helper method now. | 21:35 |
| jeblair | that entire for loop is now identical in all tests | 21:36 |
| clarkb | gotcha | 21:36 |
| clarkb | I also left a comment in there about the debug logging not sure if that was intentional to have or just test debugging | 21:36 |
| jeblair | oh that's from the copy pasta too. it's marginally useful. | 21:37 |
| jeblair | (really, there's probably one or two things in there that are useful, and it would be better to just log it/them) | 21:37 |
| jeblair | nah, it's not really useful. i'll drop it. | 21:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Factor out duplicate code in executor repo tests https://review.openstack.org/471492 | 21:42 |
| jeblair | clarkb: ^ :) | 21:42 |
| clarkb | thanks, assuming its at the tip of this stack I will get there eventually | 21:42 |
| jeblair | (yep | 21:42 |
| clarkb | https://review.openstack.org/#/c/470390/3 has interesting implications for grenade | 21:43 |
| clarkb | any idea how setup.py develop will handle that | 21:44 |
| clarkb | mordred: ^ | 21:44 |
| jeblair | SpamapS: https://review.openstack.org/456090 is relevant to earlier discussion, but i don't think it changes any conclusions | 21:44 |
| clarkb | jeblair: out of curiousity why not rewrite https://review.openstack.org/#/c/470390/3/tests/unit/test_cloner.py to assert that checking out oldbranch and master works? | 21:44 |
| clarkb | I guess that doesn't really add any coverage compared to other tests? | 21:45 |
| jeblair | clarkb: oh, for grenade i was imagining that we would have devstack-gate clone a second copy on the host, so we'd still have separate old/new checkouts | 21:45 |
| jeblair | mordred: ^ | 21:45 |
| *** hashar has quit IRC | 21:45 | |
| clarkb | gotcha | 21:46 |
| jeblair | clarkb, mordred: so we wouldn't change the underlying branch after having done a python install | 21:46 |
| jeblair | clarkb: and yeah, especially considering ^ i don't think that approach would add coverage | 21:46 |
| clarkb | ya its up to the test framework more so than zuul to affect that | 21:47 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Verify executor branch checkout https://review.openstack.org/468564 | 21:47 |
| jeblair | yup | 21:47 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_project_override https://review.openstack.org/470049 | 21:48 |
| mordred | clarkb, jeblair: I agree re: grenade above | 21:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_periodic https://review.openstack.org/470053 | 21:52 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_post_checkout https://review.openstack.org/470381 | 21:53 |
| jeblair | mordred: one thought about https://review.openstack.org/467635 -- does combining inventory and vars.yaml into one file make it harder for folks to reproduce test runs? | 21:54 |
| mordred | jeblair: oh - that's an excellent point | 21:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Enable test_post_and_master_checkout https://review.openstack.org/470385 | 21:55 |
| jeblair | mordred: (previously, vars.yaml contained "things about the test run which are useful for local reproduction" and inventory had "things you will need to completely replace") | 21:55 |
| mordred | yah | 21:55 |
| mordred | kk. let's skip that one | 21:55 |
| mordred | (the groups and inventory is the more important one) | 21:56 |
| jeblair | mordred: ok. i +2d it anyway, but just thought of that. it's all hand-wavey until we actuall have a reproduce procedure anyway | 21:56 |
| mordred | ++ | 21:58 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix passing labels to Gerrit when they are not defined in All-Projects https://review.openstack.org/465420 | 21:58 |
| jlk | well... | 21:59 |
| jlk | is the inventory file written out? | 21:59 |
| jlk | or do we just manage it internally? | 21:59 |
| mordred | yes, it's written out | 21:59 |
| jlk | so it's still there, just not as easily consumable | 22:00 |
| mordred | as well as the vars file - although we have not yet added saving those things to log server | 22:00 |
| mordred | jlk: well - today we're writing an inventory and a vars file - the change puts the vars into the inventory file | 22:00 |
| mordred | jlk: so rather than the user fetching vars.yaml and ignoring the written inventory, they'd need to fetch the inventory and then replace the host entries | 22:01 |
| clarkb | jeblair: https://review.openstack.org/#/c/470442/2 catches a legit bug in the test fixtures | 22:06 |
| clarkb | jeblair: I have approved it but it won't merge until we fix that (same thing with the cleanup child change) | 22:07 |
| jeblair | clarkb: ah yeah, tobiash's start-reporting change is going to expose race conditions for a while. | 22:10 |
| mordred | jeblair: dontcha love things that expose races? | 22:11 |
| clarkb | jeblair: in this case I think its because a project config defines a project other than itself? | 22:11 |
| jeblair | mordred: well, to be fair, the change *caused* the race :) | 22:11 |
| clarkb | oh wait thats the test that tests that hence the exception | 22:11 |
| jlk | that was a difficult sentence to parse. | 22:11 |
| jeblair | basically, we should suspect any failure of an assertion on reporting counts to be an expression of a race from the start-reporting change | 22:12 |
| jeblair | clarkb: right. a recheck would probably push that through. but i'm trying to fix all the reporting count assertions as they come up | 22:12 |
| jeblair | so i'll do that one now | 22:12 |
| jeblair | https://review.openstack.org/455711 is the change in question | 22:13 |
| jeblair | jamielennox: i get what's going on in https://review.openstack.org/463457 now, and left a comment about a further update. also, i'm curious about your thoughts on my second comment there (at 2017-05-19 20:39:43+0000) -- if we have all of the builds return their own log urls, can we automatically determine the buildset log url from that? | 22:35 |
| jlk | uh... I think I'm running into an executor bug | 22:37 |
| jlk | where it's trying to copy files that already exist? | 22:37 |
| jeblair | jlk: neat! pastebin? | 22:38 |
| jlk | capturing | 22:38 |
| jlk | there's a lot going on here: http://paste.openstack.org/show/611566/ | 22:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add github reporter status-url config option https://review.openstack.org/449794 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove test_git_cache_dir https://review.openstack.org/470389 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove test_upgrade https://review.openstack.org/470390 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove test_periodic_update https://review.openstack.org/470391 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove test_post_checkout https://review.openstack.org/470392 | 22:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename TestCloner TestExecutorRepos https://review.openstack.org/470393 | 22:43 |
| jlk | I'm guessing that shutil.copytree doesn't like the sub contents in there? | 22:43 |
| jeblair | jlk: there was a change to that recently... maybe "rm -rf /var/lib/zuul/ansible" and then try twice more? | 22:45 |
| jeblair | jlk: obviously the first time should work; will be interesting to see if the second time fails or not | 22:45 |
| jlk | so shutil.copytree says that the destination must not exist | 22:45 |
| jlk | but there's no checking in zuul code to see if the destination already exists | 22:45 |
| jeblair | jlk: ah, then i bet this is a real bug | 22:45 |
| jlk | what is the intended behavior? Replace changed files? or initial copy only? | 22:46 |
| jeblair | maybe the fix should just be to rmtree target_dir at the start of _copy_ansible_files | 22:46 |
| jeblair | jlk: i think replace (and remove obsolete) is most desirable | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Reject some untrusted config objects https://review.openstack.org/470442 | 22:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Factor out duplicate code in executor repo tests https://review.openstack.org/471492 | 22:48 |
| jeblair | clarkb: there's the rebase+fix ^ | 22:48 |
| jeblair | also, my gertty dashboard for zuulv3 is once again under 100 changes (oy!) | 22:49 |
| clarkb | jeblair: gerrit doesn't seem to know what the parent of that change is? | 22:51 |
| clarkb | oh I wonder if it s a mereg commit | 22:51 |
| jeblair | should be branch tip merge commit | 22:52 |
| clarkb | doesn't the merger do that for us in the tests? | 22:52 |
| jeblair | clarkb: do what? | 22:52 |
| clarkb | oh I get it your new test also needed the updated value | 22:52 |
| clarkb | jeblair: merge against target branch and provide that as zuul ref | 22:52 |
| clarkb | but the problem was that you also need to change your assertion to 1 in the untrusted project test. I get it now | 22:53 |
| jeblair | yep | 22:53 |
| jeblair | this series and another parallel series had an interaction, which is why it started failing in the gate (when the other series merged) | 22:53 |
| jeblair | i found it easiest to wait for this change's parent to merge, then rebase on branch tip so i'd have latest from both | 22:54 |
| jeblair | pabelanger: what's the zuulv3 relevance of https://review.openstack.org/413103 ? | 22:58 |
| pabelanger | jeblair: that was around talks about creating a minimal chroot for bwrap with DIB | 22:59 |
| pabelanger | we can likely remove the topic for now | 22:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Factor out duplicate code in executor repo tests https://review.openstack.org/471492 | 22:59 |
| pabelanger | since we are using host FS | 23:00 |
| jeblair | pabelanger: k; done | 23:00 |
| jeblair | mordred: are https://review.openstack.org/411512 and https://review.openstack.org/411473 still relevant? | 23:01 |
| jeblair | mordred: i've marked the zk-shim changes WIP | 23:01 |
| jeblair | jamielennox: https://review.openstack.org/445656 has necessary reviews but could use an update for py3 | 23:02 |
| jeblair | Shrews: https://review.openstack.org/464283 is +3able by you | 23:06 |
| jeblair | jhesketh: i'm abandoning some very old changes of yours which I don't think are relevant any more: https://review.openstack.org/304134 and https://review.openstack.org/344248 | 23:13 |
| jeblair | adam_g, mordred, jlk: see comment on https://review.openstack.org/463022 | 23:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Convert some leftover changeish mentions to ref https://review.openstack.org/465718 | 23:34 |
| jlk | blargh! | 23:35 |
| jlk | github goes down right when I'm trying to test against it. | 23:35 |
| jeblair | they have had one 9 of uptime over the past month. | 23:36 |
| jeblair | okay, to be fair, there is a second nine in their uptime percentage. it's just separated from the first by a couple of numbers which are not nines. | 23:38 |
| jeblair | 98.4924% | 23:38 |
| jlk | 95 in the past day | 23:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Reject some untrusted config objects https://review.openstack.org/470442 | 23:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Factor out duplicate code in executor repo tests https://review.openstack.org/471492 | 23:41 |
| clarkb | I wonder if they have to restart their services to deal with a memory leak too >_> | 23:41 |
| jeblair | yes, there aren't any other nines at all in today's uptime | 23:41 |
| *** jkilpatr has quit IRC | 23:42 | |
| jeblair | clarkb: i assume so, but they're more h-a so we usually don't notice. :| | 23:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: sql-reporter: add support for Ref change https://review.openstack.org/466457 | 23:44 |
| *** jkilpatr has joined #zuul | 23:45 | |
| *** jkilpatr has quit IRC | 23:50 | |
| jamielennox | jeblair: i'll fix that one up this morning shouldn't be too hard, i had actually thought it was merged already | 23:50 |
| jamielennox | jeblair: re the buildset uuid | 23:50 |
| jeblair | jamielennox: cool. and heh, yeah, i'm trying to clean house. i'm getting confused about what's merged and what's not too. :) | 23:51 |
| jamielennox | jeblair: yea, i really don't mind what that buildset identifier is, the 'Z'+ format was always a little odd for this use case but i think buildset is the right location for the identifier and thought i may as well use it | 23:52 |
| *** jkilpatr has joined #zuul | 23:53 | |
| jamielennox | it could maybe go on QueueItem instead, but i don't know why it would | 23:53 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 23:53 |
| jamielennox | turns out i completely missed a patch the first time i uploaded the series so i see why you didn't understand it | 23:54 |
| jamielennox | :) | 23:54 |
| jlk | jeblair: ^^ that seems to fix at least one part of second-startup. I think I hit another bit with the log streamer, but that's not fatal. | 23:54 |
| jamielennox | re uwsgi - i don't think you want to use uwsgi from the python side, just expose it as wsgi and let people use what they want to host it | 23:55 |
| *** jkilpatr has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!