| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Encode webhook_token secret https://review.openstack.org/473674 | 00:07 |
|---|---|---|
| pabelanger | Finally | 00:13 |
| pabelanger | got SSL working for gearman in zuul | 00:13 |
| pabelanger | I'll push up a patch in the morning, now for beer | 00:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Restore separate upload logs https://review.openstack.org/474009 | 00:15 |
| *** bstinson has quit IRC | 00:36 | |
| *** bstinson has joined #zuul | 00:36 | |
| *** yolanda has joined #zuul | 00:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add ssl support to gearman / gearman_server https://review.openstack.org/473916 | 01:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add ssl support to gearman / gearman_server https://review.openstack.org/473916 | 01:07 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Show debug logging when running zuul-bwrap https://review.openstack.org/474017 | 01:09 |
| *** yolanda has quit IRC | 01:16 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: github: gracefully handle unknown event https://review.openstack.org/473249 | 01:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Show debug logging when running zuul-bwrap https://review.openstack.org/474017 | 01:57 |
| *** toabctl has quit IRC | 02:11 | |
| *** toabctl has joined #zuul | 02:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: github: gracefully handle unknown event https://review.openstack.org/473249 | 03:06 |
| jamielennox | so i've just got caught out by the log streamer hanging thing, can someone give me a TL;DR; of what's happened there | 04:08 |
| tristanC | jamielennox: you mean the executor task is stuck? | 04:12 |
| jamielennox | tristanC: yea, it looks like the console on the node was stuck waiting for the streamer to connect | 04:13 |
| jamielennox | and it either didn't connect or something and got stuck waiting | 04:14 |
| jamielennox | there are patches from mordred and Shrews overnight that seem to fix at least part of the problem | 04:14 |
| tristanC | jamielennox: it happened to me when using task with delegation, then the zuul_stream module incorectly read_log from the host play instead of the delegated host | 04:15 |
| jamielennox | hmm, i'm not sure that's the same problem then. my role isway simpler than that | 04:16 |
| jamielennox | basically it's just running tox on the node | 04:16 |
| jamielennox | https://review.openstack.org/#/c/472839/ | 04:17 |
| jamielennox | oh, actually maybe not that one | 04:17 |
| *** isaacb has joined #zuul | 04:18 | |
| *** isaacb has quit IRC | 04:19 | |
| tristanC | to debug this I had to strace the zuul_console process on the node, it should hint why it's not sending back task output | 04:19 |
| jamielennox | yea, i had gone down that path but by the time i got to the strace it was just hung on accept(4, | 04:19 |
| jamielennox | after updating master it progresses and im getting Streamer could not join messages | 04:20 |
| tristanC | then it sounds like zuul_stream is stuck somewhere else | 04:21 |
| jamielennox | yea, but so whatever changed in the last day or two of merges has at least progressed the problem so i was just wondering what had been hit | 04:26 |
| jamielennox | i'm still broken elsewhere, but that might be a hint | 04:26 |
| *** EmilienM has quit IRC | 04:30 | |
| *** EmilienM has joined #zuul | 04:35 | |
| *** nt has quit IRC | 04:37 | |
| *** nt has joined #zuul | 04:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: github: gracefully handle unknown event https://review.openstack.org/473249 | 04:38 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Read layout from BuildSet in first merge scenario https://review.openstack.org/474064 | 05:35 |
| *** yolanda_ has joined #zuul | 08:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add support for zuul.d configuration split https://review.openstack.org/473764 | 08:46 |
| *** isaacb has joined #zuul | 09:36 | |
| *** isaacb_ has joined #zuul | 09:37 | |
| *** isaacb has quit IRC | 09:40 | |
| *** yolanda_ has quit IRC | 09:55 | |
| *** hashar has joined #zuul | 10:21 | |
| *** jkilpatr has quit IRC | 10:30 | |
| *** jkilpatr has joined #zuul | 10:48 | |
| Shrews | jamielennox: my patch from yesterday simply terminates the finger connection from the server side once the log file disappears. sounds like you're having problems with the ansible plugins | 11:00 |
| jamielennox | Shrews: yea, i'm definitely having trouble with the plugins | 11:05 |
| jamielennox | i've figured out a little bit, but i've got no idea why the streamer isn't attaching | 11:05 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Sync command from ansible https://review.openstack.org/474171 | 12:00 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Update run_command to latest ansible https://review.openstack.org/474172 | 12:00 |
| mordred | jamielennox: awesome | 12:31 |
| jamielennox | mordred: just trying to get the python 3 sorted out, i expect there will be bigger repocussions | 12:32 |
| mordred | yah | 12:32 |
| jamielennox | not sure though how to realistically have zuul-executor on python3 invoke an ansible running python2 | 12:33 |
| mordred | jamielennox: well - the ansible and python on the remote aren't connected - so using python2 on the node should be fairly easy | 12:35 |
| mordred | jamielennox: we could also be explicit and set ansible_python_interpreter for each host in the inventory | 12:35 |
| mordred | jamielennox: that still may make unittests difficult though | 12:35 |
| jamielennox | mordred: yea, i was surprised by that as well, but i was hitting 'unicode not defined' | 12:35 |
| jamielennox | so i'm not sure how that was bubbling up | 12:36 |
| mordred | jamielennox: so we should definitely fix at least our command module stuff | 12:36 |
| mordred | but it's probably a good idea from an ops perspective to make sure build nodes have python2 as default python | 12:37 |
| jamielennox | yea, i'm just hitting more python3 rough edges than i was expecting | 12:38 |
| jamielennox | because the nodes should be python2 | 12:39 |
| mordred | jamielennox: in the bonny prod env? or in tests? | 12:42 |
| jamielennox | mordred: bonny prod | 12:42 |
| mordred | jamielennox: do you have python3 also installed on your nodes? | 12:42 |
| jamielennox | shouldn't be by default | 12:43 |
| jamielennox | so yea, not sure how thatcame through | 12:43 |
| mordred | yah - I was just looking to see if ansible maybe set ansible_python_interpreter to python3 if ansible itself was invoked as python3 | 12:43 |
| jamielennox | yea, we force set dib python to 2 | 12:45 |
| mordred | nod | 12:45 |
| mordred | that's super weird then | 12:45 |
| jamielennox | hmm, but py2 and py3 are both installed on the node | 12:45 |
| jamielennox | not sure how thatends up in the image - butit's there | 12:46 |
| jamielennox | so yea, maybe if ansible is running py3 it looks for py3 on the host? | 12:46 |
| jamielennox | but 'python' is 2.7 | 12:47 |
| Shrews | jamielennox: i don't think so? it should just default to the system python on the node. which version is that? | 12:53 |
| jamielennox | 'python' is 2.7 | 12:53 |
| Shrews | /usr/bin/python, yeah? | 12:54 |
| jamielennox | right | 12:54 |
| jamielennox | i can reset that stack of patches and see where it's comingfrom | 12:54 |
| jamielennox | but it didn'tgive me much of a backtrace | 12:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove item.layout https://review.openstack.org/474188 | 12:55 |
| jeblair | jamielennox: 474188 is a followup to 474064 | 12:56 |
| jamielennox | jeblair: oh - yea, i thought item.layout wasn't used anywhere. i accidently included that removal in a different patch | 12:56 |
| jamielennox | but never chased it up afterwards | 12:57 |
| jeblair | jamielennox: ok, glad we saw the same thing :) | 12:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Enhance sphinx plugin https://review.openstack.org/473544 | 13:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Sync command from ansible https://review.openstack.org/474171 | 13:08 |
| jamielennox | Shrews, mordred: http://paste.openstack.org/show/612547/ | 13:09 |
| jamielennox | which is not actually that informative | 13:09 |
| jamielennox | but reproducable | 13:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update run_command to latest ansible https://review.openstack.org/474172 | 13:10 |
| *** dkranz has joined #zuul | 13:11 | |
| jeblair | jamielennox: that's fixed in https://review.openstack.org/474172 right? | 13:13 |
| *** isaacb_ has quit IRC | 13:13 | |
| jamielennox | jeblair: yes, but i'm still not sure why command.py on the test node is being run with py3 | 13:14 |
| jeblair | oh gotcha | 13:14 |
| *** openstackgerrit has quit IRC | 13:18 | |
| mordred | jeblair, jamielennox: yah - I think figuring out why py3 is being used on the test node is super important | 13:22 |
| jeblair | mordred: time to ask ansible folks? | 13:23 |
| jamielennox | mordred, jeblair: i think maybe time to ask, because it seems to be works as intended: https://github.com/ansible/ansible/blob/v2.3.0.0-1/lib/ansible/inventory/__init__.py#L508 | 13:25 |
| jeblair | mordred, jamielennox: our worker nodes are xenial, py2 and py3 are both installed, python is py2. but i don't think we've seen this? | 13:25 |
| jamielennox | doh, not that one | 13:25 |
| jamielennox | nvm, obviously that's not it | 13:26 |
| * mordred asking | 13:26 | |
| jeblair | jamielennox: are you using a venv to run zuul-executor and/or ansible-playbook? | 13:27 |
| jamielennox | venv, and then mapping it through bwrap | 13:27 |
| jeblair | jamielennox: how do you map it through bwrap? | 13:28 |
| jamielennox | jeblair: nothing special, whatever zuul is setting up by default | 13:29 |
| jeblair | jamielennox: ok just the stuff in the getPopen method of the zuul bubblewrap driver. gotcha. | 13:29 |
| jamielennox | what's the switch to make ansible not delete the modules it creates | 13:29 |
| jamielennox | can't find it | 13:29 |
| jamielennox | ANSIBLE_KEEP_REMOTE_FILES=1 | 13:30 |
| *** nt has quit IRC | 13:33 | |
| *** nt has joined #zuul | 13:33 | |
| jamielennox | mordred, jeblair: so that's an interesting problem and one that tristanC bought up earlier that i didn't realize - what happens if you are running command with delegate_to: localhost | 13:38 |
| mordred | jamielennox: yah - we need to use the delegated host and not the ansible host | 13:39 |
| mordred | in zuul_stream | 13:39 |
| jamielennox | but zuul_console is probably not running on the executor | 13:39 |
| jamielennox | so does that work? | 13:39 |
| mordred | jamielennox: aha. that is an excellent point | 13:39 |
| mordred | jamielennox: you know - we can likely special-case delegate-to: localhost in zuul_stream | 13:40 |
| mordred | hrm. well, no, we may not be able to do that | 13:40 |
| mordred | because /tmp/console-{uuid}.log isn't going to be available in bubblewrap chroot | 13:41 |
| mordred | jamielennox: maybe for now we special case it to not attempt streaming and instead to just grab the output from stdout on the result object | 13:41 |
| jamielennox | hmm, i still haven't figured out the console-{uuid}.log problem - that broke me today as well | 13:42 |
| mordred | we shouldn't be running tons of long-running scripts on localhost where streaming output would be essential anyway - waiting a few seconds for the command to complete should be fine, right? | 13:42 |
| mordred | jamielennox: lemme make the localhost special-case for you real quick while it's in my brain | 13:43 |
| jamielennox | i think i'd be fine with that, at least for now | 13:43 |
| jeblair | mordred: command module will be able to write to /tmp/console-uuid.log, but zuul_stream won't be able to read it | 13:44 |
| mordred | jeblair: exactly | 13:44 |
| jeblair | mordred: we could tell command module to write to somewhere in the jobdir | 13:44 |
| jamielennox | i'm happy to let you guys debate that - i'm going to bed | 13:46 |
| jeblair | jobdir/delegated_logs or something :) | 13:46 |
| *** isaacb_ has joined #zuul | 13:46 | |
| jeblair | jamielennox: goodnight! | 13:46 |
| jamielennox | i haven't got streamed logs working for the regular jobs yet | 13:46 |
| jamielennox | anyway - night | 13:46 |
| *** openstackgerrit has joined #zuul | 13:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ssl support to gearman / gearman_server https://review.openstack.org/473916 | 13:49 |
| pabelanger | jeblair: mordred: SpamapS: Shrews: ^SSL gearman is ready for review now | 13:50 |
| mordred | jamielennox: night! | 13:51 |
| jeblair | pabelanger: it looks like you created a test fixture CA with a password -- maybe for our test fixtures we should not have a password? | 13:52 |
| jeblair | pabelanger: also, you can drop many commands if you use the ones i suggested | 13:53 |
| jeblair | pabelanger: left a comment with suggested commands. otherwise looks good :) | 13:59 |
| pabelanger | jeblair: Right, I should have documented the password | 14:01 |
| jeblair | pabelanger: yeah, but a password that everyone knows isn't very useful, so let's just skip it. :) | 14:01 |
| pabelanger | sure | 14:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ssl support to gearman / gearman_server https://review.openstack.org/473916 | 14:12 |
| pabelanger | jeblair: thanks, updated using your syntax for openssl now | 14:12 |
| jeblair | pabelanger: w00t! | 14:13 |
| jeblair | Shrews, fungi: do you want to look over 473916? | 14:13 |
| pabelanger | jeblair: can we use the same syntax for zuulv3.o.o / ze01.o.o ssl certs? | 14:13 |
| jeblair | pabelanger: i think so, though we may want to use a password for our actual ca (drop -nodes option). we may want to check in with fungi or clarkb who may have opinions on how we should set up our internal ca. also SpamapS's suggestion of CA.sh is a good one. maybe we should use that? | 14:16 |
| pabelanger | jeblair: sure, I'll move the discussion to #openstack-infra also | 14:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Direct streaming at delegated_to target https://review.openstack.org/474215 | 14:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Special case shell logging on localhost https://review.openstack.org/474216 | 14:21 |
| mordred | jeblair, jamielennox: ^^ | 14:21 |
| mordred | tristanC: you too | 14:22 |
| mordred | jeblair: I've also added a todo comment because clearly there is a piece of flawed logic in there :) | 14:22 |
| mordred | but I didn't want to make this too complex this instant | 14:22 |
| jeblair | mordred: i love it when those come with TODOs! they so seldom do... | 14:22 |
| *** isaacb_ has quit IRC | 14:24 | |
| *** isaacb has joined #zuul | 14:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle lists of streamers https://review.openstack.org/474230 | 14:38 |
| mordred | jeblair: I need to squash those two, because the first one was missing something I just added to the second one | 14:39 |
| *** yolanda_ has joined #zuul | 15:23 | |
| *** hashar has quit IRC | 15:33 | |
| *** dmsimard has joined #zuul | 15:35 | |
| fungi | it's worth debating whether we want an infra-wide ca, or just have one on the zuul (scheduler) server and sign clients there with some more seamless automation. downside is that if someone compromises the zuul server they can sign new client certs and pretend to be an authorized client, but if that's the case then there are bigger problems, right>? | 15:51 |
| fungi | i used to do something similar years ago with per-customer vpn gateways in a service provider setting | 15:52 |
| *** isaacb has quit IRC | 15:59 | |
| gundalow | Hey :) | 16:00 |
| gundalow | For people that will be at London AnsibleFest Contributors Summit the agenda is now online https://public.etherpad-mozilla.org/p/ansible-summit-june-2017-agenda | 16:00 |
| gundalow | Zuul is Afternoon Track 2 | 16:00 |
| gundalow | Not sure if you want to add any stuff to your specific agenda in https://public.etherpad-mozilla.org/p/ansible-summit-june-2017-Zuul | 16:01 |
| jlk | o/ | 16:12 |
| jlk | thanks gundalow | 16:13 |
| gundalow | jlk: Thanks for all the great stuff you are doing | 16:13 |
| gundalow | TEST ALL THE THINGS | 16:13 |
| * jlk goes to review all the things | 16:20 | |
| *** nt has quit IRC | 16:39 | |
| *** yolanda_ has quit IRC | 16:40 | |
| *** yolanda_ has joined #zuul | 16:40 | |
| *** nt has joined #zuul | 16:42 | |
| *** yolanda_ has quit IRC | 16:43 | |
| *** yolanda_ has joined #zuul | 16:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add build.started state flag https://review.openstack.org/473811 | 16:47 |
| jeblair | mordred: mapped that to an existing storyboard task ^ | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: add configuration documentation https://review.openstack.org/463328 | 16:53 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline reject filter for github https://review.openstack.org/474001 | 16:54 |
| jlk | Picking up the last recorded task on the github support story | 16:56 |
| jeblair | jlk: what's that? | 16:56 |
| jlk | handle sending reports for events that aren't based on a pull request (eg a push event) | 16:57 |
| jeblair | gotcha | 16:57 |
| jlk | the current report code assumes there is a PR to report to | 16:57 |
| jlk | oh I guess I should write up the task (or maybe a whole new story) for cross-repo deps | 17:00 |
| jeblair | jlk: ah yeah -- also, before we start work on implementing that, i think we may want to take a look at https://review.openstack.org/451423 | 17:01 |
| jlk | nod | 17:01 |
| jeblair | it may be related in that i think much of the CRD stuff will need to move out of the gerrit driver into the pipeline manager (so we can do cross-connection dependencies) | 17:01 |
| jeblair | okay, i just updated a bunch of stories; next time the cron runs we should have a much tidier board | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add a success-url for status.json test https://review.openstack.org/473604 | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Support finger ports in finger URL https://review.openstack.org/473103 | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add build.started state flag https://review.openstack.org/473811 | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use worker_name for job cancellation and remove manager https://review.openstack.org/474288 | 17:04 |
| mordred | jeblair, jlk: rebased that removing the move-to-model bits | 17:05 |
| jlk | k | 17:05 |
| *** nt has quit IRC | 17:16 | |
| *** nt has joined #zuul | 17:18 | |
| *** dkranz has quit IRC | 17:36 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Limit github reporters to event types https://review.openstack.org/474300 | 17:39 |
| pabelanger | mordred: Shrews: would you like to review and +3 473889? | 17:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Clean up docstring on SSLZuulTestCase https://review.openstack.org/474308 | 17:55 |
| *** yolanda__ has joined #zuul | 18:09 | |
| *** yolanda_ has quit IRC | 18:10 | |
| pabelanger | jeblair: mordred: SpamapS: I am noticing a delay of about 1 min between each ansible-playbook run. Is it possible something with bubblewrap or sshagent is coming into play here? http://paste.openstack.org/show/612580/ | 18:19 |
| mordred | pabelanger: worth looking in to / profiling | 18:19 |
| pabelanger | Ya, just starting that now. Jobs are definitely running slowing on zuulv3.o.o now | 18:20 |
| pabelanger | or, longer. Depending on how you look at it | 18:20 |
| jeblair | mordred, pabelanger: or waiting for console stream tasks to end? | 18:22 |
| jeblair | pabelanger: i think the verbose changes landed. how about we update zuul on ze01, and restart it, then run "zuul-executor verbose" and see what additional logging tells us | 18:22 |
| pabelanger | jeblair: wfm | 18:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add a success-url for status.json test https://review.openstack.org/473604 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use worker_name for job cancellation and remove manager https://review.openstack.org/474288 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Support finger ports in finger URL https://review.openstack.org/473103 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add build.started state flag https://review.openstack.org/473811 | 18:26 |
| mordred | jeblair: I was thinking console stream tasks - but then I'd expect to see delays between tasks too | 18:28 |
| pabelanger | ze01.o.o in graceful mode now | 18:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Create nodepool dictionary for ansible inventory https://review.openstack.org/473889 | 18:30 |
| jeblair | pabelanger: should be okay to hard stop, why graceful? | 18:30 |
| pabelanger | jeblair: I didn't think we had jobs running :) but ya, hard should also work | 18:30 |
| pabelanger | okay, hard stop worked | 18:35 |
| jeblair | pabelanger: how did you hard stop after graceful? | 18:35 |
| pabelanger | I just ran zuul-executor stop | 18:36 |
| pabelanger | and it worked | 18:36 |
| jeblair | huh. shouldn't have. | 18:36 |
| pabelanger | oh, interesting we have a bunch of zuul-executor processes running | 18:37 |
| pabelanger | I guess we'll need our systemd service file sooner then later | 18:37 |
| mordred | would be good to figure out why they're not shutting down | 18:38 |
| jeblair | pabelanger: i don't understand the systemd connection | 18:38 |
| mordred | we've got a bunch of subprocess and thread stuff going on - it's entirely possible we're not cleaning up properly and causing something to hang | 18:38 |
| jeblair | don't kill them | 18:38 |
| jeblair | open("/tmp/b82946f7a81a496bbfa45451606c34b2/work/logs/job-output.txt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) | 18:38 |
| jeblair | one of them is looping on that | 18:38 |
| pabelanger | service zuul-executor status | 18:39 |
| pabelanger | shows us creating multiple processes | 18:39 |
| pabelanger | I was thinking it might be related to our systemd-sysv-generator usage | 18:39 |
| jeblair | so that one at least looks like a log stream subproc that didn't shut down | 18:40 |
| jeblair | mordred: i think that's pre-switch-to-threads | 18:40 |
| jeblair | i think that also landed? | 18:40 |
| Shrews | you will see an additional zuul-executor for the finger daemon | 18:40 |
| mordred | jeblair: yah - I think restarting with the thread changes landed before we debug that too much furtheris likely good | 18:41 |
| jeblair | yep. gimme a min to go through the rest of the procs | 18:41 |
| mordred | also - with Shrews patch to stop streaming when the file goes away | 18:41 |
| jeblair | mordred, pabelanger, Shrews: they are all stuck on that loop, so i think they are all leaked streaming subprocs. i agree, we should ignore these and see if behavior is improved with those 2 patches | 18:42 |
| jeblair | pabelanger: i'm done investigating; i think we can kill them all now unless anyone wants to poke at anything else | 18:42 |
| mordred | jeblair: nope - killem | 18:43 |
| Shrews | interesting that's where they'd be looping | 18:43 |
| jeblair | all dead | 18:43 |
| jeblair | pabelanger: feel free to proceed with restart now | 18:44 |
| pabelanger | jeblair: ack | 18:44 |
| jeblair | Shrews: i assume that's "tail -f" behavior that you removed | 18:45 |
| mordred | but why would they be trying to re-open the file - I'd expect them to be stuck trying to read from it | 18:45 |
| jeblair | Shrews: (inode changed (because it was deleted); keep trying to reopen file) | 18:45 |
| jeblair | mordred: file was deleted | 18:45 |
| mordred | nod | 18:45 |
| pabelanger | ze01.o.o running with verbose enabled | 18:46 |
| mordred | ah - yes, I see it now | 18:46 |
| jeblair | pabelanger: i'm watching f04a3587f69f401a85b940ca537c73e5 | 18:49 |
| jeblair | looks like 2 seconds from getting the job to first ansible-playbook invocation | 18:49 |
| jeblair | no delay from first bubblewrap to ansible output | 18:50 |
| jeblair | but now we're waiting after the first play finished | 18:51 |
| pabelanger | I see a 1min delay for starting tox/pre job | 18:51 |
| jeblair | pabelanger: that's not the first playbook though, right? | 18:51 |
| jeblair | i'm trying to establish whether this happens before each, after each, or between each playbook invocation | 18:52 |
| pabelanger | right, I think it is happening after | 18:52 |
| pabelanger | our exit code is delayed by 1min for some reason | 18:52 |
| jeblair | mordred, pabelanger: can you merge https://review.openstack.org/473222 please? | 18:53 |
| pabelanger | +3 | 18:53 |
| jeblair | with the thread set to daemon, i dont't *think* that should cause a delay | 18:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: zuul_stream: handle empty line https://review.openstack.org/473222 | 19:00 |
| jeblair | pabelanger: there may be a 0-10 second delay from stopping the watchdog | 19:06 |
| pabelanger | right, I do remember that | 19:07 |
| pabelanger | root 28413 0.0 0.0 4380 752 ? S 19:09 0:00 sleep 60 | 19:10 |
| pabelanger | I just seen that on ze01.o.o | 19:10 |
| jeblair | pabelanger: zuul does not run as root | 19:10 |
| pabelanger | Hmm | 19:10 |
| jlk | With gerrit, does the gerrit stream for an event already have a key for dependsOn ? | 19:13 |
| jeblair | i don't see any errors from zuul_stream in the log | 19:13 |
| jeblair | jlk: yes -- that's git dependencies between gerrit changes | 19:13 |
| jeblair | jlk: i reckon that doesn't really apply to github | 19:13 |
| pabelanger | I did also notice a few [bwrap] <defunct> processes too. any idea why that would be? | 19:13 |
| jlk | ahhh, so that's like if I just make my commit on top of an existing change | 19:14 |
| jlk | gotcha | 19:14 |
| jlk | yeah for github, you just include that commit in your PR since you can have multiple. | 19:14 |
| jeblair | jlk: seeing as how even if you did build a second pr on top of a first one, github would happily merge the second (and therefore also merge in the changes from the first) | 19:14 |
| jeblair | right | 19:14 |
| jlk | cool, I'll skip that block of code | 19:14 |
| jeblair | ++ interesting bits should just be what i called "commit depends" in the gerrit driver | 19:15 |
| jlk | ooh fun, I'll have to trawl every commit in a PR to see if any of them have a DependsOn... | 19:16 |
| * SpamapS is catching up | 19:24 | |
| pabelanger | other difference between zuulv3-dev and zuulv3 is we are running ansible as python3 | 19:29 |
| mordred | pabelanger: yes. this is especially important to note when anything runs on localhost | 19:35 |
| mordred | since that means the ansible _content_ runs in python3 | 19:35 |
| * jlk lunches | 19:36 | |
| *** dkranz has joined #zuul | 20:03 | |
| jeblair | pabelanger, tristanC, mordred, SpamapS: i think i'm convinced we should run bubblewrap for all playbooks, but with different options. what do you think of this writeup? https://storyboard.openstack.org/#!/story/2001070 | 20:05 |
| mordred | jeblair: ++ | 20:08 |
| pabelanger | wfm | 20:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some debug entries to executor https://review.openstack.org/474346 | 20:46 |
| jeblair | pabelanger, mordred: ^ i doubt that will show us the smoking gun, but hopefully it should help exclude some things. | 20:46 |
| mordred | jeblair: +2 | 20:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove ssh private key from ansible.cfg https://review.openstack.org/474350 | 20:51 |
| pabelanger | jeblair: also +2, feel free to approve when ready | 20:52 |
| pabelanger | jeblair: SpamapS: mordred: ^ also noticed we were still passing our private key into ansible.cfg, we should be able to drop that now right? | 20:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove job_name_in_report option https://review.openstack.org/474352 | 20:56 |
| jeblair | pabelanger: i think so | 20:56 |
| *** dmsimard has quit IRC | 20:56 | |
| jeblair | pabelanger: 474352 will fix the fact that the job reports don't look right (we could also fix our puppet to set that value to true, but now seems like a good time to clean up the option) | 20:57 |
| pabelanger | jeblair: Ah, thanks. That explains it | 21:00 |
| jeblair | SpamapS, mordred: https://review.openstack.org/473301 could use a +3 | 21:03 |
| jeblair | mordred: i think https://review.openstack.org/473966 needs a refresh | 21:06 |
| jeblair | mordred: i'm confused about https://review.openstack.org/472964 -- the lastest commit message looks like it's just a setup to let us turn off the hostname for one-node jobs with a one-liner, except it looks like the patch actually does turn it off. | 21:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add ssl support to gearman / gearman_server https://review.openstack.org/473916 | 21:10 |
| pabelanger | nice | 21:10 |
| *** jkilpatr has quit IRC | 21:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Clean up docstring on SSLZuulTestCase https://review.openstack.org/474308 | 21:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: github: retry pull_request() https://review.openstack.org/473301 | 21:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add coverage artifacts to .gitignore https://review.openstack.org/472353 | 21:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ssh private key from ansible.cfg https://review.openstack.org/474350 | 21:15 |
| mordred | jeblair: that's because I suck | 21:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some debug entries to executor https://review.openstack.org/474346 | 21:16 |
| jeblair | i'm self-reapproving that ^ for a trivial fix | 21:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove job_name_in_report option https://review.openstack.org/474352 | 21:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make sure we always log the exit line https://review.openstack.org/473966 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add a success-url for status.json test https://review.openstack.org/473604 | 21:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Change log streaming link to finger protocol https://review.openstack.org/437764 | 21:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use worker_name for job cancellation and remove manager https://review.openstack.org/474288 | 21:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract get_playhosts listing in zuul_stream to a method https://review.openstack.org/472964 | 21:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Support finger ports in finger URL https://review.openstack.org/473103 | 21:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add build.started state flag https://review.openstack.org/473811 | 21:24 |
| jeblair | mordred: i'm expecting test failures on https://review.openstack.org/474288 i left a comment. if you fix that up you may want to go ahead and fix the commit message on the parent, https://review.openstack.org/473811 | 21:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add some debug entries to executor https://review.openstack.org/474346 | 21:33 |
| *** dmsimard has joined #zuul | 21:33 | |
| mordred | jeblair: yah - I'm actually expecting test failures on the parent | 21:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove job_name_in_report option https://review.openstack.org/474352 | 21:35 |
| jeblair | mordred: whole stack reviewed | 21:35 |
| mordred | jeblair: thnaks! | 21:36 |
| *** jkilpatr has joined #zuul | 21:58 | |
| *** yolanda__ has quit IRC | 22:07 | |
| *** yolanda__ has joined #zuul | 22:15 | |
| *** yolanda__ is now known as yolanda | 22:15 | |
| jlk | Is there somewhere that explains the utility of "neededBy" ? | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Add image-id and image-name options to cloud-images https://review.openstack.org/474369 | 22:22 |
| jeblair | mordred, rcarrillocruz: ^ that's the enhancement i mentioned in my email the other day | 22:22 |
| *** yolanda has quit IRC | 22:23 | |
| jeblair | jlk: not sure; it's used so that if B depends on A, and B is approved *before* A, once A is approved, zuul can find B and, assuming it matches pipeline requirements, pull it in to a dependent (ie gate) pipeline. | 22:24 |
| jeblair | jlk: similarly to dependsOn, gerrit automatically provides neededBy to indicate git children of a change | 22:24 |
| jlk | yeah, and it also scrapes for commits | 22:25 |
| jeblair | jlk: for commit dependencies, we do a gerrit search for Depends-On footers to find any backlinks. | 22:25 |
| jeblair | right | 22:25 |
| jlk | oh hrm, it regexes for Depends-On. | 22:26 |
| jeblair | jlk: NB, there is a custom in some areas of openstack of writing "Needed-By:" footers manually in commit messages. that's for humans only and zuul ignores those. only "Depends-On" footers have any impact on zuul. | 22:26 |
| jlk | okay yeah I thought I caught some of that conversation | 22:26 |
| jlk | so I'd say that needs to carry over to github. | 22:28 |
| jeblair | jlk: this is a really interesting question for github. can we search all of github for commits which contain "Depends-On: https//github.com/foo/bar/pulls/X" ? | 22:28 |
| jlk | oh eww. Kinda, but.. eww. | 22:28 |
| jeblair | i guess i just like problems that start with "search all of ..." :) | 22:29 |
| jeblair | i hope that if github is going to suck all of the world's software development into their system, they'd at least make a nice search index for us. :) | 22:30 |
| jlk | I'm reading the gerrit driver, it's querying for message:<change_id> | 22:31 |
| jlk | is that querying every open change? | 22:32 |
| jeblair | jlk: i think maybe at the time gerrit did not support a regex query on that field... or some kind of similar complication, which was why we queried for the id at all, then narrowed it down with a regex locally. | 22:32 |
| jlk | I see. | 22:32 |
| jeblair | jlk: quite likely *every* change, open or not. | 22:33 |
| jlk | seems expensive | 22:33 |
| jeblair | jlk: that could probably be optimized... | 22:33 |
| jeblair | jlk: it's indexed. pretty fast actually. | 22:33 |
| jeblair | jlk: "message:<change_id>" is "return every change where <change_id> appears in the commit message" | 22:33 |
| jlk | I see | 22:33 |
| jeblair | runTimeMilliseconds: 4 | 22:34 |
| jeblair | that's from a random query i just did | 22:34 |
| jlk | so you can search github for every commit, for the string, but probably not a regex | 22:35 |
| jlk | https://developer.github.com/v3/search/#search-commits | 22:35 |
| jlk | not exactly sure what the limitations of the "search term" string are. | 22:36 |
| jeblair | jlk: we were nice and set re.IGNORECASE in gerrit. but i think that's about the only thing you'd lose if you can search for the whole string "Depends-On: foo". | 22:36 |
| jlk | could we get away with just searching already known data, the cache of changes, if all the change commit messages were within that cache? | 22:37 |
| jlk | might get cache misses after a restart | 22:37 |
| jeblair | jlk: the misses could be a problem here -- this is especially useful for changes people approved a long time ago and it's just now that their deps got around to being approved. | 22:38 |
| jlk | nod | 22:39 |
| dmsimard | I guess you guys have come across this ? I've been living in a cave until recently so from catching up in my feed: https://coreos.com/blog/introducing-zetcd | 22:43 |
| jlk | dmsimard: we chatted about it a bit | 22:43 |
| jlk | jeblair: e.g. https://github.com/search?l=&q=co-authored-by%3A+Jesse+Keating&ref=advsearch&type=Commits&utf8=✓ | 22:43 |
| dmsimard | not good enough to consider moving away from zk ? :p | 22:43 |
| jlk | dmsimard: not for zuulv3 | 22:43 |
| jlk | dmsimard: there was talk of trying to run the zuul test suite against a zetcd instance instead of OG Zookeeper, but I don't know if anybody got around to it. | 22:44 |
| dmsimard | I was just mostly curious but thanks :) | 22:45 |
| jlk | oh this is gross. I'd have to search for the commits with that string in the commit message, then search the PRs to see if they have that commit as part of the PR. Can be done, just a lot of API calls. | 22:51 |
| jeblair | i love their data model | 22:56 |
| jeblair | jlk: any chance *that* gets better with graphql? :) | 22:56 |
| jlk | yeah.... maybe? | 22:57 |
| mordred | jeblair, jlk: I wonder if we could record the neededby info on the pr in github itself - like in a comment or something, such that it could be easily searched | 22:57 |
| mordred | since they do have the idea of cross-referencing things in comments and I _think_ I saw something about that in the api somewhere | 22:57 |
| jlk | in theory you could record it in the pull request body | 22:57 |
| jlk | which is usually the commit message itself, but not always, and not when you have multiple commits | 22:58 |
| jlk | but then we'd have to react to edits to the body | 22:58 |
| mordred | right- but the neededby is zuul computed | 22:58 |
| mordred | I was more wondering if when zuul notices that b depends-on a, it could do $something on the github api related to a that would leave a breadcrumb it could later search for | 22:59 |
| jlk | ah. | 22:59 |
| * mordred just thinking out loud | 23:00 | |
| jeblair | it feels a little weird externalizing this onto something user-visible, even if it might be useful information for humans... i'd love it if that could be hidden (or at least optionally hidden) | 23:00 |
| jlk | kinda feels wrong to be using github itself as the datastore for zuul metadata | 23:00 |
| mordred | jlk, jeblair | 23:00 |
| mordred | jlk, jeblair: I agree with both of you | 23:00 |
| jlk | this gets even more fun and twisted if we start to think about cross-connection depends | 23:00 |
| clarkb | github PRs can have their titles, body, and comments searched | 23:00 |
| jeblair | it's a good idea, and i'm struggling with it since i know there are people that would love "automatically update prs to leave needed-by breadcrumbs". i just bet there are also people that would be like "stop spamming my comments". | 23:01 |
| clarkb | does that include the commit message data? | 23:01 |
| jlk | clarkb: not directly it would seem | 23:01 |
| jlk | you can search for PRs that include a commit, but not the message within the commit | 23:02 |
| jeblair | jlk: i guess the good news is that in practice, because of the extreme specificity of the search, we're only likely to get back a small number of commits, and typically a 1:1 relation to prs.... | 23:04 |
| jlk | yeah | 23:04 |
| jlk | oh hrm. did we change "image" to "label" in the job specifications? | 23:05 |
| jeblair | jlk: yes | 23:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Special case shell logging on localhost https://review.openstack.org/474216 | 23:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle lists of streamers https://review.openstack.org/474230 | 23:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Direct streaming at delegated_to target https://review.openstack.org/474215 | 23:17 |
| jlk | Who likes regexes, and can help me build a shoddy regex quickly? | 23:18 |
| jlk | something that can match "Depends-On: https://github.com/ansible/ansible/pull/12345" | 23:19 |
| clarkb | jlk: ^[Dd]epends-[Oo]n: https://github.com/.+/.+/pull/[0-9]+ | 23:21 |
| clarkb | may want to make it completely case insensitive | 23:21 |
| mordred | clarkb: do you need to anchor that at the end? | 23:21 |
| clarkb | mordred: possibly, I'm not sure if there are parameters allowed too so left it open | 23:22 |
| jlk | re.compile(r"^Depends-On: https://github.com/.+/.+/pull/[0-9]+$", | 23:22 |
| clarkb | like ?foo=bar&baz=otherthing | 23:22 |
| jlk | oh maybe I need +*$ ? | 23:22 |
| jlk | I don't think anything else is allowed. | 23:23 |
| jlk | ends with the number | 23:23 |
| clarkb | what you have with the $ shoudl be fine if thats the case | 23:23 |
| clarkb | don't add the *, + is modifer that means one or more, * means zero or more (but we need at least one digit) | 23:23 |
| SpamapS | jlk: you're going to want to make github.com flexible for GHE users | 23:28 |
| mordred | SpamapS: ++ | 23:29 |
| clarkb | ugh does that mean we'll also have to support both github apis? | 23:30 |
| clarkb | or just tell GHE users to upgrade once zuul is reading to use the new one? | 23:30 |
| SpamapS | Well for now the one we support is in GHE | 23:30 |
| SpamapS | Who knows how long Github will take to update GHE to support v4 | 23:30 |
| mordred | or how long it'll take us to update to support v4 | 23:31 |
| mordred | SpamapS: I get the sense though that gh is already built on top of the graphql stuff, so maybe rolling v4 out to ghe won't be a long process | 23:32 |
| SpamapS | indeed, it may be more that they're making the graphql known | 23:33 |
| SpamapS | and supported | 23:33 |
| SpamapS | Also, I might contradict myself here by saying we probably shouldn't care about GHE unless somebody with access to a GHE is willing to do automated testing somehow.. eventually. | 23:34 |
| jlk | SpamapS: of course, this is just a quick and dirty to make sure the rest of my code is working | 23:36 |
| jlk | also I still think it's an interesting thought conversation to have on how we'll handle cross-connection dependency searching | 23:37 |
| SpamapS | ACK | 23:46 |
| jlk | oh great. So I can match the Depends-On: string, but then I have to manipulate it down to the relevant parts of a PR URI | 23:47 |
| clarkb | jlk: python has a lib for that (urlparse) | 23:49 |
| jlk | yeah | 23:49 |
| mordred | jlk: https://github.com/ansible/ansible/pull/12345 isn't directly usable? do you need to instead make that be ansible/ansible 12345 ? | 23:58 |
| jlk | yeah, given how we make use of github3.py | 23:58 |
| mordred | nod | 23:58 |
| jlk | I'm going to do this with a couple rsplits() | 23:59 |
| jlk | on '/' | 23:59 |
| mordred | ++ | 23:59 |
| mordred | since you already regex'd it - you know it has a more strict form than arbitrary url | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!