| fungi | heads up for those not following #openstack-infra, suspecting we merged a new memory leak in the past few days. zuulv3.o.o only made it ~10 hours from the last restart before exhausting ram | 00:24 |
|---|---|---|
| *** xinliang has quit IRC | 00:37 | |
| jlk | damn :( | 00:39 |
| *** xinliang has joined #zuul | 00:50 | |
| *** xinliang has quit IRC | 00:50 | |
| *** xinliang has joined #zuul | 00:50 | |
| jeblair | i've restarted openstack's zuul with a manual revert of 511355 which seems like an obvious first candidate | 01:07 |
| *** tobiash has quit IRC | 01:15 | |
| mnaser | growth rate seems much better http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 01:22 |
| *** tobiash has joined #zuul | 01:24 | |
| openstackgerrit | Tom Barron proposed openstack-infra/zuul-jobs master: Collect output from coverage job https://review.openstack.org/512910 | 01:57 |
| SpamapS | jeblair: weakref has failed me in such things before. Turns out you still have to find all the references :-/ | 02:36 |
| dmsimard | jeblair: I wonder if it is worth considering moving from 'git push' to a synchronize (which also uses rsync) here: https://github.com/openstack-infra/zuul-jobs/blob/ecd0d55ce175caf8bc66404c65d2b35c39cf5e49/roles/mirror-workspace-git-repos/tasks/main.yaml#L7 | 03:19 |
| dmsimard | jeblair: It would allow us to drop the requirement on delegating to localhost and thus making the role testable/runnable outside of trusted context | 03:20 |
| *** openstackgerrit has quit IRC | 03:22 | |
| tobiash | dmsimard: that's an optimization to push changes incrementally to the nodes and make use of the cache | 03:33 |
| dmsimard | tobiash: I don't see what's the difference in the outcome between using git push vs rsync in this context | 03:33 |
| dmsimard | but I'm testing a patch :) | 03:33 |
| tobiash | Synchronize could defeat that as the git data structure on disk can be very different even with same content (pack files) | 03:34 |
| tobiash | I think it's safer and cheaper to rely on git for that incremental update | 03:36 |
| dmsimard | tobiash: perhaps we can consider that, it seems that in practice, rsync is even sometimes faster than git: https://serverfault.com/questions/344546/which-is-better-for-website-backup-rsync-or-git-push | 03:38 |
| dmsimard | tobiash: I'll put up a patch and we can discuss it there | 03:40 |
| tobiash | Ok | 03:40 |
| clarkb | dmsimard: we already had a big discussion about it fwiw | 03:41 |
| clarkb | git doesn't work that way | 03:41 |
| dmsimard | clarkb: why not | 03:41 |
| dmsimard | clarkb: I've tested it locally to make sure I was making sense before proposing that you know :( | 03:41 |
| clarkb | dmsimard: because if I have a nova repo and you have a nova repo nothing requires their files to be the same | 03:42 |
| clarkb | so an rsync can end up doing a complete copy of the repo | 03:42 |
| dmsimard | i.e, on one machine "git clone http://git.o.o/foo/bar", one another machine with a fetch+checked out review, rsync that and it just works -- no need to check out anything etc | 03:42 |
| dmsimard | clarkb: is that really going to happen in practice ? the delta between the git repos and the checked out ref should be minimal | 03:42 |
| clarkb | dmsimard: yes it happens in practice | 03:43 |
| clarkb | particularly since things get repacked at different times so the packfiles will be different | 03:43 |
| dmsimard | blehhhhhhhhh | 03:43 |
| clarkb | so if you want to make use of the cache you have to let git figure out what data needs to be moved | 03:44 |
| clarkb | basically oyu can only use rsync reliably if you want ot make an entire copy of the repo, not to make incremental updates | 03:44 |
| dmsimard | I suppose the node couldn't do a git pull from the merger ? Rather than the executor pushing it out ? | 03:44 |
| dmsimard | I'm really just trying to make that role not rely on localhost (executor) tasks so that it can be untrusted | 03:45 |
| tobiash | Zuul3 has a push architecture for reasons | 03:45 |
| tobiash | Such that the nodes don't need a back channel to the control plane | 03:46 |
| clarkb | ya, v2 did the pull but in many setups that is problematic because access to things | 03:46 |
| dmsimard | I guess | 03:48 |
| dmsimard | fine then, I'll find another way | 03:48 |
| dmsimard | ಠ_ಠ | 03:48 |
| tobiash | Maybe there is a git module which could do this | 03:48 |
| dmsimard | tobiash: what do you mean ? | 03:49 |
| tobiash | (which could be allowed for untrusted projects) | 03:49 |
| dmsimard | tobiash: there is an ansible git module, but that doesn't change the fact that we'd run it from the executor | 03:49 |
| dmsimard | and running things on the executor in untrusted context is a no-no | 03:49 |
| tobiash | dmsimard: it's not that strict | 03:49 |
| tobiash | There are a few modules where zuul hooks into and checks paths | 03:50 |
| tobiash | These are then (partly) a | 03:50 |
| tobiash | Allowed un untrusted context | 03:50 |
| dmsimard | tobiash: is review.openstack.org up for you ? | 03:50 |
| tobiash | No | 03:51 |
| tobiash | Unreachable | 03:51 |
| dmsimard | oh, saw the backlog in #openstack-infra | 03:51 |
| *** xinliang has quit IRC | 06:17 | |
| *** xinliang has joined #zuul | 06:26 | |
| *** hashar has joined #zuul | 06:48 | |
| *** _ari_ has quit IRC | 08:06 | |
| *** weshay|ruck has quit IRC | 08:07 | |
| *** _ari_ has joined #zuul | 08:09 | |
| *** weshay has joined #zuul | 08:09 | |
| *** mnaser has quit IRC | 08:30 | |
| *** mnaser has joined #zuul | 08:37 | |
| *** electrofelix has joined #zuul | 08:45 | |
| *** hashar is now known as hasharAway | 09:53 | |
| kklimonda | is there a requirement that nodepool-launcher will be connected to the same internal network as nodes it's spawning? interface_ip of the VM is used for keyscan, but that resolves to private IP (and not FIP) in my deployment https://github.com/openstack-infra/nodepool/blob/feature/zuulv3/nodepool/driver/openstack/handler.py#L200 | 10:46 |
| kklimonda | also, https://github.com/openstack-infra/nodepool/blob/feature/zuulv3/nodepool/driver/openstack/handler.py#L175 talks about Public IPs, but checks interface_ip | 10:47 |
| tobiash | kklimonda: that sounds like a bug in clouds.yaml or shade | 11:12 |
| tobiash | the interface_ip should be the ip nodepool can use for connecting | 11:13 |
| *** openstackgerrit has joined #zuul | 11:25 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: WIP: Add script for deterministic key generation https://review.openstack.org/513003 | 11:25 |
| tobiash | jeblair: this is my poc script which generates the keys for every project in the main.yaml (supports only simple main.yamls yet) ^ | 11:26 |
| tobiash | maybe it makes sense also to introduce a driver concept for this and let the admin decide if he wants to generate a secret per project or let them be derived from a master key | 11:27 |
| tobiash | with the master key we also could do on-the-fly key generation (with caching) and avoid writing them to disk | 11:28 |
| kklimonda | tobiash: thanks, seems to be my misinterpretation of the "default_interface" in clouds.yaml | 11:30 |
| tobiash | kklimonda: you have default_interface in clouds.yaml? | 11:30 |
| kklimonda | tobiash: yes, I've been struggling with shade and tried couple of different settings to get it to assign FIPs and that was part of the config | 11:32 |
| kklimonda | tobiash: apparently it wasn't needed, so I just removed it | 11:33 |
| kklimonda | shade is still assigning FIP from the wrong pool - I can't figure out how to make it work, but at least it's assigning something | 11:33 |
| kklimonda | (before, I was spawning instances in 1.1.1.0/24 network and shade decides it's public, so no FIP needed) | 11:34 |
| tobiash | kklimonda: fip pools is probably a mordred question | 11:35 |
| kklimonda | yeah, already pinged him on #openstack-dev but I think he's in US TZ? | 11:35 |
| tobiash | Yes, he's in us tz | 11:36 |
| Shrews | kklimonda: i'm not really around just yet (gotta go deal with breakfast), but maybe some of things here will help: http://git.openstack.org/cgit/openstack/os-client-config/tree/doc/source/user/network-config.rst | 11:51 |
| mordred | kklimonda, Shrews: I see a couple of issues (also, I'm in europe this week) | 12:03 |
| mordred | kklimonda, Shrews, tobiash: the first is that we don't have any flags for overriding whether a network is a FIP network or not ... BUT ... more importantly, I think we need to swap the order of how we're finding the correct network from which to get a FIP to using the _by_router method first | 12:05 |
| mordred | and we need to update that to be able to look for a router that is attached to the network that the port on the server is attached to | 12:06 |
| tobiash | looks like it returns the external network of the last router found? | 12:06 |
| tobiash | https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L5412 | 12:06 |
| mordred | tobiash: yah - it's a bit too simplistic - it doesn't take in to account whether or not the router in quesiton is attached to the same network as the server | 12:07 |
| tobiash | kklimonda: you maybe also just have to add the external network you want to use to the clouds.yaml with routes_externally: true (see Shrews link) | 12:08 |
| tobiash | mordred: so I'm lucky, also just had the case that I have two external networks and shade used the correct one by accident :) | 12:08 |
| mordred | tobiash: I think the issue is actually https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L5462-L5464 | 12:09 |
| mordred | tobiash: we collect all router:external networks and then just use the first one if there is more than one ... that's not a great approach | 12:09 |
| mordred | (of course, it would be SUPER great if we didn't have to do any of this logic :) ) | 12:09 |
| mordred | tobiash: but I think if we swap the order of those AND update the router method to be able to be queried with an internal network id - "find floating network for internal network by router" - I think it should work more better | 12:11 |
| mordred | but I think we also need to add the ability to turn floating_network on and off in clouds.yaml like the other qualities | 12:11 |
| tobiash | there's probably a hack which already works if you specify all networks in clouds.yaml and use routes_externally=false for every net except the fip net you want to use | 12:12 |
| mordred | tobiash: yah - I thought so too roriginally - but we don't seem to filter external_ipv4_floating_networks at all like we do for the others | 12:13 |
| tobiash | wow, then I'm probably super lucky :) | 12:14 |
| mordred | yah. I think so :) | 12:14 |
| kklimonda | yes, routes_externally isn't taken into account - I have configuration like that http://paste.openstack.org/show/623958/ but as mordred just said, shade picks the first FIP pool it's been given | 12:15 |
| tobiash | kklimonda: can you switch the gateway on the router to the ither fip network as a workaround? | 12:16 |
| mordred | kklimonda: yah - sorry about that ... I think you and tobiash are the first two people I'm aware of who have more than one router:external network - and tobiash seems to have been lucky so far -so you're the first person to report the issue. lucky you! | 12:16 |
| pabelanger | morning! | 12:17 |
| kklimonda | mordred: this could be as well due to opencontrail not behaving 100% as expected | 12:17 |
| mordred | kklimonda: I don't think it'll be super hard to fix - but I might not be able to fully finish fixing it until tomorrow (at a conference, still need to write some slides) | 12:17 |
| kklimonda | mordred: no problem, short term I can just patch things myself, and wait for the proper solution :) | 12:17 |
| mordred | kklimonda: well - so far I think whatyou have described is just a scenario not taken in to account. opencontrail could also be not behaving - but I tink there is a definite error | 12:17 |
| tobiash | in my case I'll try to avoid floating ips in my new deployment | 12:18 |
| kklimonda | yes, I still think this is something that should be handled in a more robust way, just saying that opencontrail may be the reason I'm the lucky guy to hit this :D | 12:18 |
| pabelanger | so, with zuulv3.o.o and hung web streaming links, I think the issue is we start stream when the job launches, but if zuul-merger is slow, or times out, it results in a web stream that doesn't work, until merging is finished. So, maybe we only setup webstream after we are done merging and actually running ansible | 12:20 |
| pabelanger | our we add queued / preparing / blue (web stream url starts) / red or green or failure | 12:20 |
| pabelanger | to the zuulv3.o.o status page for jobs | 12:21 |
| tobiash | pabelanger: does the scheduler get intermediate results from the executor? | 12:21 |
| tobiash | pabelanger: is this possible with gearman? | 12:21 |
| *** dkranz has joined #zuul | 12:22 | |
| mordred | tobiash: always avoid floating ips if possible - real ips are so much better ... :) | 12:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 12:23 |
| pabelanger | tobiash: in the case last night, no because we couldn't couldn't merge to changes (gerrit went offline) | 12:23 |
| tobiash | mordred: I'm planning to use only internal ips and placing zuul and nodepool into the same private network | 12:23 |
| mordred | ++ | 12:23 |
| tobiash | pabelanger: hrm, the executor sends the work data just before calling runPlaybooks | 12:27 |
| tobiash | pabelanger: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n711 | 12:27 |
| pabelanger | tobiash: is that went our webui changes links to stream.html? | 12:31 |
| pabelanger | when* | 12:31 |
| *** jhesketh_ has joined #zuul | 12:38 | |
| *** jhesketh has quit IRC | 12:43 | |
| tobiash | pabelanger: I think so, the executor tells the scheduler the build url | 12:54 |
| tobiash | pabelanger: and as soon as the build url is set the scheduler treats the job as running in the status.json | 12:54 |
| tobiash | pabelanger: but I didn't find the point yet where the executor starts serving the stream for the build | 12:55 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 12:58 |
| tobiash | pabelanger: maybe we just have to touch the job-output.txt before anouncing the url | 12:59 |
| dmsimard | jeblair, mordred: ^ squashed https://review.openstack.org/#/c/511821/ into https://review.openstack.org/509436/ as per requested, also adjusted the format to make it more parseable. | 12:59 |
| tobiash | pabelanger: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/lib/log_streamer.py?h=feature/zuulv3#n97 | 13:00 |
| mordred | dmsimard: I think we want to swap the depends-on | 13:02 |
| mordred | dmsimard: so that we can land the test - then when we land the debug it has a test to make sure it doesn't break | 13:02 |
| mordred | but it looks great otherwise | 13:02 |
| dmsimard | oh, sure | 13:02 |
| mordred | dmsimard: I abandoned the otherthing | 13:02 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 13:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header https://review.openstack.org/509436 | 13:03 |
| Shrews | can someone enlighten me as to why https://review.openstack.org/512637 is running the tox-py35 job when i've explicitly asked it not to with my branch matcher? | 13:06 |
| pabelanger | Shrews: http://logs.openstack.org/37/512637/14/check/tox-py35/acba5e6/zuul-info/inventory.yaml | 13:13 |
| pabelanger | says a variant is getting applied | 13:13 |
| pabelanger | but not sure why | 13:15 |
| Shrews | pabelanger: my .zuul.yaml is defining the variant, but it seems to be ignoring it | 13:15 |
| pabelanger | is openstack-python35-jobs-no-constraints template being used? | 13:15 |
| Shrews | yes | 13:15 |
| Shrews | that's where tox-py35 is coming from | 13:15 |
| pabelanger | yah, it seems like your variable of tox-py35 didn't apply | 13:17 |
| pabelanger | I would expect to see it in inventory | 13:17 |
| pabelanger | Hmm | 13:21 |
| pabelanger | it is possible that your .zuul.yaml is matching the branch matcher | 13:21 |
| pabelanger | then falls back to openstack-python35-jobs-no-constraints ? | 13:21 |
| pabelanger | no, that does seem likely | 13:22 |
| pabelanger | I am not sure, sounds like a jeblair question | 13:22 |
| pabelanger | now I am curious what the answer is | 13:22 |
| Shrews | i would expect templates to be processed first, then in-repo config | 13:22 |
| Shrews | i wonder if it's a bug with branch matcher logic when the branch references itself? like, i wonder if setting it non-voting would work? | 13:23 |
| pabelanger | yah, I would expect non-voting to work | 13:24 |
| pabelanger | I haven't done much testing / work with branch logic yet | 13:25 |
| mordred | tobiash: question on https://review.openstack.org/#/c/511858 | 13:42 |
| mordred | tobiash: tl;dr on both changes - should the default be to do upload-artifacts-to-logs-always and publish-artifacts-always with flags to make logs-only-on-change and artifacts-only-on-not-change? or the opposite | 13:44 |
| mordred | tobiash: now that you mention it - I can argue for either direction in my head :) | 13:44 |
| mordred | Shrews, pabelanger: I agree - I would expect that to not run | 13:47 |
| tobiash | mordred: hm both ways sound reasonable at first thought | 13:58 |
| tobiash | mordred: but as a user I would expect that if I put somthing in artifacts (I think of that as the api) it should be uploaded unless I force skip it | 14:00 |
| tobiash | mordred: skipping would then be an optimization to save space in certain conditions (although I think in this case the job should not put the artifacts there in the first place) | 14:02 |
| tobiash | mordred: this artifacts folder could be thought of like the git staging area | 14:03 |
| mordred | well - yes - I agree that the job shoulnd't put it there - I think the goal isn't specifically to skip uploading as much as to have a split-upload system - so artifacts are uploaded to logs for changes and artifacts server for post/release type jobs | 14:03 |
| mordred | so the 'I want these uploaded' is handled by putting them into artifacts/ - but where they are uploaded to is *hopefully* an admin decision | 14:03 |
| mordred | (if we do this correctly) | 14:03 |
| tobiash | ah, now I get your point | 14:04 |
| mordred | it's possible that a deployer might not have a logs/tarballs split like we do - in which case the distinction is extra complexity ... | 14:04 |
| tobiash | mordred: what do you think about having this decision earlier (in the base job?) which set_facts the destination? | 14:06 |
| mordred | tobiash: we'd have to persist those facts with a zuul_return ... but we could do that - what does that do for us? | 14:07 |
| tobiash | mordred: aka having an logs/artifacts dir and a publish/artifacts dir | 14:07 |
| mordred | interesting | 14:09 |
| mordred | tobiash: so - one of the things I was hoping to solve for on our side of things is not letting proposed patches overwrite published artifacts | 14:10 |
| mordred | tobiash: but instead to only upload those after a patch has merged (so we don't accidentally publish a nova-master.tar.gz that contains content thatnever actually landed) | 14:10 |
| mordred | that might be more of a concern for us than for other people though ... | 14:11 |
| tobiash | mordred: and you don't want to have a release job for this which adds the publish artifact on top? | 14:12 |
| tobiash | mordred: just thinking if it makes sense to just upload all artifacts to logs and do the publish decision based on a child job which has a publish post playbook | 14:13 |
| mordred | tobiash: ah - what I want to have actually is a release job that has a dependency - so that we can have a single job that builds artifacts regardless of whether it's check, gate, post or release ... | 14:14 |
| *** sambetts|afk is now known as sambetts | 14:14 | |
| mordred | tobiash: and then a 'release-openstack-tarball' job could run in release pipeline and download the artifact from tarballs and upload it to pypi | 14:14 |
| tobiash | mordred: sounds like a good idea to me | 14:15 |
| Shrews | mordred: good, so it's more than just one of us that's confused | 14:15 |
| mordred | tobiash: the amount of copy/paste at the moment between build and publish jobs got on my nerves :) | 14:15 |
| *** yolanda has joined #zuul | 14:16 | |
| tobiash | mordred: you could do this either with job dependency and download-upload dance or you could solve that with job inheritance and the release job just inherits from the package job | 14:17 |
| dmsimard | Wow, incoming in 2.5: https://user-images.githubusercontent.com/26403/31723399-59674f14-b3e4-11e7-9e25-0080ce5378a4.png | 14:17 |
| tobiash | mordred: where the download-upload dance would be more flexible and generic probably | 14:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:18 |
| tobiash | dmsimard: what's rescured? | 14:18 |
| dmsimard | tobiash: block/rescue/always | 14:18 |
| tobiash | ah, exception handling :) | 14:19 |
| mordred | tobiash: yah - I kind of like flexible/generic download/upload - the dance could ALSO be implemented potentially as a promotion too | 14:19 |
| odyssey4me | tobiash yeah, that's been there since 2.0, but pre 2.5 it reports failed tasks as failed, without reporting them as rescued too | 14:19 |
| odyssey4me | it's hella confusing | 14:19 |
| mordred | tobiash: like if we started using swift, we could have the job just tell swift to copy the object to the 'production' location - or we could have the job add a tag to a docker image in a registry or something | 14:19 |
| * mordred wave shands | 14:20 | |
| dmsimard | odyssey4me: yeah ara struggles with that too, there's no easy way to tell that a failure was rescued | 14:22 |
| dmsimard | odyssey4me: so it leads to false positives in playbook status result | 14:23 |
| tobiash | mordred: that sounds cool | 14:47 |
| jeblair | Shrews, pabelanger: yeah, i think if we don't want a job to run on certain branches, we need to avoid adding that job with no branch matcher in project-config, because that's going to add a variant that matches every branch. the best thing to do here may be to drop the python3-jobs template, and just add tox-py35 in-repo to feature/zuulv3. | 14:52 |
| dmsimard | Ansible 2.4.1 pushed back next week, they're spinning a new rc | 14:52 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Collect output from coverage job https://review.openstack.org/512910 | 14:52 |
| jeblair | Shrews, pabelanger: we can also drop the template and add tox-py35 in project-config. but either way, i think we need to drop the template. | 14:53 |
| jeblair | Shrews, pabelanger: once a project-pipeline has at least one job variant that matches that combination, the job is going to run. further variants with matchers can only *alter* how it runs. | 14:54 |
| jeblair | Shrews, pabelanger: so if we never want a job to run on master, we need to never have a variant without a branch matcher (implied or explicit) | 14:55 |
| pabelanger | okay cool! Good information to know | 15:03 |
| *** maxamillion has quit IRC | 15:44 | |
| *** maxamillion has joined #zuul | 16:15 | |
| Shrews | jeblair: thanks for explaining. submitted https://review.openstack.org/513092 to remove that template. <--- pabelanger | 16:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix branch ordering when loading in-repo config https://review.openstack.org/513094 | 16:34 |
| pabelanger | so, just thinking, is there any reason not to have projects.yaml live in opentack-zuul-jobs? Then we could have proposed a depends-on for that to confirm, right? But since it is trusted-project, we need to land it first? | 16:34 |
| pabelanger | or is my brain not working | 16:34 |
| jeblair | pabelanger: what's "projects.yaml" ? | 16:35 |
| jeblair | pabelanger: are you suggesting moving project definitions from project-config to ozj? | 16:36 |
| jeblair | pabelanger: project definitions for projects other than the current project have to be in a config-project | 16:37 |
| jeblair | Shrews: if you have a sec to look into "open finger connections from zuul-web can keep executor subprocesses around even after stopping zuul-executor" that would be helpful | 16:38 |
| pabelanger | jeblair: okay, ya, that makes sense. config-project can only hold that | 16:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Revert "Use weakref for change cache" https://review.openstack.org/513097 | 16:40 |
| pabelanger | mark that up to brain not working | 16:40 |
| *** bhavik1 has joined #zuul | 17:18 | |
| *** electrofelix has quit IRC | 17:41 | |
| *** sambetts is now known as sambetts|afk | 17:46 | |
| *** bhavik1 has quit IRC | 17:47 | |
| Shrews | jeblair: while comment on that "open finger connections" issue, i think i discovered a very valid reason those remain. | 18:07 |
| Shrews | jeblair: tl;dr The forked process is doing the streaming. Streaming doesn't end until the log is removed. Without the executor running to remove it, they linger. | 18:08 |
| Shrews | that's the current working theory, anyway | 18:09 |
| jeblair | Shrews: hrm, the executor should have stopped the job and removed the dir by that point | 18:11 |
| jeblair | these are, afaik, clean shutdowns... though maybe something went wrong with these specific directories | 18:11 |
| Shrews | jeblair: can we validate that? if so, that blows that theory out | 18:12 |
| jeblair | Shrews: we may be able to... we should log those exceptions. but i think we'll need to wait for this to happen again and then lsof the process to find its uuid, etc. we can also inspect the state of the filesystem then. | 18:16 |
| jeblair | at least we have a direction to look now | 18:16 |
| Shrews | jeblair: agreed. i'll continue looking at the code and pondering scenarios, but having something concrete to inspect would be helpful | 18:17 |
| clarkb | we were leaking build dirs on the executors on shutdown | 18:17 |
| clarkb | I think the older ones we attributed to keep setting but there were ones from thatday too | 18:18 |
| Shrews | oh, yeah, keep definitely affects streaming | 18:18 |
| *** weshay is now known as weshay|ruck|brb | 18:30 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 19:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 19:07 |
| *** weshay|ruck|brb is now known as weshay|ruck | 19:18 | |
| *** yolanda has quit IRC | 19:36 | |
| *** jkilpatr has quit IRC | 20:10 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** hasharAway has quit IRC | 20:18 | |
| *** openstackgerrit has joined #zuul | 20:19 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix branch ordering when loading in-repo config https://review.openstack.org/513094 | 20:19 |
| *** jkilpatr has joined #zuul | 20:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Revert "Use weakref for change cache" https://review.openstack.org/513097 | 20:26 |
| *** dkranz has quit IRC | 20:30 | |
| *** dkranz has joined #zuul | 20:34 | |
| *** dkranz has quit IRC | 20:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add management event queue length to status json https://review.openstack.org/513182 | 21:00 |
| *** yolanda has joined #zuul | 21:20 | |
| *** openstackgerrit has quit IRC | 21:48 | |
| *** bstinson has quit IRC | 22:12 | |
| *** openstackgerrit has joined #zuul | 22:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Merge tenant reconfiguration events https://review.openstack.org/513195 | 22:13 |
| *** bstinson has joined #zuul | 22:19 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 22:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Merge tenant reconfiguration events https://review.openstack.org/513195 | 22:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 22:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 22:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Disable tox-siblings https://review.openstack.org/513205 | 23:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add _projects to convert project list to dictionary https://review.openstack.org/512868 | 23:17 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Fix error in job parser https://review.openstack.org/513201 | 23:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Give layout objects a unique ID https://review.openstack.org/513207 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
