| *** gothicmindfood has quit IRC | 00:47 | |
| leifmadsen | yea, until I can figure out how best to deal with logs, that'll have to do | 00:48 |
|---|---|---|
| *** gothicmindfood has joined #zuul | 00:50 | |
| SpamapS | You can turn that on and off at runtime too yeah? | 01:06 |
| * SpamapS has done that with `zuul-executor keep` and `zuul-executor nokeep` | 01:06 | |
| *** JasonCL has quit IRC | 01:07 | |
| *** JasonCL has joined #zuul | 01:19 | |
| *** threestrands has joined #zuul | 01:55 | |
| *** JasonCL has quit IRC | 01:56 | |
| *** jkilpatr has quit IRC | 01:58 | |
| *** JasonCL has joined #zuul | 02:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: license: remove dos line break https://review.openstack.org/531057 | 02:17 |
| *** rlandy|bbl is now known as rlandy | 02:20 | |
| *** rlandy has quit IRC | 02:23 | |
| *** JasonCL has quit IRC | 02:36 | |
| *** JasonCL has joined #zuul | 02:56 | |
| SpamapS | Oy | 03:16 |
| SpamapS | if you get something like this, you don't get a job-output.json | 03:16 |
| SpamapS | 2018-01-03 19:24:43.553246 | UNEXPECTED ANSIBLE ERROR | 03:16 |
| SpamapS | 2018-01-03 19:24:43.553297 | ERROR! Unexpected Exception: list index out of range | 03:16 |
| SpamapS | looks like the log streamer failed | 03:17 |
| jhesketh | clarkb: sorry I didn't notice your ping re pidfiles, I did however see the change merge and it looked good to me | 03:26 |
| *** jkilpatr has joined #zuul | 03:26 | |
| clarkb | jhesketh: thanks for looking, seems to work on ze04 at least | 03:30 |
| *** jkilpatr has quit IRC | 04:37 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip \r from build UUID in fingergw https://review.openstack.org/530973 | 07:10 |
| *** threestrands has quit IRC | 07:25 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:28 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:00 | |
| *** toabctl has quit IRC | 10:16 | |
| *** toabctl has joined #zuul | 10:23 | |
| *** threestrands has joined #zuul | 10:51 | |
| *** threestrands has quit IRC | 10:51 | |
| *** threestrands has joined #zuul | 10:51 | |
| *** threestrands has quit IRC | 11:10 | |
| *** openstack has quit IRC | 11:28 | |
| *** openstack has joined #zuul | 13:10 | |
| *** ChanServ sets mode: +o openstack | 13:10 | |
| *** rlandy has joined #zuul | 13:30 | |
| Shrews | w00t: finger 648c95821b0a4c25adb79d20b2a7ab25@zuulv3.openstack.org | 13:46 |
| Shrews | mordred: did you see sivel's question for us on https://github.com/ansible/ansible/pull/34420#issuecomment-355165847 ? | 13:53 |
| Shrews | i'm not quite sure what he's talking about with "duplicate callbacks" | 13:53 |
| Shrews | leifmadsen: just fyi, turning on keep will make log streaming threads hang around forever b/c they depend on the log file getting removed to stop streaming | 13:57 |
| leifmadsen | that's fine, in this case it's just for testing/docs | 13:58 |
| leifmadsen | but too good to note, I'll try and add that | 13:58 |
| Shrews | i figured. just wanted to mention it | 13:58 |
| leifmadsen | yea, that's good info | 13:58 |
| leifmadsen | thx much | 13:58 |
| *** sshnaidm|afk is now known as sshnaidm|mtg | 13:58 | |
| mordred | Shrews: I didn't - looking now | 14:15 |
| mordred | leifmadsen: one of our near-term todo list items is a base job (or a set of potential base jobs) that can be used by folks for easy getting started | 14:16 |
| leifmadsen | mordred: yea, at this point, that's what is next in the documentation | 14:16 |
| mordred | woot! | 14:16 |
| leifmadsen | I'm not going to document how to do anything post-install | 14:16 |
| leifmadsen | I leave that to the experts :) | 14:16 |
| leifmadsen | so don't w00t too hard heh | 14:16 |
| leifmadsen | but I now have a process to get PRs on GitHub triggering a Hello World! via nodepool --> openstack | 14:17 |
| leifmadsen | and reporting a positive test | 14:17 |
| leifmadsen | at this point, I just need to move it into something a bit cleaner, and integrated into the existing docs to replace the non-existent quickstart | 14:18 |
| leifmadsen | I was talking to pabelanger about maybe doing a "real" quickstart that would really just use a periodic job, and run something useless on the executor so that you didn't need all the peripherals | 14:18 |
| leifmadsen | but I think I'll leave that as another exercise. For now, I'm calling this Zuul From Scratch | 14:19 |
| leifmadsen | since I don't consider it a "quickstart" | 14:19 |
| leifmadsen | gotta run off to chiro, back in an hour-ish | 14:19 |
| mordred | Shrews: thanks - replied to sivel | 14:35 |
| mordred | Shrews: maybe we should figure out how to write a functional test for ansible itself that exercises our subclass-an-action-plugin usecase | 14:35 |
| mordred | leifmadsen: awesome. I was actually noodling that it would be fun to write a "Zuul the Hard Way" - referencing the kelseyhightower book "Kubernetes the Hard Way" which is his documentation/explanation of how to get k8s up and going all through manual steps and not just using existing automation | 14:37 |
| *** dkranz has quit IRC | 14:38 | |
| mordred | leifmadsen: also - the periodic job idea is a good one - I've got a slide here: http://inaugust.com/talks/test-it-like-you-deploy-it.html#/terminoloy | 14:38 |
| *** dkranz has joined #zuul | 14:38 | |
| mordred | leifmadsen: which isn't a great slide, but it's similar to a set of slides I have somewhere harder to find where I basically narrate through the four of those in order and discuss them as stages or phases of CI | 14:38 |
| mordred | leifmadsen: or as the steps people work through when starting from no CI - and honestly that the industry went through | 14:39 |
| mordred | leifmadsen: having automated nightly regression tests is revolutionary when you have nothing | 14:40 |
| mordred | leifmadsen: then instead of nightly running post-commit and using the nvie model to have a devel branch that you merge back to master when it's CI jobs go green | 14:41 |
| mordred | leifmadsen: then instead of that running tests on proposed changes like travis and the jenkins github PR plugin do | 14:41 |
| mordred | leifmadsen: finally reaching running gating jobs between code review approval and the patch landing like zuul does it | 14:41 |
| mordred | leifmadsen: so in some ways structuring a getting started guide through a similar journey might be a good way to think about it | 14:42 |
| mordred | otoh - people are pretty used to travis and friends, so running tests on github PRs is actually most likely to be the thing that people are going to be wanting to solve, so maybe jumping straight to that and treating post and periodic as later topics would point people towards thinking about the problem in terms of testing the patches they're considering landing instead of testing the state of a given branch | 14:44 |
| *** yolanda has quit IRC | 14:45 | |
| *** yolanda has joined #zuul | 14:46 | |
| rcarrillocruz | ++ | 14:49 |
| rcarrillocruz | number one question we got on the windmill workshop was 'what zuul brings that jenkins does not' | 14:50 |
| rcarrillocruz | i think explaining where testing comes from to what zuul does is better explanation to people than just dumping 'it does speculative merging blah' | 14:51 |
| rcarrillocruz | cos i saw quite a few poker faces when i tried to explain speculative merging | 14:51 |
| rcarrillocruz | :/ | 14:51 |
| gundalow | +1 to explaining things in terms of "this is how it's different to other things" | 14:53 |
| mordred | rcarrillocruz, gundalow: ++ | 14:54 |
| *** jkilpatr has joined #zuul | 15:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Rework log streaming to use logging https://review.openstack.org/531171 | 15:10 |
| mordred | Shrews: ^^ that doesn't even PARTIALLY work ... I'm pushing it up because it's basically got some POC code from me that I was poking at and I think might be enough for someone else to run with it if I don't get to it first | 15:11 |
| mordred | Shrews: pinging you because, well, at this point you're already neck-deep in log streaming :) | 15:11 |
| mordred | dmsimard: ^^ you might also find that interesting | 15:11 |
| leifmadsen | mordred: let me know if you wanna write a book together :D | 15:11 |
| mordred | leifmadsen: :) | 15:12 |
| mordred | leifmadsen: have you seen how verbose I am? | 15:12 |
| leifmadsen | mordred: yep, and I can tone that down. Always better to have more content than not enough | 15:12 |
| mordred | you say that now ... | 15:12 |
| leifmadsen | I've done it before :D | 15:13 |
| corvus | i think we can reduce the length of the 'quickstart' docs by about 30% with improvements to zuul as well -- so expect some more of that before we release | 15:18 |
| tristanC | leifmadsen: also note that nodepool is coming to fedora, perhaps the quickstart could use dnf instead of git/pip? | 15:19 |
| corvus | leifmadsen: fungi has been working on getting the generic base job defined. it exists now here: http://git.openstack.org/cgit/openstack-infra/zuul-base-jobs | 15:19 |
| tristanC | + the package takes care of the systemd service and logrotate bits too | 15:19 |
| leifmadsen | certainly would simplify things once it's available | 15:19 |
| leifmadsen | I'm all for a simple install :D | 15:21 |
| leifmadsen | corvus: yea I was looking through some of that last night. I was trying to figure out how much some of that much change when using a cloud image (vs a diskimage build) | 15:21 |
| corvus | leifmadsen: it should work with either | 15:21 |
| leifmadsen | but it was late, so I didn't get too far into ti | 15:21 |
| leifmadsen | awesome | 15:21 |
| corvus | (and if not, is a bug) | 15:21 |
| leifmadsen | +1 | 15:22 |
| fungi | leifmadsen: corvus: should next steps on teh zuul-base-jobs repo be to add better readmes for the playbooks and then update the zuul user guide section which refers to the base job provided in the standard library? | 15:24 |
| leifmadsen | yes :) | 15:24 |
| leifmadsen | it would be good to know how best to consume them, what they do, etc | 15:25 |
| fungi | cool. i mean, there's basically only one, and a copy of that one you can use to test changing it without changing it | 15:25 |
| leifmadsen | haha | 15:29 |
| leifmadsen | there's a yo dawg joke in there | 15:29 |
| Shrews | mordred: cool. that could probably just use CustomThreadingTCPServer from lib/streamer_utils.py | 15:33 |
| Shrews | mordred: that would get you ipv6 support, address reuse, fixes extraneous threads in tests, etc. | 15:37 |
| rcarrillocruz | hey folks, noob question: is there any attendance list for PTG? Like, I'm going to send an email to my manager to try to get approval to attend, but unsure if I should put myself on some etherpad as tentative or something | 15:40 |
| rcarrillocruz | never been to a PTG, just summits | 15:40 |
| mordred | Shrews: sweet | 15:42 |
| mordred | rcarrillocruz: not that I know of | 15:42 |
| rcarrillocruz | k | 15:43 |
| rcarrillocruz | thx | 15:43 |
| corvus | rcarrillocruz: you do need to register for it (and there's a $100 fee) | 15:44 |
| corvus | https://www.openstack.org/ptg/ | 15:45 |
| rcarrillocruz | ah, let me google my inbox | 15:45 |
| rcarrillocruz | thx | 15:45 |
| rcarrillocruz | had that page loaded, did not click on register, see on eventbrite the 100 fee now | 15:45 |
| *** jkilpatr has quit IRC | 15:49 | |
| *** jkilpatr has joined #zuul | 15:49 | |
| *** jkilpatr has quit IRC | 15:56 | |
| *** jkilpatr has joined #zuul | 16:11 | |
| Shrews | Anyone have any thoughts on why test_reconfigure_window_fixed might fail under Ansible 2.4? http://logs.openstack.org/09/531009/2/check/tox-py35/39475c5/testr_results.html.gz | 16:22 |
| Shrews | hrm... KeyError: 'job2' | 16:25 |
| *** jkilpatr has quit IRC | 16:26 | |
| *** jkilpatr has joined #zuul | 16:29 | |
| corvus | Shrews: the change in run times has probably exposed a race; that may be a legit bug. | 16:31 |
| *** jkilpatr has quit IRC | 16:32 | |
| *** jkilpatr has joined #zuul | 16:32 | |
| corvus | Shrews: we probably just need a safety check in BuildSet.removeBuild to not remove it twice | 16:33 |
| corvus | the reconfiguration removed it already, but then the result ("ABORTED") came back from the executor and it tried te remove it again | 16:36 |
| Shrews | corvus: does removeBuild() just need this then? if build.job.name not in self.builds: return | 16:47 |
| corvus | Shrews: yeah, i think so | 16:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Allow Ansible 2.4 https://review.openstack.org/531009 | 16:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Don't remove builds that don't exist https://review.openstack.org/531198 | 16:52 |
| Shrews | not sure if that should occur before or after the tries increment | 16:55 |
| Shrews | corvus: ^^^ | 16:55 |
| *** sshnaidm|mtg is now known as sshnaidm | 17:51 | |
| corvus | Shrews: i think that order | 18:30 |
| corvus | i've been thinking about memory usage, and poking at some experiments to see what i can learn locally. because of its internal allocator, python doesn't ever really release memory, but it reuses it after objects are garbage collected. | 18:32 |
| corvus | so it's sort of expected that zuul's memory usage will never decrease, however, it shouldn't continually increase, unless people throw a lot of config changes at it | 18:32 |
| corvus | i've verified that performing a reconfiguration and then uploading a bunch of dynamic config changes increases the memory as you'd expect, but then repeating that uses less memory the next time as the previous layouts have been garbage collected | 18:33 |
| dmsimard | at what point, if any, does nodepool delete alien images ? | 18:34 |
| dmsimard | # nodepool alien-image-list |wc -l | 18:34 |
| corvus | so if there's a leak, it's not anything obvious. however, if there is an object reference leak which leads to a layout object, it can result in a lot of extra memory sticking around. this is really hard to debug without inspecting a running system. | 18:34 |
| dmsimard | 257 | 18:34 |
| corvus | dmsimard: never | 18:35 |
| corvus | a separate, but related problem is the sheer amount of memory that a single dynamic config takes | 18:35 |
| dmsimard | corvus: thanks | 18:35 |
| corvus | it's pretty easy for openstack's zuul to exhaust its memory with only a handful of dynamic config updates. | 18:36 |
| corvus | i believe we can rework the configuration loader to re-use more of the final, parsed objects when performing dynamic updates. however, it's a substantial change in approach. it would probably take a few days to accomplish. | 18:37 |
| corvus | i think that will result in significant memory savings, though i can't quantify that yet without trying it. also, it's likely that performance will vary across repos -- a change to openstack-zuul-jobs would still incur a significant memory penalty. but a change to a more typical project shouldn't. | 18:38 |
| corvus | i think it's worth doing, though i think i will finish up the cross-source work first before i dig into that, just to have a clear head. | 18:39 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move revoke-sudo from unittest pre to job run https://review.openstack.org/529662 | 18:42 |
| clarkb | corvus: is it just that the size of the resulting dicts and lists and so on after deserializing is quite large in aggregate? | 18:43 |
| corvus | clarkb: yes, very large | 18:43 |
| corvus | clarkb: the stripped down version of our config i'm using for this test represents a 585MB increase in rss | 18:45 |
| clarkb | wow | 18:45 |
| corvus | er, it may be more like 200MB.... the documentation on units is... lacking. | 18:47 |
| corvus | still. not small. | 18:47 |
| clarkb | corvus: isn't it great when you get to guess if something is in bits or bytes and kilo/mega/giga etc | 18:47 |
| corvus | or pages :| | 18:48 |
| rcarrillocruz | WOOTZ, got greenlight on PTG budget | 19:02 |
| * rcarrillocruz dances | 19:02 | |
| * rcarrillocruz goes check hotels and stuff | 19:03 | |
| rcarrillocruz | i assume you folks stay at the same with the block reserved on the PTG web page ? | 19:03 |
| clarkb | rcarrillocruz: fungi and I will be in the ptg hotel block yes | 19:03 |
| fungi | confirmed | 19:04 |
| rcarrillocruz | k | 19:04 |
| clarkb | I think mordred tends to find a "real" hotel :) | 19:04 |
| rcarrillocruz | haha, yeah, i asked in general :-) | 19:05 |
| jlk | gotta get them starwoods | 19:05 |
| Shrews | dem sweet sweet starwood points | 19:10 |
| *** qwc has quit IRC | 19:17 | |
| *** qwc has joined #zuul | 19:27 | |
| *** JasonCL has quit IRC | 19:38 | |
| *** JasonCL has joined #zuul | 19:39 | |
| clarkb | corvus: I added a couple of zuulv3.0 (not just v3) tagged bugs from the zuulv3-issues etherpad (secrets and branches related) | 19:51 |
| clarkb | just a heads up in case you disagree that is a release blocker | 19:52 |
| *** sshnaidm is now known as sshnaidm|off | 19:53 | |
| fungi | Shrews: i feel like you're referencing the countdown episode of the it crowd | 19:55 |
| corvus | clarkb: thanks; i'll probably go over the list next week | 19:57 |
| Shrews | fungi: coincidentally, i'm wearing a lovely tnetennba today | 19:57 |
| fungi | christine frequently wears a tee shirt with that on it | 20:02 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't remove builds that don't exist https://review.openstack.org/531198 | 20:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix pep8/yamllint errors https://review.openstack.org/530685 | 20:53 |
| *** threestrands has joined #zuul | 21:35 | |
| *** dmellado has quit IRC | 21:43 | |
| *** dmellado has joined #zuul | 21:46 | |
| *** JasonCL has quit IRC | 21:49 | |
| *** JasonCL has joined #zuul | 21:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 21:52 |
| clarkb | corvus: Shrews is the "open finger connections from zuul-web can keep executor subprocesses around even after stopping zuul-executor" item on zuulv3-issues not a bug? looks like corvus put it on the etherpad the nsomeone else (shrews?) says it is expected behavior | 21:56 |
| corvus | iff it's only caused when --keep is set, then it's a minor bug that is complicated to fix and probably doesn't need to be a blocker. if it happens without --keep it's a major bug. | 22:01 |
| clarkb | ok I'll file one and put that info in for further debugging | 22:01 |
| corvus | i think Shrews believes we only saw it in conjunction with keep. i can't confirm or disprove that at this point. | 22:02 |
| Shrews | clarkb: it's no longer an issue b/c we've gone from the fork model to thread model | 22:02 |
| clarkb | Shrews: gotcha so if the zuul-web process goes away there are no children to keep serving those files? | 22:02 |
| corvus | Shrews: oh was that it? i guess i was confusing it with a different bug? | 22:02 |
| clarkb | thats what asyncio gives us? | 22:02 |
| Shrews | asyncio is not threaded. we use socketserver for that | 22:03 |
| corvus | clarkb: your description was about zuul-executor | 22:03 |
| corvus | (zuul-web was just the client which was holding the connections open -- the issue was on the executor log streamer side) | 22:03 |
| Shrews | because we used to fork a new process to handle finger requests, those were remaining around even when the executor exited (iirc) | 22:04 |
| *** JasonCL has quit IRC | 22:04 | |
| clarkb | corvus: oh right zuul web is the front end making connection to the executor backend | 22:04 |
| clarkb | in any case if we aren't forking a process per connection then we should be fine | 22:05 |
| *** JasonCL has joined #zuul | 22:05 | |
| Shrews | the keep thing is separate and is not a bug | 22:05 |
| corvus | well, it's a bug :) | 22:06 |
| Shrews | we just haven't found a good way to terminate streaming unless the log file disappears | 22:06 |
| corvus | it's -- to borrow a phrase from intel -- working as designed. :) | 22:06 |
| corvus | it's both of those things at the same time :) | 22:06 |
| clarkb | corvus: also all other log streaming tools work this way too :) | 22:06 |
| clarkb | I'll file a bug for "Cannot currently terminate log streaming until the log file being streamed is deleted" | 22:07 |
| clarkb | is ^ accurate? | 22:07 |
| corvus | clarkb: that's the thing that only happens if you set keep. | 22:07 |
| corvus | so it might be better described as "log streaming may not be terminated correctly if keep is set" | 22:07 |
| clarkb | thanks | 22:08 |
| *** JasonCL has quit IRC | 22:10 | |
| corvus | i need a word that succinctly means reverse dependencies (ie, "needed by") | 22:17 |
| clarkb | parentof? | 22:19 |
| corvus | clarkb: well, in this case it'd be 'childof'. but that makes me think of git and i'm focused on the depends-on headers | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 22:28 |
| clarkb | successor/predeccesor? | 22:29 |
| corvus | oh that's a nice pair | 22:37 |
| *** JasonCL has joined #zuul | 22:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!