| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 00:01 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 00:01 |
| SpamapS | pabelanger: you just need to get our ansible plugins loaded when you run the check-syntax | 00:24 |
| pabelanger | SpamapS: right, that's what I plan on doing in the morning. | 00:24 |
| SpamapS | I wonder if we could go further up the chain than zuul-bwrap and also have a zuul-ansible-playbook | 00:24 |
| SpamapS | that would just setup an ansible dir and a nice empty job dir for testing (with a statically created inventory maybe) | 00:25 |
| SpamapS | would make iterating on playbooks quite nice actually | 00:25 |
| SpamapS | I too am ready to EOD though :) | 00:25 |
| pabelanger | maybe zuul_console should just be a playbook / role, external to zuul git repo. I don't think we have any zuul bits in it | 00:26 |
| pabelanger | would have to think about that played as trusted / untrusted repos | 00:26 |
| mordred | SpamapS: that's actually kind of on my todo list - the zuul-ansible-playbook command | 00:32 |
| mordred | although I won't get upset if someon ebeats me to it | 00:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop printing shell commands if we have a task name https://review.openstack.org/484995 | 00:45 |
| mordred | jeblair:, pabelanger, SpamapS: ^^ | 00:48 |
| mordred | http://paste.openstack.org/show/615792/ is output from it | 00:51 |
| *** harlowja has quit IRC | 01:13 | |
| tristanC | mordred: re javascript bundling, there is also the python xstatic format that could be used instead of r.js/bower/.. | 02:00 |
| mordred | tristanC: yah - I would like to avoid that if we can - packaging javascript in python packages to avoid javascript tools doesn't seem super-scalable long term to me | 02:06 |
| mordred | tristanC: however, you'll be happy to know that grunt andbower and no longer the right cools to use :) | 02:06 |
| mordred | (I did a bunchof learning over the weekend) | 02:06 |
| mordred | tristanC: 'yarn' and 'webpack' seem to be the happymakers now - and yarn even has distro packages available | 02:08 |
| tristanC | as long as it's not Makefile written in javascript... :-) | 02:11 |
| mordred | tristanC: this I whole-heartedly agree with :) | 02:22 |
| *** harlowja has joined #zuul | 04:31 | |
| *** harlowja has quit IRC | 04:47 | |
| tobiash | jeblair: I have a comment at https://review.openstack.org/#/c/484962 | 05:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.d configuration split documentation https://review.openstack.org/482405 | 05:30 |
| tobiash | mordred: commented at https://review.openstack.org/#/c/484992 | 05:38 |
| *** harlowja has joined #zuul | 05:46 | |
| *** bhavik1 has joined #zuul | 06:05 | |
| *** isaacb_ has joined #zuul | 06:23 | |
| *** harlowja has quit IRC | 06:42 | |
| *** amoralej|off is now known as amoralej | 06:44 | |
| *** isaacb_ has quit IRC | 07:04 | |
| *** isaacb_ has joined #zuul | 07:08 | |
| *** hashar has joined #zuul | 07:19 | |
| *** hashar has quit IRC | 07:19 | |
| *** hashar has joined #zuul | 07:19 | |
| *** bhavik1 has quit IRC | 07:41 | |
| *** isaacb_ has quit IRC | 07:48 | |
| *** isaacb_ has joined #zuul | 07:58 | |
| *** isaacb_ has quit IRC | 07:59 | |
| mordred | tobiash: yes - thank you - my synthetic test locally it seems was not good enough :) | 08:17 |
| mordred | tobiash: ok. I have duplicated enough things that I now have the same failure | 08:26 |
| *** xinliang has quit IRC | 08:28 | |
| *** xinliang has joined #zuul | 08:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 08:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 08:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 08:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop printing shell commands if we have a task name https://review.openstack.org/484995 | 08:36 |
| mordred | tobiash: that should fix the first two - and when I wake up from my next nap, I'm going to refactor the shared code into a library location too | 08:36 |
| xinliang | tobiash: ping | 08:47 |
| *** openstackgerrit has quit IRC | 08:49 | |
| *** isaacb has joined #zuul | 08:49 | |
| *** isaacb has quit IRC | 08:58 | |
| *** isaacb has joined #zuul | 09:04 | |
| *** isaacb has quit IRC | 10:17 | |
| *** isaacb_ has joined #zuul | 10:17 | |
| *** hashar is now known as hasharLunch | 10:27 | |
| *** jkilpatr has quit IRC | 10:36 | |
| tobiash | mordred: +2 | 10:45 |
| tobiash | mordred: jetlag? ;) | 10:46 |
| tobiash | xinliang: what's up? | 10:46 |
| tobiash | mordred: just tried your patches now and they seem to work now :) | 10:50 |
| tobiash | mordred: http://paste.openstack.org/show/615826/ | 10:50 |
| *** isaacb_ has quit IRC | 11:05 | |
| tobiash | mordred: just a nit I see in the logs is that there is no empty line between the playbooks | 11:08 |
| *** jkilpatr has joined #zuul | 11:11 | |
| pabelanger | morning | 11:21 |
| tobiash | hi | 11:21 |
| *** hashar has joined #zuul | 11:39 | |
| *** isaacb_ has joined #zuul | 12:06 | |
| *** openstackgerrit has joined #zuul | 12:42 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 12:42 |
| tobiash | mordred: ^^^ is my try to combine our two approaches for web streaming | 12:43 |
| tobiash | mordred: works fine in my deployment | 12:43 |
| tobiash | mordred: not sure yet if it works with relative websocket urls (I guess not due to missing wss://) | 12:43 |
| *** dkranz has joined #zuul | 12:44 | |
| *** isaacb_ has quit IRC | 12:44 | |
| tobiash | mordred: ok, tested... relative urls are invalid... | 12:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:01 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:02 |
| tobiash | mordred: ^^^ this should also solve the (default) relative websocket url case | 13:03 |
| *** isaacb has joined #zuul | 13:09 | |
| *** amoralej is now known as amoralej|lunch | 13:11 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul.(yaml|d) loading order and add tests https://review.openstack.org/482411 | 13:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:40 |
| tobiash | mordred: now with (hopefully) green tests | 13:40 |
| *** isaacb has quit IRC | 13:44 | |
| tobiash | pep8 gods don't like me... | 13:53 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 13:53 |
| pabelanger | tobiash: almost as fun as the tetris god | 13:56 |
| tobiash | lol | 13:56 |
| *** isaacb has joined #zuul | 14:02 | |
| *** amoralej|lunch is now known as amoralej | 14:17 | |
| *** isaacb has quit IRC | 14:48 | |
| *** isaacb has joined #zuul | 14:49 | |
| jeblair | Shrews: 484414 looks good with a minor nit | 15:43 |
| Shrews | jeblair: oh, oops. that shouldn't be in there. you prefer a follow up, or a new PS? | 15:44 |
| jeblair | Shrews: i'm the only reviewer with a vote right now, may as well revise it? | 15:44 |
| jeblair | i'm happy to re+2 | 15:44 |
| Shrews | ack | 15:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Support UUID as builder identifier https://review.openstack.org/484414 | 15:46 |
| *** isaacb has quit IRC | 15:54 | |
| mordred | tobiash: zomg - that looks great | 15:54 |
| mordred | jeblair: so - the apache config on zuulv3.o.o dind't update | 15:54 |
| jeblair | bummer | 15:55 |
| mordred | jeblair: well - I'm confused, because the puppet patch that added zuul-web also added the apache config update | 15:59 |
| jeblair | mordred: yeah, i'm looking and i'm not seeing the problem yet | 15:59 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add html based websocket client for console stream https://review.openstack.org/483503 | 16:05 |
| jeblair | mordred: that is super weird | 16:06 |
| jeblair | Jul 18 16:59:13 zuulv3 puppet-user[20412]: (/Stage[main]/Zuul/Httpd::Mod[proxy_wstunnel]/Httpd_mod[proxy_wstunnel]/ensure) created | 16:06 |
| pabelanger | mordred: tobiash: is there a screenshot of UI some place? | 16:06 |
| jeblair | i think that's the puppet run that should have done it, but no mention of the config | 16:06 |
| jeblair | mordred: i kind of want to rm the file and kick puppet | 16:07 |
| jeblair | maybe we should switch this to -infra | 16:07 |
| tobiash | :) | 16:07 |
| tobiash | pabelanger: you mean from the live streaming log? | 16:08 |
| pabelanger | tobiash: ya, the html based client | 16:08 |
| Shrews | pabelanger: if we restart zuul we should be able to see for ourselves, yeah? | 16:09 |
| Shrews | i'm very excited about seeing it too | 16:09 |
| tobiash | pabelanger: not yet, but it just looks like a black terminal with a small autoscroll checkbox on the top right | 16:10 |
| pabelanger | Shrews: actually, I think we did restart last night | 16:11 |
| *** isaacb has joined #zuul | 16:11 | |
| Shrews | pabelanger: but the html serving stuff *just* merged | 16:12 |
| Shrews | 483503 | 16:12 |
| Shrews | should be clickable links on the status page now | 16:13 |
| jeblair | mordred: please let me know when you have a few moments to chat about https://review.openstack.org/484992 | 16:16 |
| tobiash | Shrews: hopefully it works out of the box... took me a while to figure out the correct apache config | 16:18 |
| pabelanger | Oh, Isee. we need to run zuul-web to server up static page | 16:18 |
| tobiash | Shrews: but I currently also have a frankensteined apache config in this regard | 16:19 |
| tobiash | Shrews: static ui on host, status.json in scheduler container, websocket client and service in zuul-web container | 16:20 |
| * tobiash thinks serving all stuff (except or including status.json?) from zuul-web would simplify this drastically | 16:21 | |
| jeblair | yeah, i think we'll get there before we release 3.0 | 16:22 |
| * mordred has a write up on that to clean up and send out ... | 16:24 | |
| mordred | jeblair: sup? | 16:24 |
| jeblair | mordred: i'm concerned about the fragility of the playbook path stripping. it's broken once already before the change even landed. :) | 16:25 |
| jeblair | mordred: let's agree on what we want to see there, and then find the best way to achieve it | 16:26 |
| mordred | jeblair: kk. (and yes, me too) | 16:26 |
| jeblair | mordred: what do you want it to look like? can you splat an example? | 16:26 |
| mordred | jeblair: so - I know I _definitely_ don't want to see /tmp/{long-string} - cause that's not generally interesting | 16:26 |
| jeblair | mordred: agreed | 16:27 |
| mordred | jeblair: I think 2017-07-19 17:35:32.069389 | PRE-RUN [git.openstack.org/openstack-infra/project-config/playbooks/base/test] | 16:27 |
| mordred | jeblair: would be nice - that shows info that should make sense to all the users and can be used to track down the source of the playbook | 16:28 |
| jeblair | mordred: okay. that omits two pieces of information which may be useful: a) whether the playbook was trusted, and b) the branch | 16:28 |
| jeblair | (to be fair, the current thing only gives us a; we've never had b in the v3 job output. but we do have b in a lot of v2 output) | 16:29 |
| jeblair | mordred: can i convince you that one or both of those things are worth adding, in some way, to the output? (not nececarily with path components, but maybe in other ways) | 16:30 |
| mordred | jeblair: we could pass all three as variables and then have PRE-RUN [trusted : git.openstack.org/openstack-infra/project-config/playbooks/base/test : master] | 16:30 |
| mordred | jeblair: totally! I tihnk those are very useful pieces of info | 16:31 |
| mordred | or maybe: PRE-RUN [ git.openstack.org/openstack-infra/project-config/playbooks/base/test : trusted : master ] ("what are we running and what are its qualities") | 16:31 |
| jeblair | mordred: cool. we might consider "PRE-RUN [trusted : git.openstack.org/openstack-infra/project-config/playbooks/base/test@master]" which matches the way zuul represents repo+branch combinations elsewhere (like in error messages) | 16:32 |
| mordred | jeblair: ++ | 16:32 |
| jeblair | mordred: or if you like the other form better, we can change the other stuff to match. either way | 16:32 |
| mordred | nah - I'm cool witih @master | 16:32 |
| jeblair | mordred: okay, so let's pass all of that with dedicated variables | 16:33 |
| jeblair | mordred: that implicitly solves the fragility concern | 16:33 |
| jeblair | mordred: and i think we should probably do this before landing your series, because i think some of the path manipulations there are going to break on some playbooks. | 16:34 |
| mordred | jeblair: mind if I do a followup? I need to make that change in two places and I think I'd rather just make the patch to extract that code into a library file real quick then make the change in once place | 16:34 |
| mordred | (we an wait til we'rehappy with the followup to land the first one) | 16:35 |
| jeblair | mordred: sure -- as long as land them together | 16:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Default tox to python3 https://review.openstack.org/485272 | 16:36 |
| jeblair | mordred: lemme add the branch info to the jobdirplaybook | 16:37 |
| mordred | jeblair: its in there | 16:37 |
| mordred | jeblair: self.trusted_project_index[(canonical_name, branch)] = root | 16:37 |
| jeblair | mordred: that's only for trusted though | 16:38 |
| mordred | ah | 16:38 |
| mordred | yes | 16:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add branch to JobDirPlaybook https://review.openstack.org/485273 | 16:42 |
| jeblair | mordred: ^ i thought there might be more to it than that. :) | 16:43 |
| mordred | jeblair: while I've got yo ulooking at that ... | 16:44 |
| mordred | jeblair: canyou add canonical_name too? | 16:44 |
| jeblair | ya | 16:44 |
| mordred | thanks! (and it's a JobDirPlaybook that gets passed as playbok to runAnsiblePlaybook right? | 16:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 16:46 |
| jeblair | mordred: yep, so you should be able to pull everything off of there | 16:46 |
| mordred | jeblair: I kinda want to write out a vars file with a zuul_execution dict - the list of -e vars is getting long :) | 16:50 |
| jeblair | mordred: wfm. want to go ahead and switch it to zuul.execution ? | 16:51 |
| jeblair | i don't feel strongly about that. :) | 16:52 |
| *** hashar has quit IRC | 16:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 16:53 |
| pabelanger | mordred: jeblair: so, thinking out loud here, but what do you think about adding zuul_console into zuul-jobs in the library directory? Looking at the action plugin, I don't see any zuul specific code. It would be related to ^ | 16:53 |
| *** dkranz has quit IRC | 16:53 | |
| jeblair | pabelanger: would that then get used by jobs running from zuul-jobs? | 16:55 |
| mordred | jeblair: well, we can't do zuu.execution, because we have zuul dict in inventory and we don't have dict merging turned on | 16:56 |
| mordred | jeblair: so it would have to be a different top-levle | 16:56 |
| mordred | jeblair: I think for now let's just pass them as we're doing and we can swing back and clean up later | 16:56 |
| jeblair | mordred: ok. | 16:56 |
| pabelanger | jeblair: I think it would be blocked today, since it is untrusted right? | 16:56 |
| jeblair | pabelanger: i don't think we block modules | 16:56 |
| pabelanger | ah | 16:56 |
| pabelanger | so, maybe we want to run it? It would make iterations easier | 16:57 |
| jeblair | pabelanger: so basically, it would get used by any jobs in zuul-jobs which ran it. which includes the default base job. except we don't run the default base job, so we won't actually use it. | 16:57 |
| jeblair | pabelanger: i think the upshot is that it will be very confusing, because it will never be clear which zuul_console is being run. | 16:58 |
| jeblair | pabelanger: and we're always going to need it in zuul itself because zuul-jobs must not be required for zuul to function, and i don't consider zuul functioning without console logs. | 16:58 |
| pabelanger | right now all jobs in zuul-jobs reference zuul_console so we need it right now | 16:59 |
| pabelanger | also right now | 16:59 |
| pabelanger | :) | 16:59 |
| jeblair | pabelanger: we *have* it right now. zuul installs it. | 16:59 |
| jeblair | pabelanger: i don't think we should add it to other repos. if we need it for linting, then i think the install solution you came up with is probably best. | 16:59 |
| pabelanger | jeblair: oh, Isee what you are saying. We don't want zuul to depend on zuul-jobs | 17:00 |
| jeblair | pabelanger: correct | 17:00 |
| pabelanger | hmm... thinking | 17:00 |
| pabelanger | okay, then sounds like 485277 is okay for now then | 17:01 |
| pabelanger | and we are already working upstream with ansible to improve console streamming | 17:02 |
| mordred | pabelanger, jeblair: ++ I agree with the above | 17:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 17:06 |
| *** dkranz has joined #zuul | 17:06 | |
| *** isaacb has quit IRC | 17:07 | |
| pabelanger | okay, squashed into single commit ^ | 17:07 |
| mordred | jeblair: uhm - https://review.openstack.org/#/c/485273/ doesn't seem to be enqueing into the gate | 17:12 |
| jeblair | mordred: https://review.openstack.org/484995 are we sure we don't want to keep logging shell commands? that seems useful? | 17:12 |
| jeblair | mordred: 273 is on your stack | 17:13 |
| jeblair | mordred: so you can make that followup patch | 17:13 |
| mordred | jeblair: oh. duh. I suck | 17:13 |
| jeblair | mordred: i mean, i like nice task names, but considering how much we currently use "bash -x" i think we must feel very comfortable seeing the actual commands executed in the job log. | 17:14 |
| jeblair | mordred: (and i'd kind of hate for folks to avoid task names because they obscure the shell command) | 17:15 |
| mordred | jeblair: indeed. let's skip thatpatch for now then, it's easy enough to add back | 17:15 |
| mordred | jeblair: I'll rebase it out in just a sec | 17:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 17:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make playbook run meta info less fragile https://review.openstack.org/485284 | 17:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacer after playbook stats rather than before playbook https://review.openstack.org/485285 | 17:16 |
| mordred | jeblair: hrm. one sec - that's printing a string version of trusted into the json | 17:18 |
| jeblair | mordred: also, oops, we're omitting the actual playbook path | 17:20 |
| mordred | jeblair: k. I've got an update to the second to the last patch - but I'll wait so we don't step on eachother | 17:21 |
| jeblair | okay, i'll revise 273 now | 17:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add project info to JobDirPlaybook https://review.openstack.org/485273 | 17:23 |
| jeblair | mordred: ^ i changed the var name, sorry | 17:23 |
| *** harlowja has joined #zuul | 17:25 | |
| *** harlowja has quit IRC | 17:25 | |
| *** harlowja has joined #zuul | 17:26 | |
| pabelanger | mordred: https://review.openstack.org/#/c/485277/ is ready for re-review too | 17:29 |
| mordred | pabelanger: I think that's absolutely the right call - good thought (I was thinking about the problem yesterday and didn't have an answer I liked :) ) | 17:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make playbook run meta info less fragile https://review.openstack.org/485284 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacer after playbook stats rather than before playbook https://review.openstack.org/485285 | 17:33 |
| mordred | jeblair, pabelanger, tobiash: ok. I think THAT should finally get it :) | 17:33 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Install zuul to access zuul_console https://review.openstack.org/485277 | 17:36 |
| jeblair | mordred: lgtm. when the tests come back green, i'll switch my -2 to a +3 on the first patch | 17:41 |
| mordred | jeblair: wot | 17:41 |
| jeblair | aknot! | 17:42 |
| pabelanger | mordred: jeblair: https://review.openstack.org/#/q/topic:zuul-jobs-remove-openstack is green again and ready for a lively review process :D | 17:51 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add support for nodepool testing of openSUSE 42.3 https://review.openstack.org/484476 | 17:52 |
| *** amoralej is now known as amoralej|off | 18:01 | |
| *** jkilpatr has quit IRC | 18:06 | |
| *** jkilpatr has joined #zuul | 18:21 | |
| *** jkilpatr has quit IRC | 18:56 | |
| tobiash | mordred: tox-py35 timed out at https://review.openstack.org/#/c/485284 | 18:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Check out implicit branch in timer jobs https://review.openstack.org/485329 | 19:00 |
| jeblair | mordred, pabelanger, fungi: ^ that's something i'd like to start preparing for before we start work on periodic or post jobs. | 19:01 |
| jeblair | fungi: (your conversation this morning in infra reminded me of that) | 19:02 |
| jeblair | there's going to be more to it than that. for one, that still needs a bunch of test updates. and another thing we should expect to get out of this is a nice documentation of what the zuul variables should be for different types of jobs. | 19:03 |
| * jeblair lunches | 19:03 | |
| fungi | jeblair: nice! | 19:04 |
| *** jkilpatr has joined #zuul | 19:05 | |
| fungi | that would indeed simplify a fairly common pattern across our periodic job defs | 19:05 |
| pabelanger | jeblair: left comment, possible typo | 19:06 |
| mordred | jeblair, Shrews: I believe if we restart zuul-web and zuul-executor we'll start getting websocket console streaming | 19:14 |
| Shrews | sounds about right | 19:15 |
| tobiash | mordred: fingers crossed | 19:22 |
| tobiash | mordred: you'd need to restart the scheduler for websocket console streaming as it prepares the urls to be displayed | 19:34 |
| mordred | tobiash: oh - right. good point | 19:44 |
| * mordred is going to restart scheduler and zuul-web now | 19:45 | |
| * Shrews wonders why we have both a zuulv3 and zuulv3-dev.o.o | 19:48 | |
| tobiash | wasn't zuulv3-dev.o.o superseeded by zuulv3.o.o? | 19:48 |
| * Shrews is 503'd | 19:50 | |
| * tobiash too | 19:50 | |
| Shrews | well those are interesting exceptions | 19:51 |
| tobiash | is zuul-web already restarted/down? | 19:51 |
| mordred | zuul-web isn't starting | 19:52 |
| mordred | BUT | 19:52 |
| Shrews | what is a websocket upgrade header? | 19:52 |
| mordred | if I run it by hand it does | 19:52 |
| mordred | so - one sec, let's try again | 19:52 |
| tobiash | mordred: how does the apache forward look like? | 19:53 |
| tobiash | mordred: you need to handle websocket separately in apache | 19:53 |
| Shrews | http://paste.openstack.org/show/615896/ for the curious | 19:53 |
| mordred | http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.vhost.erb#n39 | 19:54 |
| mordred | tobiash: we installed mod_proxy_wstunnel | 19:54 |
| mordred | as well | 19:54 |
| tobiash | mordred: yes, but you need to forward this to a ws:// or wss:// url | 19:55 |
| tobiash | http://paste.openstack.org/show/615897/ | 19:55 |
| tobiash | like this | 19:55 |
| mordred | ok. zuul-web is at least properly running now | 19:55 |
| mordred | ah | 19:55 |
| tobiash | (that's part of my ansible template to configure the apache) | 19:55 |
| *** tinyurl_comSLASH has joined #zuul | 19:56 | |
| tobiash | as I said, it took me a bit to persuade apache letting websockets go through | 19:56 |
| mordred | ok - I did that by hand real quick and graceful'd apache - will receck a patch | 19:57 |
| tobiash | yay | 19:57 |
| mordred | Build ID 1bbdd00c5f2b426a9e4662ec8a3d1acb not found | 19:57 |
| tobiash | LOGS | 19:57 |
| mordred | woot | 19:57 |
| tobiash | :) | 19:57 |
| Shrews | such streaming | 19:57 |
| jeblair | http://zuulv3.openstack.org/static/stream.html?uuid=ef1c4ec1532b4b3fadda9630db5ef6fe&logfile=console.log | 19:57 |
| mordred | WOOT! | 19:57 |
| mordred | ok. I'll make the puppet to make that thing work right | 19:58 |
| mordred | tobiash: thanks! that would have taken a while to sort out I'm sure | 19:58 |
| jeblair | tobiash, Shrews, mordred: that's pretty nifty. :) | 19:58 |
| Shrews | i might watch this all day | 19:58 |
| *** tinyurl_comSLASH has left #zuul | 19:59 | |
| tobiash | so if anyone thinks this is ugly/too slow, feel free to improve the html :) | 19:59 |
| mordred | I think it's wondrous | 19:59 |
| tobiash | with long logs we experienced client side performance problems | 20:00 |
| Shrews | tobiash: i want the ability to change my font and colors!!!11!!!!11 | 20:00 |
| jeblair | --- END OF STREAM --- | 20:00 |
| jeblair | i just heard the master control program say that | 20:01 |
| Shrews | zuul-web log looks as I'd expect | 20:01 |
| tobiash | Shrews: feel free to change/css/whatever ;) | 20:01 |
| Shrews | tobiash: was only kidding, but i predict that will be the first thing folks ask for | 20:02 |
| Shrews | :) | 20:02 |
| Shrews | i actually like the white text on black | 20:02 |
| Shrews | very console-ish | 20:02 |
| tobiash | jeblair: I added this '--- END OF STREAM ---' so one sees that the job (stream) has ended | 20:03 |
| jeblair | tobiash: i love it | 20:04 |
| tobiash | :) | 20:04 |
| Shrews | "Connecting to finger server ze01.openstack.org:79" | 20:05 |
| Shrews | surprised we didn't choose a different port, but i guess we wanted finger to work, too | 20:05 |
| *** dkranz has quit IRC | 20:05 | |
| jeblair | Shrews: i think the plan is still to have a finger multiplexer like zuul-web. run it on the finger port, and then drop the executors back to unprivileged ports. so no one ever sees a ze01 hostname. | 20:06 |
| tobiash | Shrews: the finger urls are retained in the status.json, so this could be also added to the status ui | 20:07 |
| jeblair | Shrews: and yeah, supporting finger (or at least "echo foo|nc ...") as a first class citizen is definitely a goal. being able to "tail -f | grep" a streaming console log is awesome. | 20:07 |
| fungi | "END OF LINE" would have been a neat tron reference, but i guess not entirely accurate | 20:08 |
| tobiash | and what's still missing is multitenancy | 20:08 |
| tobiash | fungi: just wanted something easy to indicate the end of stream, we could also add some zuulish ascii art as end of stream marker ;) | 20:10 |
| pabelanger | nice, where did the URL for stream get populated? | 20:10 |
| fungi | hah | 20:10 |
| fungi | tobiash: it works quite well as is, but amusing suggestion nonetheless | 20:10 |
| mordred | jeblair: https://review.openstack.org/#/c/484992 is ready for you now too | 20:10 |
| tobiash | pabelanger: in the scheduler during status.json generation | 20:11 |
| jeblair | mordred: what's up with 485284? | 20:11 |
| mordred | jeblair: no clue - I was thinking we should land the stack up to it then I can rebase it so I don't have to rebase the whole stack? | 20:11 |
| jeblair | mordred: it doesn't seem to need a rebase | 20:11 |
| mordred | hrm. | 20:12 |
| tobiash | pabelanger: here https://review.openstack.org/#/c/483503/14/zuul/model.py | 20:12 |
| pabelanger | tobiash: I see it now on zuulv3.o.o status page | 20:12 |
| pabelanger | cool | 20:12 |
| mordred | I'm getting immediate end-of-stream on jobs now | 20:12 |
| mordred | like http://zuulv3.openstack.org/static/stream.html?uuid=1a3ca67388c84109979994d1722e4d72&logfile=console.log | 20:13 |
| tobiash | mordred: that happens for non-existing or ended jobs | 20:13 |
| Shrews | apache config reverted | 20:13 |
| Shrews | i think | 20:13 |
| mordred | oh. apahe config | 20:13 |
| mordred | yah | 20:13 |
| mordred | thanks puppet | 20:13 |
| Shrews | confirmed | 20:13 |
| tobiash | ah, ok | 20:13 |
| mordred | PHEW | 20:13 |
| jeblair | mordred: it's the self.change.project == None issue | 20:13 |
| jeblair | this is a situation which should not be able to happen, but we've now seen it two or three times. | 20:14 |
| tobiash | we could/should also send some error/information through the websocket if the job uuid is not found in zuul-web | 20:14 |
| jeblair | it's catastrophic | 20:14 |
| jeblair | so please, no one restart any processes in zuulv3.o.o while i look into it | 20:15 |
| jeblair | (or ze01) | 20:15 |
| jeblair | mordred: can you tell me which processes you restarted earlier? | 20:15 |
| mordred | jeblair: all of them | 20:16 |
| mordred | jeblair: I stopped the scheduler, then zuul-web, then started both of them | 20:16 |
| mordred | jeblair: then I stopped and started the executor | 20:16 |
| *** jkilpatr has quit IRC | 20:18 | |
| * tobiash is hitting eod now | 20:18 | |
| mordred | tobiash: have a good eod | 20:19 |
| * mordred hands tobiash and Shrews both pies | 20:20 | |
| jeblair | mordred: sorry, in the interest of science, i'm going to recheck a bunch of your changes in that stack. please don't do that yourself or upload new patchsets. | 20:24 |
| Shrews | pffft, science. what has that ever given us? | 20:26 |
| mordred | jeblair: go for it - I'm holding until further word | 20:27 |
| pabelanger | Shrews: the science guy! | 20:36 |
| *** jkilpatr has joined #zuul | 20:36 | |
| fungi | for science! | 20:43 |
| jeblair | mordred, fungi: i'm going to make some manual logging changes on zv3, then restart and see if this is repeatable. | 21:22 |
| mordred | jeblair: kk | 21:23 |
| jeblair | oh wow! | 21:28 |
| jeblair | i just reproduced it with the world's longest unit test | 21:28 |
| jeblair | http://paste.openstack.org/show/615912/ does fail with the error | 21:29 |
| jeblair | i have no idea how much of that is necessary. it's a faithful recreation of how that stack was built. | 21:30 |
| mordred | jeblair: well done! | 21:30 |
| mordred | jeblair: well - I at least understand the steps in the unit test | 21:31 |
| fungi | interesting | 21:31 |
| fungi | at least if it's reproducible it can probably be bisected down to a reasonable length | 21:32 |
| mordred | jeblair: (also, I remember doing those steps, so I can confirm that that's a recreation of how the stack was built) | 21:32 |
| jeblair | mordred, fungi: indeed, that reduces considerably: http://paste.openstack.org/show/615913/ | 21:42 |
| jeblair | that's 30 minutes of my life i'm not getting back | 21:42 |
| jeblair | oh wait, no i'm wrong. i went too far. | 21:43 |
| jeblair | there's the minimal case: http://paste.openstack.org/show/615914/ | 21:50 |
| jeblair | mordred, fungi: i've restarted zuul with the normal debugging level and i retract my request to preserve server state. i'm going to go fix this locally now. | 21:52 |
| fungi | indeed, still nice and concise | 22:17 |
| mordred | jeblair: cool | 22:24 |
| pabelanger | jeblair: mordred: I know you are both busy with zuulv3 things yourself, but do you mind looking at topic:zuul-jobs-remove-openstack when you have a few moments. Want to see if everybody is generally okay with that code before more playbook / role refactoring | 22:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix change history cycle detection https://review.openstack.org/485368 | 22:36 |
| jeblair | pabelanger: sure thing | 22:36 |
| jeblair | mordred: ^ | 22:36 |
| jeblair | pabelanger: i think we need mordred to weigh in on 483936 | 22:37 |
| fungi | the fix is far more concise even than the test case | 22:37 |
| pabelanger | jeblair: ya, we need to bikeshed on that still | 22:39 |
| mordred | pabelanger, jeblair I'm fine with that approach - I think we might still bikeshed about it later - but I don't think the color of that shed is a blocking concern | 22:42 |
| mordred | I've +2'd or +A'd that stack | 22:43 |
| pabelanger | mordred: Ya, I have some ideas on how we could do it, but still need to try them out. | 22:43 |
| jeblair | pabelanger: i'm not sure i understand https://review.openstack.org/484917 | 22:44 |
| SpamapS | adam_g: I'm going to go ahead and unassign you from https://storyboard.openstack.org/#!/story/2000879 (Disk monitor thread thing) | 22:44 |
| mordred | jeblair: tox already does that by default now | 22:44 |
| jeblair | pabelanger: is tox actually outputting that in our jobs now? | 22:44 |
| SpamapS | how do we feel about the urgency of that story? | 22:44 |
| pabelanger | jeblair: yup: http://logs.openstack.org/17/484917/15/check/tox-linters/d6848c0/tox/linters-1.log.txt | 22:44 |
| mordred | jeblair: yah. it does a pip freeze at the end of runs into the tox logs which we grab | 22:44 |
| pabelanger | 2017-07-18 22:31:57.632453 | ubuntu-xenial | /home/zuul/src/git.openstack.org/openstack-infra/zuul-jobs$ /home/zuul/src/git.openstack.org/openstack-infra/zuul-jobs/.tox/linters/bin/pip freeze >/home/zuul/src/git.openstack.org/openstack-infra/zuul-jobs/.tox/linters/log/linters-1.log | 22:45 |
| jeblair | mordred, pabelanger: ah i see | 22:45 |
| pabelanger | apparently tox learned how to do it | 22:45 |
| jeblair | probably worth a note to folks during rollout | 22:45 |
| mordred | jeblair, pabelanger: incidentally, reading about that caused me to learn that you can have tox write out a json report too | 22:45 |
| SpamapS | Currently, with zuul2.5, somebody could write a patch that makes a job produce super big logs, yes? | 22:45 |
| mordred | jeblair: ++ | 22:45 |
| jeblair | SpamapS: i don't think it would stop us from ptg switchover, but i think we should try to get it in before or soon after. | 22:45 |
| mordred | ++ | 22:45 |
| pabelanger | jeblair: Ya, I think we should start an upgrade docs for sure. Where would you like it to live? | 22:45 |
| SpamapS | jeblair: would it make sense to do it as some checks in trusted post jobs? | 22:46 |
| jeblair | SpamapS: currently, the "only" thing they can do is fill up the logserver, which, frankly, happens on a fairly regular bases and stops all development. in v3, they will *also* be able to kill executors in the process. :| | 22:46 |
| SpamapS | Like, could we stat all log files before transfer, sum that up, and refuse to pull back more than X bytes? | 22:46 |
| jeblair | SpamapS: yes, i think we're going to want that too (so we can separately say "this is how much you can log" and "this is how much scrap space you can have") | 22:47 |
| jeblair | SpamapS: but i think they are two things and we need both | 22:47 |
| mordred | yah. because an untrusted job is allowe dto write things to the job dir - so without the zuul-level check we don't have a way of 'blocking' it | 22:47 |
| jeblair | SpamapS: (we will almost certainly want to permit jobs more scratch space than log storage space. too much scratch space can kill the executor. and too much log space can kill the logserver) | 22:48 |
| mordred | otoh - if a job fills up the job dir on the executor, it is a tempdir and will be deleted - so it'll onlyu kill the exectutor for $time | 22:48 |
| mordred | jeblair: ++ | 22:48 |
| jeblair | yeah. failing 150 jobs at once is... awkward, so i'd rather avoid it if possible. :) | 22:49 |
| pabelanger | jeblair: SpamapS: somebody could also propose a job that requests 2k nodes still right? | 22:49 |
| jeblair | pabelanger: yep. that's another limit which we should encode. | 22:49 |
| jeblair | pabelanger: fortunately, if they ask for 2k nodes, nothing will happen. if they ask for 1400 nodes -- *that's* a problem. :) | 22:50 |
| mordred | that limit is a bit easier | 22:50 |
| mordred | jeblair: ++ | 22:50 |
| jeblair | mordred: ya | 22:50 |
| fungi | though it will also be queued indefinitely unless there is a provider capable of booting that many nodes, right? | 22:50 |
| mordred | fungi: yes. | 22:50 |
| fungi | or did we decide that jobs spanning providers was a thing? | 22:50 |
| jeblair | fungi: you're right... i guess the problem point is... what, 900 nodes? | 22:50 |
| fungi | whatever our largest provider is, right | 22:50 |
| mordred | yah - and even that will only grab those 900 nodes for an hour or two - so while not ideal - and certainly a thing that would get corrected socially very quickly ... it's not deadly | 22:51 |
| mordred | it would kind of be amusing to wath | 22:51 |
| mordred | watch | 22:51 |
| fungi | 256 actually... each of the two osic entries (s3500 and s3700) are limited to max-servers: 256 | 22:52 |
| jeblair | the provider would be monopolized for the time it takes to get 256 free nodes (probably 1-2 hours), and the time the job ran | 22:53 |
| jeblair | pabelanger: your stack is +3d | 22:54 |
| pabelanger | ty! | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove check_sudo_usage logic from tox https://review.openstack.org/484916 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create tox-pep8 jobs https://review.openstack.org/484948 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove pip freeze logic https://review.openstack.org/484917 | 22:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Require tox_envlist for tox role https://review.openstack.org/484975 | 22:54 |
| SpamapS | jeblair: I just wonder if we can control all the places that write to scratch space and avoid the overseer. | 22:54 |
| jeblair | mordred: i +3d your stack | 23:11 |
| mordred | jeblair: woot! | 23:12 |
| jeblair | SpamapS: i don't think so -- we let folks download arbitrary stuff from the internet into the executor workspace | 23:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove OOM check from tox role https://review.openstack.org/485377 | 23:28 |
| pabelanger | jeblair: mordred: do we want to consider something like reno to manage release notes in zuul-jobs? It goes back to the question about upgrade doc | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul command default function https://review.openstack.org/484279 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 23:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix json output appending https://review.openstack.org/484992 | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Emit phase banner separate from play banner https://review.openstack.org/484993 | 23:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove noise both vertical and horizontal https://review.openstack.org/484994 | 23:33 |
| *** hasharLunch has quit IRC | 23:43 | |
| mordred | pabelanger: I started a patch a while back to add reno support to zuul and nodepool but never got around to finishing it | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!