| mordred | pabelanger: the bug that we should address for sure is that on any shell command that doesn't produce output (like bindep -b when it succeeds and there are no more packages to install) - the command never produces output so we never get any packets when we read from the socket | 00:00 |
|---|---|---|
| mordred | I think maybe we should, in those cases, emit an EOF or something on the command side - or just handle it differently in the stream so we don't emit those log lines in those cases | 00:01 |
| mordred | have we restarted the executor since the latest zuul_stream patch landed? | 00:02 |
| pabelanger | I have not | 00:02 |
| mordred | pabelanger: oh - also - I wrote a python wrapper around finger to make it easy for me to finger the finger urls we produce - but I could never figure out how to get it connected to chrome like you figured out for telnet | 00:02 |
| mordred | I suppose there will be websockets for that soon enough | 00:03 |
| pabelanger | mordred: Ya, I updated my chorme to open finger:// also. I should propose a patch to zuul, like jeblair has for gertty | 00:04 |
| pabelanger | I can do that in the morning | 00:04 |
| mordred | woot | 00:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: add environment for tox shell https://review.openstack.org/483608 | 00:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - add upper-constraints to tox https://review.openstack.org/483548 | 00:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - add upper-constraints to tox https://review.openstack.org/483548 | 00:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: add environment for tox shell https://review.openstack.org/483608 | 00:23 |
| pabelanger | Hey, I got it working | 00:26 |
| pabelanger | \o/ | 00:26 |
| pabelanger | I'll clean up the patches in the morning | 00:27 |
| pabelanger | and we can bikeshed over job news for constraints things | 00:27 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add documentation for project shadowing https://review.openstack.org/483552 | 02:52 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make roles ordered https://review.openstack.org/483140 | 02:52 |
| *** xinliang has quit IRC | 04:39 | |
| *** xinliang has joined #zuul | 04:41 | |
| *** xinliang has quit IRC | 04:41 | |
| *** xinliang has joined #zuul | 04:41 | |
| tobiash | mordred: so far I didn't have issues with cross-domain things | 04:47 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Switch to using setup instead of hostvars https://review.openstack.org/483585 | 04:57 |
| *** dpawar has joined #zuul | 05:50 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add zuul.d configuration split documentation https://review.openstack.org/482405 | 06:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Fix zuul.(yaml|d) loading order and add tests https://review.openstack.org/482411 | 06:15 |
| tobiash | mordred: in the html patch you added a name to the route | 06:21 |
| tobiash | mordred: is that on purpose (and required)? | 06:22 |
| tobiash | mordred: with the name add_route blocks indefinitely and zuul-web doesn't start correctly | 06:22 |
| *** dpawar has left #zuul | 06:26 | |
| *** dpawar has joined #zuul | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:10 | |
| *** bhavik1 has joined #zuul | 08:40 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove run-cover role https://review.openstack.org/483527 | 08:47 |
| *** dpawar has left #zuul | 09:33 | |
| *** bhavik1 has quit IRC | 10:10 | |
| *** xinliang has quit IRC | 10:43 | |
| *** jkilpatr has quit IRC | 10:53 | |
| *** xinliang has joined #zuul | 10:55 | |
| *** xinliang has joined #zuul | 10:55 | |
| *** jkilpatr has joined #zuul | 11:25 | |
| *** dpawar has joined #zuul | 12:10 | |
| *** amoralej is now known as amoralej|lunch | 12:49 | |
| mordred | tobiash: oh. weird. the name was so that we could use the url_for method - I think we could do that differently | 12:53 |
| tobiash | mordred: also fighting with the fetch and promises with resolution with then... | 12:54 |
| tobiash | mordred: crap js | 12:54 |
| * tobiash needs to get coffee | 12:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Serve html statically https://review.openstack.org/483503 | 13:06 |
| mordred | tobiash: yah. I love debugging javascript | 13:06 |
| * tobiash doesn't ;) | 13:07 | |
| mordred | tobiash: that removes the name parameter and just defaults websocket_url to a relative path - and fixes a syntax error | 13:07 |
| tobiash | mordred: got it to work now with a synchronous XMLHttpRequest | 13:11 |
| mordred | tobiash: oh great! | 13:12 |
| tobiash | but the url stuff in the executor doesn;t work | 13:12 |
| tobiash | zuul-web needs the finger url coming from zuul-executor | 13:12 |
| mordred | ohhhhh. CRAPPIT yes you're right of course | 13:13 |
| tobiash | mordred: we need to do this in the scheduler during status.json formatting | 13:13 |
| mordred | yes - that makes much more sense | 13:14 |
| tobiash | mordred: you wanted to retain the finger url and web streaming url right? | 13:15 |
| tobiash | mordred: what about formatting the url for web streaming and adding a finger_url to the status.json? | 13:15 |
| mordred | ++ I think that's a great idea | 13:15 |
| tobiash | so the status ui could display both then | 13:15 |
| pabelanger | Are we talking about web url not getting displayed into status.json after job is finihed? Noticed that on zuulv3.o.o yesterday, but figured it was still not working properly | 13:16 |
| mordred | pabelanger: no - talking about plumbing in support for the websocket streaming and html page to display it | 13:16 |
| mordred | pabelanger: but that also needs to be fixed :) | 13:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Revert "Remove run-cover role" https://review.openstack.org/483926 | 13:17 |
| pabelanger | mordred: ^ if you want to revert, but merged yesterday before I could abandon | 13:18 |
| tobiash | mordred: should we being able to enable/disable url and finger_url in status.json? | 13:23 |
| tobiash | mordred: the status ui (static stuff) problably also should be served by zuul-web right? | 13:27 |
| tobiash | mordred: then the webstreaming url on the status page just could be "stream.html?uuid=..." | 13:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update to tox_envlist variable https://review.openstack.org/483934 | 13:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_environment to allow users access to shell variables https://review.openstack.org/483935 | 13:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 13:30 |
| mordred | tobiash: yes - eventually we do want to move status things to zuul-web - but we also want to allow folks to run custom local status pages in other locations - however, your thought about doing the url in the scheduler has led me down a path that I think may make a bunch of things simpler | 13:31 |
| pabelanger | mordred: did you want to restart zuulv3 to pick up our no_log fixes? | 13:36 |
| tobiash | mordred: have a prototype now, it's ok if I upload a new version of your html patch? | 13:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Serve html statically https://review.openstack.org/483503 | 13:52 |
| tobiash | mordred: prove of concept patch ^^^ | 13:52 |
| tobiash | s/prove/proof/ | 13:53 |
| *** amoralej|lunch is now known as amoralej | 13:55 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Serve html statically https://review.openstack.org/483503 | 13:57 |
| pabelanger | 2017-07-14 14:01:58.508291 | static.openstack.org | ok: 1 changed: 0 unreachable: 0 failed: 1 | 14:03 |
| pabelanger | mordred: jeblair: starting to see that on some zuulv3 jobs^ | 14:03 |
| mordred | tobiash: awesome! I like how you did that | 14:03 |
| pabelanger | There was an issue creating /srv/static/logs03 as requested: [Errno 13] Permission denied: '/srv/static/logs03 | 14:03 |
| pabelanger | looks like we have a bad URL some place | 14:03 |
| mordred | pabelanger: that sounds like we're missing a / | 14:04 |
| pabelanger | ya | 14:04 |
| pabelanger | I think I see the issue | 14:04 |
| mordred | pabelanger: I think I saw the ansible doing a join without / in between two vars | 14:04 |
| mordred | cool | 14:04 |
| mordred | tobiash: I also played with something similar - but which is untested - I'm going ot push it up just so you can look at it | 14:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Door number three on console streaming https://review.openstack.org/483947 | 14:05 |
| mordred | tobiash: similar to what you did in some places or complimentary to it - main thoughts are that we can actually pass websocket_url in the url we generate in status.json so we can skip the /info endpoint | 14:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add missing / for log server path https://review.openstack.org/483949 | 14:06 |
| pabelanger | mordred: ^ should be it | 14:06 |
| mordred | tobiash: and I tried to update the jquery web app thing to consume things | 14:06 |
| pabelanger | odd that it seems to be intermittent failures | 14:07 |
| mordred | tobiash: but I like how you're constructing url in model better than doing it in the javascript | 14:07 |
| tobiash | mordred: looking | 14:08 |
| mordred | tobiash: (long story short - I think basing things on your patch is the right way to go - mostly pushed mine up in case anything there was useful :) ) | 14:09 |
| tobiash | mordred: supplying the websocket url looks quite good for removing the info endpoint and the additional query | 14:15 |
| tobiash | mordred: I'll check if I can combine both patches in this way | 14:17 |
| *** dpawar has left #zuul | 14:18 | |
| mordred | tobiash: I think I need to also write up some thoughts about what url serving *wants* to be even if we don't get it all there immediately and get your input to make sure it works for you - since we have several different places currently that have web context but in the future there should be one - some of these config bits may also move or change | 14:21 |
| mordred | and it might be good to have a picture of what it wants to be as we work | 14:22 |
| tobiash | mordred: sounds good | 14:25 |
| pabelanger | mordred: ya, looks like all jobs are POST_FAILURE right now: 483949 is the fix, but jobs still fail before trusted playbooks | 14:25 |
| tobiash | mordred: I'm heading home now and will be in full day workshops on monday and tuesday | 14:26 |
| * tobiash raises WorkshopOverflowException | 14:26 | |
| mordred | tobiash: oh my - that sounds likea lot of fun | 14:33 |
| mordred | tobiash: should I take the mutext on trying to combine the patches until you get back wednesday? | 14:33 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add missing / for log server path https://review.openstack.org/483949 | 14:41 |
| jeblair | mordred, pabelanger: i suspect the non-ordering of roles has caused us to sometimes use the new version of the upload-logs role that i'm testing, which is why post_failures are happening. the change to order roles appears to have landed, so it might be a good idea to restart now | 14:52 |
| pabelanger | jeblair: ++ | 14:53 |
| jeblair | mordred, pabelanger: though if you have a minute to also review https://review.openstack.org/483141 it might be good to go ahead start testing that | 14:53 |
| pabelanger | looks | 14:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Remove run-cover role" https://review.openstack.org/483926 | 14:54 |
| tobiash | mordred: if you like to and have spare time feel free to do so, otherwise I'll just continue on that on wednesday | 15:01 |
| mordred | jeblair: looking | 15:03 |
| pabelanger | +2 | 15:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_environment to allow users access to shell variables https://review.openstack.org/483935 | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - add upper-constraints to tox https://review.openstack.org/483548 | 15:24 |
| jeblair | pabelanger: your workflow+0 on 483141 gave 483142 a -2 from jenkins :( | 15:26 |
| pabelanger | jeblair: odd | 15:28 |
| pabelanger | I +0 so mordred could also review | 15:29 |
| jeblair | pabelanger: it's not odd -- it's what happens when a dependent change is removed from the gate pipeline. | 15:29 |
| jeblair | mordred: are you still reviewing? | 15:29 |
| pabelanger | right, forgot about that | 15:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - add upper-constraints to tox https://review.openstack.org/483548 | 15:31 |
| pabelanger | jeblair: jlk: has anybody reached out to sigmavirus24 to tag a new release of github3.py? | 15:41 |
| jlk | hadn't gotten around to it. We were talking about replacing github3.py with just requests, or with whatever for using graphql. But then ran into roadblocks with graphql. | 15:42 |
| pabelanger | k, we are still depending on github for the package in requirements | 15:42 |
| pabelanger | I'll send out a ping and see what would be needed to do a new tag | 15:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Run playbooks with only those roles defined thus far https://review.openstack.org/483141 | 15:44 |
| jlk | thanks | 15:44 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add child roles before parent https://review.openstack.org/483142 | 15:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 15:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update to tox_envlist variable https://review.openstack.org/483934 | 15:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_environment to allow users access to shell variables https://review.openstack.org/483935 | 15:50 |
| pabelanger | mordred: okay, https://review.openstack.org/#/q/topic:zuul-jobs-remove-openstack works. Confirmed with test job | 15:56 |
| pabelanger | mordred: I think what this means, is we are going to need openstack-tox-py35 jobs (for example) that setup some of the variables needed for contraints | 15:57 |
| pabelanger | mordred: the job would also have a projet dependency on openstack/requirements | 15:58 |
| pabelanger | I am going to try writing that up now and see what it would look like | 15:58 |
| pabelanger | it also removes NOSE variables, in zuul-jobs so we can also create new playbooks to set them up | 15:58 |
| mordred | pabelanger: hrm. it's a real shame that we won't even be able to use the tox-py35 job directly from zuul-jobs :( | 16:05 |
| pabelanger | mordred: it will be our parent | 16:05 |
| pabelanger | but ya, we'll need some openstack specific things I think | 16:05 |
| mordred | pabelanger: so - remind me how the upper-constraints.txt file gets in to that dir again? | 16:07 |
| pabelanger | mordred: today we zuul-clone requirements, and cp upper-contraints.txt to workspace | 16:07 |
| *** tflink has quit IRC | 16:08 | |
| mordred | so - sake of argument real quick ... | 16:09 |
| pabelanger | basically: http://logs.openstack.org/42/483142/4/check/gate-zuul-python35/af45bd0/console.html#_2017-07-14_15_29_44_012276 | 16:09 |
| pabelanger | in zuul project, we actually have an utracked file in our repo at test time, but tox.ini isn't setup to use it | 16:09 |
| *** jeblair has quit IRC | 16:09 | |
| mordred | with your tox_environment variable, tox_environment being set will be passed in to the tox-py35 | 16:10 |
| pabelanger | Ya, https://review.openstack.org/#/c/483548/ is a dirty example | 16:11 |
| pabelanger | obviously it fails before foo.txt doesn't exist | 16:11 |
| *** jeblair has joined #zuul | 16:11 | |
| mordred | pabelanger: so that means we COULD set tox_environment: UPPER_CONSTRAINTS=upper-constraints.txt in our overridden base job | 16:11 |
| mordred | (bear with me here a second) | 16:12 |
| mordred | (thinking out loud) | 16:12 |
| pabelanger | ack | 16:12 |
| mordred | we could also actually set it to {{ zuul.executor.work_dir }}/src/git.openstack.org/openstack/requirements/upper-constraints.txt | 16:12 |
| pabelanger | Right | 16:13 |
| mordred | so that the env var is always set to point to the u-c file - and we make sure the file is there just by adding openstack/requirements to the required_projects list | 16:13 |
| mordred | hrm. that still would make us make an openstack-tox-py35 job that added required_projects | 16:14 |
| pabelanger | Right | 16:14 |
| mordred | unless we ALSO added that to our base job - but I think that's potentially a bit much | 16:14 |
| *** tflink has joined #zuul | 16:15 | |
| mordred | wait a sec - jeblair, can you add required projects or variables to a job in a variant? you can add variables, right? | 16:15 |
| pabelanger | I think it is worth testing, but I am a little concerned base would start to get a little large over time | 16:15 |
| jeblair | mordred: both, yes, afaik | 16:15 |
| mordred | pabelanger: yes - totally agree | 16:15 |
| jeblair | mordred: but variants have to be defined in the same repo (except project-pipeline variants) | 16:16 |
| mordred | jeblair: gotcha. so we can't define a tox-py35 variant in project-config or openstack-zuul-jobs that adds that required_project and variable | 16:16 |
| jeblair | mordred: correct | 16:17 |
| mordred | kk. well, I agree with pabelanger then that we need an openstack-tox-py35 (or honestly we could just call it openstack-py35 or something) | 16:17 |
| mordred | that has tox-py35 as its parent | 16:17 |
| pabelanger | I am going to get the naming wrong, but what we need need to do is have tox job in openstack-zuul-jobs, which have our proper tox_environment things. then we could still use tox-py35. would that be a shadow job? | 16:17 |
| mordred | pabelanger: it would just be a child job - parent: tox-py35 | 16:18 |
| jeblair | pabelanger: if it's the same name, it's shadowing. so if you wanted to call it 'tox-py35' you can do that with shadowing. if you want to call it 'openstack-py35' then it's inheritance. | 16:18 |
| jeblair | pabelanger: basically, shadowing is so you can *replace* a job defined by the stdlib with one of your own. | 16:19 |
| mordred | ooh. this might be too much -- but what if we called it "pti-python35" - so that we made it clear that it's a job that has specific needs as defined by the OpenStack PTI? | 16:19 |
| pabelanger | right, so we could shadow tox-py35 in openstack-zuul-jobs and have its parent be tox-py35 in zuul-jobs? | 16:19 |
| jeblair | mordred: appealing, but i think openstack-py35 may be more clear on first glance. | 16:20 |
| mordred | jeblair: kk | 16:20 |
| pabelanger | let me mock up openstack-py35, see how it would look | 16:20 |
| mordred | pabelanger: cool | 16:20 |
| jeblair | pabelanger: we can't actually have openstack-zuul-jobs/tox-py35 with shadow parent zuul-jobs/tox-py35 due to the way config loading/parsing is implemented. we may be able to do that later, but it'll take some refactoring. | 16:21 |
| mordred | and I think in this case having an openstack-py35 is actually nicer | 16:21 |
| jeblair | (all jobs are loaded and parsed in order, so we can't inherit from something later in the config, and a shadowed job is always later) | 16:21 |
| mordred | since it allows projects like zuul, which doens't use upper-constraints, to use plain tox-py35 | 16:22 |
| pabelanger | right | 16:22 |
| mordred | pabelanger: so I think what you've got going here is solid | 16:22 |
| jeblair | mordred, pabelanger: puppet has run; i'm going to restart zv3 | 16:23 |
| pabelanger | kk | 16:24 |
| mordred | jeblair: kk | 16:25 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add support to test fedora-26 https://review.openstack.org/482568 | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: DNM - add upper-constraints to tox https://review.openstack.org/483548 | 16:30 |
| pabelanger | Oh, ha | 16:31 |
| pabelanger | we need to add openstack/requirements to zuulv3 configuration :D | 16:31 |
| pabelanger | otherwise, syntax error for testing | 16:31 |
| jeblair | why is the file in that message "None"? | 16:32 |
| pabelanger | noticed that too | 16:32 |
| jeblair | i keep seeing that. it should actually have the file path | 16:32 |
| jeblair | it does in the unit tests. :| | 16:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 16:32 |
| jeblair | http://paste.openstack.org/show/615413/ | 16:36 |
| jeblair | like that | 16:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 16:37 |
| mordred | jeblair: ok ^^ I think I did that right with the layout changes | 16:37 |
| mordred | pabelanger, jeblair: if we need to add o/r to be able to test this, how about we go ahead and add both o/r and shade since shade follows u-c - adding shade wasn't really next on our list - but having a repo that behaves more strictly like 'openstack' seems like a thing we need now | 16:42 |
| jeblair | mordred: i think you have half-undid some of the work i did for the roles stuff. left comments. | 16:42 |
| mordred | jeblair: thanks - fixing | 16:42 |
| jeblair | mordred: wfm. | 16:43 |
| jeblair | mordred, pabelanger: i'd like to perform some live debugging on zv3.o.o to try to understand why the file isn't showing up in the error message. | 16:44 |
| jeblair | i'm going to install a locally modified zuul with extra debug statements | 16:44 |
| pabelanger | mordred: ++ to shade too | 16:44 |
| pabelanger | jeblair: go for it, I'll get some coffee / food and prepare to add o/r and shade to zuulv3 configuration | 16:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 16:46 |
| jeblair | mordred: lgtm. why the dict in "dict(args['secrets'])" ? | 16:48 |
| mordred | jeblair: was copying the pattern in: | 16:48 |
| mordred | def prepareAnsibleFiles(self, args): | 16:48 |
| mordred | all_vars = dict(args['vars']) | 16:48 |
| mordred | both of which I think intend to make a copy | 16:48 |
| mordred | we could probalby, you know, use copy() instead :) | 16:49 |
| jeblair | mordred: gotcha. yeah. we should change that to copy on vars, so we know why. i think we can drop it for secrets though since we're not modifying it. | 16:50 |
| jeblair | mordred: oh. | 16:51 |
| jeblair | mordred: i just thought of something. | 16:51 |
| mordred | jeblair: what's that? | 16:51 |
| jeblair | mordred: i'd really like to avoid people being able to supply variables in the 'zuul' namespace. so why don't we also copy secrets, and check for that and del secrets['zuul'] if it exists. | 16:51 |
| mordred | jeblair: ++ | 16:52 |
| jeblair | mordred: that silly question about dict() led to something useful. :) | 16:53 |
| mordred | jeblair: should we silently remove? or return an error? | 16:53 |
| mordred | I think error - we should inform people that we're not doing what they think | 16:53 |
| jeblair | mordred: i'm okay with either. silent removal is what happens now because of the way we stomp on vars. if you want to return an error, that's cool; we should do it in both places. | 16:54 |
| jeblair | (vars and secrets) | 16:54 |
| mordred | where do we strip it from vars? | 16:54 |
| mordred | or are you saying we should also add the check to both places | 16:54 |
| jeblair | mordred: that's what the copying thing is. we copy vars, then we *set* the zuul value. it will silently overwrite it if it exists. | 16:54 |
| jeblair | mordred: so yeah, if you want to raise an error, perform the check for both vars and secrets. | 16:54 |
| mordred | jeblair: ah - yes. gotcha | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 16:59 |
| mordred | jeblair: alright - I think that one is solid now | 17:00 |
| jeblair | mordred: nope, one more thing | 17:01 |
| jeblair | (inline comment) | 17:02 |
| mordred | gah | 17:02 |
| mordred | jeblair: also - while you're looking at things: https://review.openstack.org/#/c/482889/ and https://review.openstack.org/#/c/482926/ | 17:02 |
| mordred | jeblair: GAH | 17:03 |
| mordred | jeblair: also - feels like maybe a thing we want to move into the configloader so we can give quick errors without allocating nodes | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 17:04 |
| jeblair | mordred: yeah, should be possible as well. let's leave this there though since we're talking about passing data between jobs as variables. | 17:05 |
| mordred | jeblair: ++ | 17:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets https://review.openstack.org/484000 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: gzip console log before uploading it https://review.openstack.org/483611 | 17:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add /etc/hosts and /etc/nsswitch.conf to the bubblewrap https://review.openstack.org/483210 | 17:21 |
| mordred | jeblair: well - th block-zuul-variable patch is an excellent test case for putting a size-limit on console output :) | 17:25 |
| mordred | jeblair: 182.62M | 17:25 |
| jeblair | i'd love to know why that happens | 17:26 |
| jeblair | i didn't think anything was supposed to go to the console when we run in testr | 17:27 |
| jeblair | i'm too deep down my stack to look at it though | 17:27 |
| *** dmsimard is now known as dmsimard|afk | 17:30 | |
| mordred | jeblair: in this case, I believe the problem is that there are real exceptions, so it's gathering all the things | 17:31 |
| *** timolo has joined #zuul | 17:31 | |
| mordred | jeblair: heh. turns out we cannot block zuul right there - becuase zuul has already passed variables named zuul | 17:34 |
| mordred | fixing | 17:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets https://review.openstack.org/484000 | 17:39 |
| mordred | jeblair: ^^ THAT should do it now | 17:39 |
| *** timolo has quit IRC | 17:43 | |
| *** dpawar has joined #zuul | 17:43 | |
| jeblair | aha! i think it may be that we're using cyaml in prod, but not in unit tests | 17:48 |
| mordred | jeblair: oh! we should maybe use cyaml in unit tests | 17:55 |
| mordred | yah - we don't have libyaml-dev in bindep | 17:56 |
| jeblair | woohoo! i have reproduced locally by install the c lib | 17:57 |
| jeblair | i will reset the prod server to branch tip now | 17:58 |
| jeblair | done | 17:58 |
| mordred | jeblair: woot | 18:00 |
| *** dpawar has quit IRC | 18:19 | |
| *** dpawar has joined #zuul | 18:24 | |
| SpamapS | the funny thing about that | 18:31 |
| SpamapS | is that we mostly did cyaml to make unit tests faster ;) | 18:31 |
| jeblair | i'm pretty sure i can get zuul errors down to the actual line (like here's the line where you named an unknown project). but i'll leave that for later since i'm in the middle of several other things | 18:36 |
| jeblair | i'm going to just fix the file/line pointers, and replace the pprinted data with the equivalent yaml directly from the file | 18:37 |
| jeblair | part of why i'm doing the second thing is that cyaml marks don't come with snippets | 18:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Handle cyaml Marks https://review.openstack.org/484013 | 18:41 |
| jeblair | SpamapS, mordred, pabelanger: ^ | 18:42 |
| SpamapS | jeblair: sometimes I wonder if the cyaml is worth the trouble. :_P | 18:43 |
| SpamapS | I'm sure it's a whole percent or two of total CPU.. and maybe a couple more percent RAM.. but.. :-P | 18:44 |
| jeblair | SpamapS: me too. but when i say to myself "how much slower can parsing yaml in python be?" i give myself a withering glare. | 18:44 |
| SpamapS | The RAM problems may be worse than the CPU | 18:44 |
| SpamapS | Have seen 10k yaml docs use up 50MB of memory for no reason. :-P | 18:44 |
| *** bhavik1 has joined #zuul | 18:45 | |
| SpamapS | probably naively creates lots of cycles. | 18:45 |
| SpamapS | I wonder if it behaves better on py3 which is supposed to have better cycle gc | 18:45 |
| jeblair | i guess we will find out | 18:45 |
| *** dpawar has quit IRC | 18:52 | |
| *** bhavik1 has quit IRC | 19:04 | |
| *** amoralej is now known as amoralej|off | 19:21 | |
| pabelanger | jeblair: mordred: I have manually done SIGHUP on zuulv3. Working on a patch to puppet-zuul now | 19:36 |
| pabelanger | jeblair: I think you said we can just use the same logic from zuul.init from zuul.o.o right>? | 19:36 |
| jeblair | pabelanger: yep | 19:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Handle cyaml Marks https://review.openstack.org/484013 | 20:08 |
| pabelanger | openstack-py35 success! | 20:08 |
| pabelanger | https://review.openstack.org/#/c/484027/ | 20:08 |
| pabelanger | and used correct constraints file | 20:09 |
| pabelanger | http://logs.openstack.org/27/484027/2/check/openstack-py35/3c3300a/tox/py35-0.log.txt | 20:09 |
| pabelanger | mordred: ^ | 20:09 |
| jeblair | pabelanger: have you looked at the recent post-failures yet? | 20:10 |
| pabelanger | jeblair: no, I just noticed them | 20:10 |
| pabelanger | checking some logs now | 20:10 |
| jeblair | pabelanger: oh, i think it may be just the docs jobs | 20:11 |
| jeblair | pabelanger: http://paste.openstack.org/show/615428/ | 20:12 |
| pabelanger | Hmm | 20:12 |
| jeblair | "ERROR! the role 'run-docs' was not found in /tmp/92c6cb40dc4940289cc16a13f5481ac5/work/src/git.openstack.org/openstack-infra/openstack-zuul-jobs/playbooks/tox/roles:/tmp/92c6cb40dc4940289cc16a13f5481ac5/ansible/playbook_0/role_0/zuul-jobs/roles:/tmp/92c6cb40dc4940289cc16a13f5481ac5/work/src/git.openstack.org/openstack-infra/openstack-zuul-jobs/playbooks/tox" | 20:12 |
| jeblair | forgot the actual error in the pastebin | 20:12 |
| pabelanger | wonder if we broke something | 20:13 |
| jeblair | where is run-docs supposed to be? | 20:14 |
| pabelanger | openstack-zuul-jobs right now | 20:14 |
| jeblair | pabelanger: but the job is defined in zuul-jobs? | 20:15 |
| jeblair | oh, no. openstack-zuul-jobs as well | 20:15 |
| pabelanger | no, openstack-doc-build is also in openstack-zuul-jobs, in zuul.yaml for project-config I think | 20:15 |
| pabelanger | Ya, project-config | 20:16 |
| jeblair | pabelanger: what's in project-config? | 20:16 |
| pabelanger | the project section for openstack-infra/zuul-jobs | 20:16 |
| jeblair | gotcha | 20:16 |
| pabelanger | So many place to look :D | 20:17 |
| jeblair | so the role path in that error indeed does not contain openstack-zuul-jobs/roles | 20:17 |
| jeblair | but it's in the job definition | 20:18 |
| jeblair | so it seems like there may be an error with the recent role stuff | 20:18 |
| jeblair | i've turned on keep and am rechecking so i can poke at the jobdir | 20:19 |
| pabelanger | openstack-zuul-jobs/playbooks/tox doesn't look right for a role path | 20:20 |
| pabelanger | from your ERROR above | 20:20 |
| jeblair | pabelanger: that's probably from ansible, but we'll know in a minute. | 20:20 |
| jeblair | pabelanger: (ansible has an implicit role path next to the playbook its running) | 20:21 |
| pabelanger | ya, tox/roles is also surprising | 20:21 |
| pabelanger | jeblair: ya | 20:21 |
| jeblair | roles_path = /tmp/3c6feda8f9644a56981c5c27fd966d38/ansible/playbook_0/role_0/zuul-jobs/roles | 20:23 |
| jeblair | so yeah, the weird additions are just ansible. we correctly have zuul-jobs, but incorrectly are missing openstack-zuul-jobs. | 20:24 |
| pabelanger | I wonder if we could just symlink the roles directly in the openstack-zuul-jobs folder | 20:24 |
| jeblair | pabelanger: we do | 20:25 |
| jeblair | the error looks like it's in the scheduler. it didn't tell the executor that was a role path for that playbook. | 20:25 |
| jeblair | pabelanger: if you want to poke around at the new playbook/role directory setup, you can look at /tmp/3c6feda8f9644a56981c5c27fd966d38/ansible on ze01 | 20:26 |
| pabelanger | jeblair: k, looking | 20:26 |
| jeblair | i'm turning keep off, but that dir will stick around | 20:26 |
| pabelanger | going to try something | 20:30 |
| jeblair | pabelanger: can you be more specific? | 20:31 |
| pabelanger | remote: https://review.openstack.org/484039 wip: symlink role directory | 20:32 |
| pabelanger | so, doing that actually causes some changes to ansible-lint | 20:32 |
| jeblair | pabelanger: oh please don't do that. :) | 20:32 |
| pabelanger | I wonder if that is because our roles are not getting loaded | 20:33 |
| jeblair | pabelanger: you wonder if what is because our roles are not getting loaded? | 20:33 |
| pabelanger | jeblair: so, ansible-lint will fail on 484039 | 20:33 |
| pabelanger | bacause, I think we are now linting roles with that change | 20:33 |
| jeblair | pabelanger: i still don't understand what you're saying | 20:34 |
| pabelanger | this is unrelated to problem at hand | 20:34 |
| pabelanger | but, I don't think we are properly linting our roles today with ansible-lint | 20:34 |
| pabelanger | http://logs.openstack.org/39/484039/1/check/tox-linters/959513d/job-output.txt for more detail | 20:36 |
| pabelanger | but ya, I will abandon that for now | 20:36 |
| *** jkilpatr has quit IRC | 20:49 | |
| * SpamapS is so happy there's an ansible-lint :) | 20:53 | |
| pabelanger | okay, fixed | 20:56 |
| pabelanger | remote: https://review.openstack.org/484043 Ensure we load roles for linting | 20:56 |
| pabelanger | that is how I usually set it up for the other roles I do | 20:57 |
| jeblair | pabelanger: lgtm. what's 0012? | 20:58 |
| jeblair | also, i have locally reproduced the roles path error | 20:59 |
| pabelanger | jeblair: Commands should not change things if nothing needs doing. For idempotent tasks I belive | 21:00 |
| pabelanger | I can update patch to reference that | 21:01 |
| *** jkilpatr has joined #zuul | 21:06 | |
| pabelanger | okay, added --syntax-check and renamed to tox -elinters too | 21:07 |
| pabelanger | that should better help us debug ansible things locally | 21:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix multiple role paths https://review.openstack.org/484047 | 21:10 |
| jeblair | pabelanger: there's the problem ^ | 21:10 |
| jeblair | tuples strike again | 21:10 |
| pabelanger | +2 | 21:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle cyaml Marks https://review.openstack.org/484013 | 21:27 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix multiple role paths https://review.openstack.org/484047 | 21:29 |
| jeblair | restarting zv3 | 21:57 |
| pabelanger | ack | 21:58 |
| jeblair | done | 21:58 |
| pabelanger | k, starting rechecks | 21:59 |
| pabelanger | much better | 22:10 |
| jeblair | switching 'keep' on to debug the base-test stuff i'm working on | 22:12 |
| pabelanger | jeblair: ack. think I'm calling it for today. might poke at a few things over the weekend | 22:13 |
| jeblair | pabelanger: thanks, enjoy! | 22:14 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!