| pabelanger | mordred: jeblair: it looks like changes to executor rw / ro paths will require a restart of zuul-executor. I believe we talked about adding reload support to zuul.conf for zuul-executor before | 02:36 |
|---|---|---|
| pabelanger | I've restarte ze01 to pickup latest zuul.conf | 02:40 |
| pabelanger | Boom! | 02:51 |
| pabelanger | https://docs.openstack.org/sandbox/ | 02:51 |
| pabelanger | afs publishing from zuulv3 | 02:51 |
| pabelanger | remote: https://review.openstack.org/499876 Copy contents inside html folder for python-docs job | 02:51 |
| pabelanger | should fix the html issue | 02:51 |
| pabelanger | jeblair: mordred: ^for the morning | 02:52 |
| *** notlobryaj has joined #zuul | 05:18 | |
| *** notlobryaj has left #zuul | 05:19 | |
| *** bhavik1 has joined #zuul | 06:53 | |
| *** qwc has joined #zuul | 07:02 | |
| qwc | Hi, we got an obvious gerrit -> zuul/gearman-> jenkins configuration. Now we have a project with submodules, both the parent project and the submodule have it's Jenkinsfile and the proper configuration for gating inside zuul, but now we want to build the parent with the submodule when a change happens... how? (current approach: special jenkins job with special Jenkinsfile, which checks out the change from the Zuul Merger, any other options with zuul itself maybe?) | 07:12 |
| qwc | when a change inside the submodule happens <- | 07:12 |
| *** electrofelix has joined #zuul | 08:50 | |
| *** bhavik1 has quit IRC | 10:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add /nodes and /nodes.json to webapp https://review.openstack.org/499969 | 11:00 |
| *** hashar has joined #zuul | 11:01 | |
| *** hashar is now known as hasharAway | 11:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add /node-list to the webapp https://review.openstack.org/499969 | 11:03 |
| *** _ari_|conf is now known as _ari_ | 11:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 12:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 12:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 12:02 |
| tristanC | I'm now running nodepool-launcher with python3, which conveniently enables to run both zuul-server/nodepoold and zuul-executor/nodepool-launcher on the same node :-) | 12:10 |
| *** dkranz has joined #zuul | 12:40 | |
| mordred | tristanC: \o/ :) | 13:10 |
| Shrews | mordred: with https://review.openstack.org/499755, i'm seeing 'item1' and 'complex1' being output solo in the job stream for some reason | 13:14 |
| Shrews | mordred: any idea why? | 13:14 |
| Shrews | mordred: http://logs.openstack.org/55/499755/3/check/zuul-stream-functional/57043b9/stream-files/stream-job-output.txt | 13:14 |
| Shrews | 2017-08-31 20:57:45.167544 , for example | 13:16 |
| Shrews | oh, that happened in the previous review, too. hrm. | 13:18 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle logging ansible python errors properly https://review.openstack.org/499397 | 13:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove -vvv from playbook invocation for log streaming test https://review.openstack.org/499753 | 13:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for debug statements https://review.openstack.org/499608 | 13:21 |
| mordred | Shrews: it's cause log streaming isn't deterministic | 13:24 |
| mordred | Shrews: the stdout from the job gets output whenever it gets output | 13:24 |
| mordred | Shrews: NOW - I'm a little concerned that we only see item1 in the loop and not item2 or item3 | 13:24 |
| Shrews | mordred: that was my next question :) | 13:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't output complex items in the summary line https://review.openstack.org/499755 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 13:31 |
| mordred | Shrews: ^^ slightly silly perhaps, but Ijust realized when we're looking at logging changes we should consider how they look for multinode too | 13:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 13:45 |
| Shrews | jeblair: sorry for my ignorance here, but reading (and re-reading) your zuul-cloner email, i'm not sure i grasp why the current zuul-cloner won't work and we need a shim to replace it. Maybe it's because i've never used it. | 13:58 |
| Shrews | and maybe it's because i've only had 1 cup of coffee | 13:58 |
| *** hasharAway has quit IRC | 14:11 | |
| mordred | Shrews: so - the thing is that zuul-cloner does what is being done on the executor then pushed now | 14:14 |
| mordred | Shrews: so existing zuul-cloner won't work because existing zuul-cloner fetches git refs from zuul based on ZUUL_ env vars | 14:15 |
| mordred | and in v3 there is no zuul git server to fetch refs from | 14:15 |
| mordred | conceptually, however, the thing a user is trying to do when running zuul-cloner "I want the repos to be on this machine in the correct branch states" is still valid of course - it's just that most of those git states will have already been put onto thebuild node | 14:16 |
| mordred | so a shim that knows there are src repos with all the good stuff in src/ will let folks transition those jobs to remove their zuul-cloner calls | 14:17 |
| mordred | Shrews: it's also an issue becaue manyof the calls happen in devstack hook scripts that people have already in their own repos, so we can't even just get really clever with migration tool | 14:18 |
| Shrews | mordred: so, as an example, http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/bifrost.yaml#n18 | 14:20 |
| Shrews | mordred: all of those named repos will already exist on the node (b/c your migration script will create a v3 job to do that?). | 14:20 |
| Shrews | mordred: so we need the shim to realize that and do the right thing for them | 14:21 |
| mordred | Shrews: that's right | 14:28 |
| Shrews | i did not realize that v2 zuul could serve git repos | 14:29 |
| mordred | that's actually how all the git got on to the nodes - and was a problem for some of the third-party ci systems like gozer at hp | 14:30 |
| mordred | as they wanted to run their zuul service on machines inside the hp firewall, but use nodes in the public cloud as build resources | 14:30 |
| mordred | but since v2 expected the build node to clone from zuul - well, firewall | 14:30 |
| Shrews | and they couldn't call home. yah, can see that being an issue | 14:31 |
| mordred | so all we need from the new zuul-clone is just for it to know how to clone/copy/link things from src/{{ zuul.project.canoncal_name }} to . | 14:33 |
| mordred | and then let people konw they can just skip that step altogether | 14:33 |
| Shrews | mordred: seems easy enough. i'll take a stab at it. i assume just replacing the current zuul-cloner code is what we want here? | 14:35 |
| mordred | jeblair: two thoughts - a) we could print a warning every time new zuul-cloner is run with a message like "you don't need this tool anymore, repos are in src already and jobs can/should have required-projects:" | 14:36 |
| mordred | Shrews: yah - I think we need a standalone script with no depends so it's easy to wget/cp into place | 14:36 |
| jeblair | mordred: yeah, and we can use logstash to track those | 14:37 |
| mordred | jeblair, Shrews: ^^ zuul-cloner today takes an argument which is the base location from which to clone repos ... should we keep that and have new zuul-cloner just clone master of those repos from that location if they aren't in src ? | 14:37 |
| jeblair | mordred: re 14:33 < mordred> so all we need from the new zuul-clone is just for it to know how to clone/copy/link things from src/{{ zuul.project.canoncal_name }} to . | 14:37 |
| mordred | jeblair, Shrews: that way if we get required-projects wrong in migration it won't break people's jobs? | 14:37 |
| mordred | or do you think it should just hard-fail so that people can konw they need to udpate a job? | 14:38 |
| jeblair | mordred: not necessarily "." -- zuul-cloner species a destination which may not be '.'. but that's not too hard. | 14:38 |
| mordred | jeblair: indeed | 14:38 |
| jeblair | mordred: no, we should have it fail, but we should have it say to add the project to 'required-projects'. | 14:38 |
| mordred | also - not fully related to this - but I note that most of our invocations of zuul-cloner list git://git.openstack.org ... should we maybe make a sed patch to replace those with https:// instead? | 14:39 |
| mordred | jeblair: ++ | 14:39 |
| jeblair | mordred: because the job will be wrong (ie, it may be missing prepared changes) otherwise. | 14:39 |
| jeblair | mordred: if we weren't about to get rid if it, yes. :) | 14:39 |
| mordred | well - it's not just zuul-cloner | 14:39 |
| mordred | also a bunch of enable_plugin heat git://git.openstack.org/openstack/heat | 14:39 |
| jeblair | yeah, but let's save that for a rainy day? | 14:40 |
| mordred | k | 14:40 |
| * Shrews doesn't think it can get much rainier than mordred's home state, atm | 14:43 | |
| mordred | Shrews: I agree with you | 14:46 |
| mordred | Shrews, jeblair: you may find http://logs.openstack.org/49/500049/2/check/zuul-stream-functional/ea1ca5f/job-output.txt.gz interesting - I changed the stream test job to be a 3 node job so that we could see 2 nodes of output (although in retrospect I probably could have just changed the playbook to use "all" and havethe controller ssh to itself - but whatev) | 15:07 |
| mordred | mainly so that we could also see how the logging works or doesn't for multinode | 15:07 |
| mordred | gha | 15:08 |
| mordred | I meant: http://logs.openstack.org/49/500049/2/check/zuul-stream-functional/ea1ca5f/stream-files/stream-job-output.txt | 15:08 |
| Shrews | mordred: cool. will have to modify the greps to include node name | 15:11 |
| dmsimard | mordred: some projects are using ZUUL_ vars though, so those are gone too ? | 15:11 |
| dmsimard | http://codesearch.openstack.org/?q=ZUUL_&i=nope&files=&repos= | 15:12 |
| *** EmilienM has quit IRC | 15:12 | |
| *** EmilienM has joined #zuul | 15:12 | |
| dmsimard | Also, some projects tend to look if Zuul cloner is present to determine if they're running in a gate environment | 15:15 |
| dmsimard | Some examples: http://codesearch.openstack.org/?q=.*(which%7Cif.*-e.*).*cloner.*&i=nope&files=&repos= | 15:16 |
| dmsimard | jeblair: ^ | 15:16 |
| dmsimard | slightly simplified search http://codesearch.openstack.org/?q=.*(which%7Cif.*).*cloner.*&i=nope&files=&repos= | 15:17 |
| *** dmsimard has quit IRC | 15:19 | |
| *** dmsimard has joined #zuul | 15:19 | |
| mordred | dmsimard: the ZUUL_ vars are considered deprecated in v3 - we need to make a role that produces them / provides them to things - and we may want to come up with a flag we can provide that people can check to see if they're running inside a zuul or not | 15:28 |
| mordred | dmsimard: maybe if [ $USER = "zuul" ] ... ;) (not really though, as someone else running a different zuul could have the user account on nodes be not-named-zuul | 15:31 |
| dmsimard | mordred: yeah, there's also people running 3rd party CI which won't be on zuul v3 yet | 15:32 |
| mordred | yup | 15:32 |
| dmsimard | I'll point it out on the ML thread so that we don't forget we have users relying on different ways to detect if zuul cloner or zuul vars are available. | 15:33 |
| dmsimard | whew, RDO is finally out | 15:44 |
| dmsimard | I can have a life again | 15:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul.timeout variable to jobs https://review.openstack.org/500114 | 15:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update fetch-sphinx-output to collect contents of directory https://review.openstack.org/500116 | 15:56 |
| *** rbergeron has joined #zuul | 16:04 | |
| qwc | hrm, did anyone read my question earlier today? | 16:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 16:27 |
| jeblair | qwc: i'm sorry, i don't have any experience with submodules. i'm not sure zuul will handle them correctly. we have avoided their use in openstack due to the complexity. | 16:46 |
| mordred | implementing support for them should be easier to get right in v3, since the git operations are contained and not in job content anymore - but detecting and dealing with speculative states between parent and submodule would be very non-trivial | 16:54 |
| mordred | especially once you add in the gerrit tracking submodule magic and in-tree job definitions | 16:56 |
| mordred | qwc: all that is to say there is not a general solution currently. however, since you know the specific parent repo and submodules, you could treat them as any other repos in a multi-repo job and use zuul to get parent and submodule repo onto the machine separately (ignoring that the submodule is a submodule- just treat it as a second project) | 16:58 |
| mordred | qwc: then have your job definition start by copying/moving the repo from zuul for the submodule into the correct subdir in your parent repo's source dir | 16:59 |
| mordred | (and in fact, once you do that you'll be halfway to just treating the two as separate repos without a submodule relationship that need integration testing between them) | 17:00 |
| *** electrofelix has quit IRC | 17:01 | |
| *** harlowja has quit IRC | 17:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix python3 encoding issues with zuul_afs https://review.openstack.org/500137 | 17:36 |
| pabelanger | jeblair: mordred: python3 fixes for zuul_afs and excludes^. Write as bytes and encode as utf8. | 17:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 17:40 |
| dmsimard | I have some time again now that we finally shipped RDO. Any low hanging fruits I can gnaw my teeth on ? | 17:49 |
| dmsimard | ahhhhhh maybe after food. That's a good idea. I'll do a round of reviews as well. | 17:52 |
| Shrews | dmsimard: any idea what's up with this? http://logs.openstack.org/32/499732/6/check/zuul-stream-functional/13002be/job-output.txt.gz#_2017-09-01_17_44_27_377780 | 17:55 |
| dmsimard | Shrews: looks like a broken pipe error by rsync but you know that already :D | 17:57 |
| dmsimard | ara-output is the directory the integration job writes to | 17:57 |
| dmsimard | Maybe the directory isn't there ? Maybe were confusing the control and the subnode ? | 17:57 |
| dmsimard | "/home/zuul/ara-output\" failed: No such file or directory (2) | 17:58 |
| Shrews | oh, maybe the last failing task of the functional playbook has something to do with it | 17:58 |
| jeblair | devstack-legacy-tempest-dsvm-neutron-full SUCCESS in 1h 33m 04s | 17:59 |
| dmsimard | Ah maybe it prevented the ara generate command from running, yeah | 17:59 |
| dmsimard | jeblair: woot | 17:59 |
| jeblair | mordred: patchset 28 of https://review.openstack.org/497699 passes; i just pushed up patchset 29 which reworks things a little bit to try to converge on something that's closer to a form we'll want for the migration script. i think one more evolution past that will probably make it so that the devstack jobs don't need to be much different than any other auto-converted job. | 18:01 |
| Shrews | hrm, i wonder if that just caught a bug. the 'always' block of mordred's failure playbook doesn't seem to be present in the output | 18:01 |
| Shrews | hrm, yeah. i think this is a logging bug | 18:08 |
| dmsimard | let me see | 18:08 |
| jeblair | Shrews, dmsimard: te Generate ARA html task is in the playbook that failed, so it didn't run. | 18:08 |
| jeblair | s/te/the/ | 18:09 |
| jeblair | that's why there's no ara-output directory | 18:09 |
| jeblair | Shrews: so yeah, i think you're right -- validation failed, aborted main playbook, ara did not run, then the rsync pull of ara-output failed in the post playbook | 18:09 |
| dmsimard | Shrews: this should be wrapped in a block: https://review.openstack.org/#/c/498209/14/playbooks/zuul-stream/functional.yaml | 18:09 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix python3 encoding issues with zuul_afs https://review.openstack.org/500137 | 18:09 |
| dmsimard | the ara generate and gzip should be in an always | 18:09 |
| dmsimard | let me send a patch | 18:10 |
| Shrews | jeblair: yeah. my test is expecting output from the always block in https://git.openstack.org/cgit/openstack-infra/zuul/tree/playbooks/zuul-stream/fixtures/test-stream-failure.yaml?h=feature/zuulv3 , but it isn't there | 18:11 |
| jeblair | dmsimard: if you move 'ara generate' to an always, and the main block succeeds but it fails, will the playbook still exit non-zero? | 18:11 |
| jeblair | dmsimard: if you move 'ara generate' to an always, and the main block succeeds but ara fails, will the playbook still exit non-zero? | 18:11 |
| jeblair | (clarified ^) | 18:11 |
| *** harlowja has joined #zuul | 18:12 | |
| dmsimard | If an error occurs in a "rescue" or "always" directive inside a block, it will fail, yes | 18:12 |
| jeblair | cool, that plan sounds good then :) | 18:12 |
| dmsimard | If there is an error in the "block" directive, it will run the "rescue" directive if there is one and then run the "always" directive | 18:13 |
| dmsimard | If nothing in the rescue and always directive fail, the playbook exits 0 | 18:13 |
| jeblair | (i was considering whether it would be appropriate to move the 'ara generate' to a post playbook. it depends on whether we think we're "testing" ara as part of this test. i tend to think we are, so we should keep it in the main playbook if possible. | 18:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Always generate the ARA report, even on failure https://review.openstack.org/500159 | 18:16 |
| dmsimard | Shrews: something like that ^ | 18:16 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Always generate the ARA report, even on failure https://review.openstack.org/500159 | 18:17 |
| dmsimard | jeblair: blocks are awesome when using the include_role task instead of the playbook role directive | 18:18 |
| dmsimard | jeblair: because it allows you to wrap one or more entire roles and properly catch/exit as required | 18:18 |
| dmsimard | jeblair: for example: https://github.com/rdo-infra/weirdo/blob/219d6f065495fcfab68aa90d191f62d247bb7f92/playbooks/packstack-scenario001.yml#L23-L37 | 18:18 |
| dmsimard | So in that provided example, we'll run two roles 1) install packstack and 2) recover logs | 18:19 |
| jeblair | dmsimard: yeah, i thought so -- we use them in zuulv2.5, but i just wanted to double check | 18:19 |
| dmsimard | However, if #1 fails, logs won't be recovered because it's the next task -- so we have a rescue that runs the log recovery and then properly fails | 18:19 |
| jeblair | pabelanger, clarkb: https://review.openstack.org/500114 will be handy for devstack job conversion | 18:22 |
| mordred | jeblair: ++ | 18:24 |
| jeblair | i'm pushing https://review.openstack.org/499854 through | 18:26 |
| jeblair | that changes the base job to use the cached repos | 18:26 |
| mordred | dmsimard: maybe both things should be in the block - so that if there's a mistake that causes the first one to fail we get the ara output? | 18:26 |
| jeblair | so keep an eye out for breakages cauesed by that | 18:26 |
| dmsimard | mordred: in the context of the integration job, the first ansible playbook run is supposed to pass | 18:27 |
| dmsimard | so I suppose if it fails we want to know about it and not exit zero :) | 18:27 |
| Shrews | mordred: so, trying to find out where the missing output went from job-output.txt... if i have my local env setup correctly, i see the output in the generated file, but not in http://logs.openstack.org/32/499732/6/check/zuul-stream-functional/13002be/stream-files/stream-job-output.txt | 18:27 |
| Shrews | mordred: admittedly, i'm using a different playbook to test block-always | 18:27 |
| jeblair | dmsimard: i don't think that change is right -- *that's* not the task that's failing | 18:28 |
| mordred | dmsimard: well - the second one is supposed to "fail" - but it's not supposed to fail the functional playbook because we have failed-when in there | 18:28 |
| jeblair | right that ^ | 18:28 |
| jeblair | dmsimard: it's failing in Shrews' proposed change because he validates output, and he found a bug | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add helper script for doing local log streaming tests https://review.openstack.org/500161 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix spacer-lines to work with multi-node and items https://review.openstack.org/500162 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop output start and end times for each task https://review.openstack.org/500163 | 18:28 |
| dmsimard | ahhh | 18:28 |
| mordred | Shrews: ^^ I made you a script - because I got annoyed having to look in my scrollback | 18:29 |
| mordred | but also so that testing log streaming locally doesn't have to be quite so arcane | 18:29 |
| dmsimard | Shrews: ok I slightly misunderstood the failing issue -- were you going to set up that block/always then ? | 18:29 |
| jeblair | mordred, Shrews, dmsimard: we're all agreed that these should actually be unit tests, right? | 18:29 |
| mordred | no | 18:29 |
| jeblair | and we're going to do that after the ptg? | 18:29 |
| Shrews | mordred: great, just when i put all the pieces in place by hand :/ | 18:29 |
| jeblair | great! | 18:29 |
| Shrews | mordred: i mean... thanks! :) | 18:29 |
| jeblair | we should have talked about that before merging these changes with so little discussion. | 18:29 |
| jeblair | i think these should be unit tests. | 18:30 |
| Shrews | dmsimard: no | 18:30 |
| dmsimard | jeblair, mordred: I'm planning to copy/paste some stuff over from ARA's unit tests to add unit tests for the zuul callbacks | 18:30 |
| jeblair | right, i thought that ^ was the plan | 18:30 |
| dmsimard | the "hard" work of faking ansible resources and calling callback methods is already done so mostly copy pasta | 18:30 |
| Shrews | jeblair: unit tests would cover these nicely. i thought it would be good to have this in place now since it's so easy | 18:30 |
| Shrews | b/c, well, it's finding bugs now :) | 18:31 |
| jeblair | Shrews: right, that i can agree with. i just thought since there's more and more structure going into the tree around this that it was worth checking in on the plan | 18:31 |
| dmsimard | Shrews: +1, it's low effort and gives us something in the meantime | 18:31 |
| jeblair | if the plan is do this until we have time to swing around post-ptg and add (functional) unit tests, that's great. :) | 18:32 |
| mordred | jeblair: I believe as much of this as is possible should be in unittests | 18:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.timeout variable to jobs https://review.openstack.org/500114 | 18:32 |
| mordred | jeblair: but iirc there are some things we legit cannot test wrt ansible interaction in unittests because of local execution issues - and also sometimes it's useful to have a human-readable result (like docs-draft) when discussing impact of visual choices | 18:33 |
| jeblair | mordred: agreed. perhaps after we import dmsimard's framework (and possibly with some tricks of our own) we can manage to solve both of those problems in a local-only context. | 18:35 |
| mordred | jeblair: ++ | 18:36 |
| dmsimard | It's quite possible my framework ends up horrible and you real python developers show me how it's done but at worst it'll give us a starting point and we also end up improving ARA's tests -- win/win :D | 18:37 |
| mordred | also, once we have that we can likely move the zuul-stream job we made to a check-experimental or something like that | 18:37 |
| dmsimard | I like to say "I don't know what magicmock is, or what it does, but I'm happy it's here" | 18:38 |
| mordred | or, I guess we could put a path on it so that it runs if zuul_stream is touched - since that's the primary times when we want to see what the effect of the change would be with our eyes | 18:38 |
| jeblair | mordred: we can also ^ yeah that | 18:38 |
| mordred | in fact, honestly - we could do that now :) | 18:38 |
| jeblair | dmsimard: i have a pretty skeptical opinion of mocks. we try to run things "for real" as much as possible. | 18:38 |
| jeblair | zuul runs a lot of ansible-playbook processes :) | 18:39 |
| jeblair | in its unit tests | 18:39 |
| dmsimard | jeblair: oh, yeah, my implementation mostly just mocks classes/nametuples so I don't have to re-write the entire object. I've looked at doing literal imports from ansible too and create "real-life" objects (tasks, etc.) but haven't got around to it yet. | 18:39 |
| dmsimard | I don't mock functions or hide implementation details | 18:40 |
| dmsimard | I wouldn't know how, anyways | 18:40 |
| dmsimard | That stuff makes my brain explode | 18:40 |
| jeblair | yeah, i think it's a skill worth not having | 18:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only run zuul-stream testing on callback changes https://review.openstack.org/500168 | 18:42 |
| pabelanger | mordred: jeblair: can you remind me, with a child job, should I expect the a parent run playbook to be executed, or do we only execute a single run playbook | 18:45 |
| mordred | single run | 18:45 |
| Shrews | i need to extricate myself from this annoying coffee shop. bb shortly | 18:45 |
| pabelanger | mordred: okay, and why is that again? | 18:46 |
| *** hashar has joined #zuul | 18:46 | |
| mordred | pabelanger: so if you make a child job that inherits from a job thathas a run playbook, the child job's run takes over | 18:46 |
| *** hashar has quit IRC | 18:46 | |
| *** hashar has joined #zuul | 18:46 | |
| mordred | pabelanger: dunno if I could give a good why - it makes sense to my brain though - a child job to me says "I want qualities of the parent job but I want to run my own content instead" | 18:48 |
| mordred | pabelanger: are you running in to a case where it's not what you're expecting? | 18:48 |
| pabelanger | mordred: sure, give me a minute to push up some code, and maybe see why mutliple runs was what I was hoping for. | 18:49 |
| mordred | pabelanger: cool | 18:49 |
| pabelanger | mordred: but, this doable with a single child run too | 18:49 |
| mordred | pabelanger: (concrete examples are so much easier to reason about) | 18:49 |
| pabelanger | remote: https://review.openstack.org/500155 WIP: Refactor run-docs role to use tox role | 18:50 |
| pabelanger | mordred: ^so that is the code I was looking at | 18:50 |
| pabelanger | mordred: basically, openstack-doc-build parents to tox-doc, which then parents to tox, tox has a run playbook which will call tox role | 18:51 |
| pabelanger | however, because of a single run in the job | 18:51 |
| pabelanger | https://review.openstack.org/#/c/500155/3/playbooks/tox/docs.yaml needs to be written where I call the tox role myself | 18:52 |
| pabelanger | where, if multiple runs we supported, I'd just have run-docs in openstack-zuul-jobs tox/docs.yaml playbook | 18:52 |
| pabelanger | not a huge issue, but could result is deduping of data | 18:52 |
| pabelanger | I think this goes back to the discussions of having job playbooks be reusabled, a top of roles being reusable | 18:53 |
| mordred | pabelanger: well - two things | 18:54 |
| mordred | pabelanger: a) you need to override the run playbook anyway as the run playbook for tox-docs is actually fundamentally incorrect for openstack | 18:54 |
| mordred | pabelanger: (so replace is actually important in this case) | 18:54 |
| mordred | pabelanger: but b) the run-docs content you've got after this split looks like post-run content to me | 18:55 |
| pabelanger | mordred: a) yes, we can achieve that by setting tox_extra_args: -vv python setup.py build_sphinx unless there is something else I am forgetting | 18:55 |
| pabelanger | mordred: b) ya, that is possible too | 18:55 |
| pabelanger | perhaps in this case, moving that into post-run, will do what I wanted | 18:56 |
| jeblair | pabelanger: back to the abstract question, if you're a visual person, perhaps this will help: https://docs.openstack.org/infra/manual/zuulv3.html#job-inheritance | 18:56 |
| jeblair | pabelanger: an interesting thought experiment is: if, say, the unittests job were to run its main playbook, where in that sequence would it run? | 18:57 |
| jeblair | pabelanger: (i can think of no satisfactory answer, which is among the reasons we don't attempt it) | 18:57 |
| mordred | pabelanger: tox_envlist too ... but I think once you move b) to being a post-run, then you can just set tox_extra_args and tox_envlist on the job and go back to having openstack-doc-build use the run playbook from the tox base job | 18:57 |
| pabelanger | jeblair: as I now understand it, it wouldn't today. However, if it did, I would expect it to run between tox-py27 pre-run and tox-py27 run. | 18:58 |
| jeblair | pabelanger: why not between run and post-run? | 18:58 |
| pabelanger | however, I think mordred post-run comment is likely the better way to organize this issue, I can test that quickly enough. | 18:59 |
| pabelanger | jeblair: Hmm, thinking | 18:59 |
| pabelanger | jeblair: in my mind, unittests is the parent, so, I'd expect that to run before the child, in this context | 19:00 |
| jeblair | pabelanger: okay, but i'd ask you to consider that's an arbitrary choice that might work for some contexts, but not others. moreover, i think the concept of inheritance is somewhat nullified if you can't override the behavior. as we've seen in this example, once the tasks in a job are correctly deconstructed into pre/main/post buckets, being able to override the main portion is key to the whole process. | 19:02 |
| jeblair | pabelanger: that construct allows us to make a basic "publish docs" job for any language, and then inherit replacing the run playbook for specific langugages, eg, python-docs, java-docs, etc | 19:03 |
| pabelanger | jeblair: yup, I agree with that. It wasn't thing I actually thought about, until I was looking at the openstack-doc-buid job, and took me a few moments to understand why tox - run playbook wasn't run | 19:04 |
| mordred | jeblair: could I get you to look at https://review.openstack.org/#/c/499809/ real quick? | 19:05 |
| jeblair | mordred: this process has, surprisingly enough, convinced me that bundling things in secrets comes as close as possible to our opposing goals of facilitating in-tree management and reusability. | 19:08 |
| jeblair | mordred: i feel like you are convinced otherwise | 19:08 |
| jeblair | mordred: i think we need to discuss this as both infra-team and zuul-team | 19:09 |
| jeblair | mordred: but there are several approaches that *work* | 19:09 |
| Shrews | umm, didn't we fix the "worker found in a dead state" thing? | 19:09 |
| jeblair | mordred: and it's desirable to land one of them so we don't (and i'm not using the word casually this time) bikeshed on this. | 19:09 |
| jeblair | mordred: so i'm happy to +2 those and discuss this later (at the ptg seems ideal). but if there's a possibility we choose to stick with bundled secrets, do you want to unbundle them now at the risk of rebundling them later? | 19:10 |
| jeblair | mordred: or would you rather keep them bundled then unbundle later? | 19:11 |
| pabelanger | Shrews: I understood a new version of python fixed the issue, we should confirm we are still using that on zuul-executors | 19:12 |
| jeblair | Shrews: we did so by installing a python that mordred built (and has in a ppa) on ze01; perhaps it got "upgraded" | 19:12 |
| jeblair | SpamapS: know anything new about the python SRU? | 19:12 |
| Shrews | jeblair: i thought that was the case (was unsure if we were still running the fixed version). just saw the error on https://review.openstack.org/500162 | 19:12 |
| SpamapS | jeblair: it's sitting in the queue still. SRU team only drops everything for Criticals, and this is just a High unfortunately. :-/ | 19:14 |
| jeblair | Shrews: oh, that's not zuul dying, that's ansible-on-node dying | 19:15 |
| jeblair | Shrews: that node won't have a correct python | 19:15 |
| jeblair | Shrews: can probably fix by installing/upgrading python from the ppa | 19:16 |
| jeblair | which isn't in our mirror | 19:16 |
| SpamapS | jeblair: https://launchpad.net/ubuntu/xenial/+queue?queue_state=1&queue_text= <-- the queue it's sitting in | 19:16 |
| jeblair | SpamapS: thanks! | 19:17 |
| pabelanger | jeblair: Shrews: we should be able to do the same thing we did for bwrap in tools/setup-tests.sh with test-setup role | 19:18 |
| jeblair | mordred: er, i have to grab lunch now; chat when i get back? | 19:18 |
| SpamapS | jeblair: once it gets accepted it should appear on this report https://people.canonical.com/~ubuntu-archive/pending-sru.html (and we should get notifications if we're subscribed to the bug) | 19:18 |
| pabelanger | jeblair: mordred: to loop back to discuss a few minutes ago, do you mind looking at 500155,3 and 500155,4, and indicate which way would be easier for you to support. Both will produce the same result | 19:20 |
| mordred | jeblair: sure! | 19:24 |
| mordred | pabelanger: oh wow - can we verify from someone whether or not we still actually need the HUDSON_PUBLISH_DOCS env var? | 19:26 |
| mordred | :) | 19:26 |
| pabelanger | mordred: ya, I haven't looked into that yet. But sure, still need to covert out the zuul.env vars also | 19:27 |
| pabelanger | mordred: TIL: http://git.openstack.org/cgit/openstack/neutron/tree/doc/source/conf.py#n76 | 19:28 |
| pabelanger | mordred: so, we'd likely need to work with docs team and clean that up | 19:28 |
| mordred | pabelanger: oy. ok. later for that :) | 19:29 |
| mordred | pabelanger: I definitely like patchset 4 better | 19:29 |
| mordred | pabelanger: although I left acode review | 19:29 |
| pabelanger | mordred: ah, thanks. I'll push up a change after jeblair has a chance to look | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:34 |
| kklimonda | with zuul, if I have two repositories A and B, and two patchsets Aa and Ba that are dependent on each other (think two repositories creating a single build), when I create a one way dependency (Aa Depends-On: Ba) would zuul handle it by not building the Bb patchset, and building both Aa and Bb in a single step? If not, if there is anything we can do to achieve that? | 19:35 |
| Shrews | mordred: w00t. got your test-logs.sh to work. took some magicalness to source all the right repos and the right versions | 19:36 |
| kklimonda | s/Bb/Ba/g | 19:37 |
| mordred | kklimonda: zuul does not currently support cyclic dependencies such as that- instead we propose one Aa, then a second Bb depends-on Aa - then if there needs to be a consumption of Bb in A, a third Ab depends-on Bb | 19:39 |
| mordred | adding support for co-dependent changes would be possible and has been discussed, but it's been on the backburner because OpenStack has decided that it would prefer that such co-dependent changes be disallowed | 19:41 |
| Shrews | ok, found the bug | 19:43 |
| mordred | \o/ | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 19:44 |
| Shrews | mordred: fyi, it was the "- block: - always:". changed it to "- block: always:" | 19:45 |
| Shrews | weird that ansible doesn't barf on that | 19:46 |
| mordred | Shrews: yah. weird | 19:46 |
| mordred | Shrews: but love the latest version of the patch | 19:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 19:53 |
| Shrews | dmsimard: why did you WIP https://review.openstack.org/500159? lgtm | 19:59 |
| mordred | pabelanger, Shrews: coupla easy ones? https://review.openstack.org/#/c/499364 https://review.openstack.org/#/c/489758/ | 19:59 |
| mordred | Shrews: also - you might find https://review.openstack.org/#/c/489719/ interesting - it allows running tox unittests against git sources from zuul | 20:01 |
| mordred | or, it theoretically does so at least | 20:01 |
| mordred | right now it breaks spectacularly :) | 20:01 |
| Shrews | mordred: can a template inherit from another template? looks like that might be a better setup, if so | 20:03 |
| Shrews | s/better/simpler/ | 20:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 20:03 |
| mordred | Shrews: I don't think so? | 20:03 |
| mordred | Shrews, pabelanger: ooh, https://review.openstack.org/#/c/498916/ can go in now too | 20:05 |
| * mordred hands Shrews and pabelanger pies | 20:05 | |
| Shrews | then we could have a publish-to-pypi (that does it quietly), and a publish-to-pypi-announce. but this works too | 20:05 |
| kklimonda | mordred: hmm, yes - splitting affected reviews into 3 parts so that the cycle is broken is something that I want to propose. We'll see how it goes :) | 20:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 20:10 |
| mordred | kklimonda: cool! our basic rationale is that it's a good habit to be in especially for distributed systems where asking someone to upgrade two systems simultaneously might be extra hard | 20:11 |
| clarkb | it also aids with bisectability | 20:11 |
| mordred | yah | 20:11 |
| SpamapS | tomato, tomato, extra hard, impossible... let's call the whole thing off | 20:11 |
| kklimonda | mordred: our usecase is a bit different, we have a large C++ codebase that is unfortunately split into two repositories, where both are part of a single build process, and are tightly dependent on each other. | 20:13 |
| clarkb | kklimonda: have you seen openstack? ;) | 20:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 20:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add helper script for doing local log streaming tests https://review.openstack.org/500161 | 20:15 |
| kklimonda | clarkb: touche ;) | 20:15 |
| mordred | kklimonda: yah - this is a pain we know :) | 20:15 |
| kklimonda | I've always been under an impression that openstack modules can at least be built in isolation though | 20:15 |
| kklimonda | (and tested with mocks) | 20:16 |
| clarkb | it depends on how you want to define built in python land | 20:16 |
| clarkb | to test openstack we install and deploy something like 40 tightly integrated python repositories | 20:17 |
| clarkb | in a single "build" | 20:17 |
| clarkb | though we've trimmed down from that high water mark its closer to 15 now I think | 20:17 |
| jeblair | mordred, pabelanger: back; pabelanger which thing did you want me to look at? | 20:19 |
| jeblair | mordred: and what do you think of what i said re secrets / how do you want to proceed? | 20:20 |
| pabelanger | jeblair: 500155,3 and 500155,4, would like to see if you have a preference. Both should (minus the bug mordred pointed out) do the same thing | 20:21 |
| mordred | jeblair: hrm. so - once you're done looking at the pabelanger patches - unfortunately the process got me to the opposite place as you - I really liked your suggestion of putting the specific values into the playbooks in the base job and thought that hit the right balance for my personal sensibilities | 20:24 |
| mordred | jeblair: I'm not opposed to just keeping it as it is now, it's just a bunch more duplicated opaque secret bundles | 20:24 |
| jeblair | mordred: mostly, i think this is fine for infra, but a difficult pattern to use in zuul-jobs. if we did so, we would be forced to use site variables. maybe that's okay. | 20:26 |
| mordred | yah - I don't think it's the pattern we shoudl always use for sure | 20:27 |
| jeblair | (it's just a tradeoff; site variables are slightly less flexible, but they are good for low-level zuul-jobs) | 20:27 |
| mordred | jeblair: and maybe it's ok for the pile of copy/pasta of secret data to exist for now, since ultimately we hope to solve several of them with static nodes | 20:27 |
| mordred | maybe it would annoy me less if it was actually different keys :) | 20:29 |
| jeblair | mordred: yeah, though not log fileserver obviously :) | 20:29 |
| jeblair | right, and at least in the cases where we have different servers, they probably *should* be different keys :) | 20:29 |
| jeblair | mordred: okay, i'm convinced we have a shared understanding of the problems at this point; i suggest we talk about this in earnest at the ptg with more people, and you pick the direction we go for now. | 20:31 |
| mordred | jeblair: I've abandoned the patchesfor now - will just add three new secrets to the file | 20:31 |
| jeblair | mordred: ok | 20:31 |
| jeblair | mordred: i am at least glad we got to the point were we have a complete viable alternative to look at, thanks. i don't think we understood the whole problem space until then. | 20:32 |
| mordred | jeblair: \o/ | 20:33 |
| jeblair | mordred, clarkb, pabelanger: i'd like to merge https://review.openstack.org/497699 which adds the devstack-legacy job. it's running now, and it's very close, i think to what we want to have ready for the migration script. mordred can you take a look at it with that in mind? | 20:42 |
| pabelanger | looking | 20:43 |
| mordred | jeblair: yup- looking | 20:43 |
| jeblair | mordred: i actually think the entire playbook can just morph into our "compatability" playbook for all auto-migrated jobs. | 20:45 |
| mordred | jeblair: that's pretty - and yes, I think you're right - although there's something that can totally screw the yaml parser when we inline converted stuff and I don't remember what it is ... | 20:47 |
| mordred | jeblair: +2 from me - and nice work | 20:47 |
| jeblair | mordred: apostrophies ? | 20:47 |
| mordred | yah. I think so | 20:48 |
| jeblair | mordred: but yeah, maybe we end up making that a script | 20:48 |
| pabelanger | jeblair: +2, nice work | 20:48 |
| mordred | jeblair: at the very least if it's not PRECISELY the same it can be the model for it | 20:48 |
| jeblair | yeah, *structurally* i think that's what we're looking for | 20:51 |
| jeblair | i mean, the nodepool stuff should be a role | 20:51 |
| jeblair | and the env vars should be more inclusive | 20:51 |
| jeblair | and the script should probably be a script | 20:51 |
| jeblair | but all the elements are there :) | 20:51 |
| mordred | yup. it doesn't have to be pretty | 20:51 |
| mordred | although it IS | 20:51 |
| mordred | jeblair: I think with pabelanger's doc publisher stuff adn that legacy script that actually covers all the jobs from both nodepool and zuul | 20:51 |
| jeblair | two down! ;) | 20:52 |
| mordred | jeblair: :) | 20:52 |
| pabelanger | Yay | 20:52 |
| mordred | jeblair: honestly with a publish-to-pypi template, python-jobs, python35-jobs and infra-publish-jobs - that's actually a LOT of projects down ;) | 20:53 |
| jeblair | yep | 20:53 |
| mordred | jeblair: I just thought of a fun synthetic test we could write for project-config's zuul.yaml | 20:56 |
| mordred | jeblair: parse the yaml and grab every secret that contains a ssh_known_hosts and fqdn and try to ssh to it with that host key | 20:57 |
| mordred | don't even need to decrypt the secret - just see if the ping to the host works and returns the same host key - like a lint check | 20:57 |
| mordred | (I say this while I'm putting in three different host keys and trying very hard to make sure I'm not copy-pasting them backwards or something) | 20:58 |
| jeblair | mordred: ++ | 20:59 |
| mordred | jeblair: also - I was thinking about / pondering an update to the zuul.d code ... largely because as I work with secrets I keep wanting to put them in their own file because they're so big ... | 21:01 |
| mordred | jeblair: what if we supported a file named for each object type - so zuul.d/secrets.yaml zuul.d/pipelines.yaml - and if we find any of them in the zuul.d dir load them first before run-partsing the rest? | 21:02 |
| mordred | too weird? | 21:02 |
| jeblair | mordred: not actually necessary for one -- regardless of the order of the files, the *types* are always loaded in the same order | 21:03 |
| jeblair | mordred: file order only matters if you have, say, jobs split over multiple files | 21:03 |
| jeblair | mordred: so you can create secrets.yaml and jobs.yaml now and achieve the exact same behavior as a single file | 21:04 |
| mordred | jeblair: oh - good point - and excellent | 21:04 |
| mordred | jeblair: so if we did that and had the migration script emit a legacy.yaml - that would naturally load correctly with jobs.yaml loading first then legacy.yaml yah? (which is, I assume, the order we want them loadded) | 21:06 |
| jeblair | mordred: yes (they'll both have jobs, so lexography matters here, but j < l so that's the order) and yes (that sounds like a reasonable order) | 21:07 |
| *** hashar has quit IRC | 21:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 21:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 21:40 |
| pabelanger | mordred: openstack-build-docs for shade: http://logs.openstack.org/01/500201/1/check/openstack-doc-build/9771ffc/html/ looks good | 21:58 |
| mordred | pabelanger: woot! | 22:32 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove project stanza https://review.openstack.org/499278 | 23:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 23:24 |
| *** olaph has quit IRC | 23:36 | |
| *** olaph has joined #zuul | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!