| *** jkilpatr has joined #zuul | 00:31 | |
| *** dkranz has quit IRC | 01:49 | |
| *** isaacb has joined #zuul | 05:26 | |
| *** dpawar has joined #zuul | 05:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add tenant column to the buildset reporter table https://review.openstack.org/484256 | 06:15 |
|---|---|---|
| *** dpawar has left #zuul | 06:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add tenant column to the buildset reporter table https://review.openstack.org/484256 | 06:28 |
| *** amoralej|off is now known as amoralej | 07:03 | |
| *** dpawar has joined #zuul | 08:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Fix zuul command default function https://review.openstack.org/484279 | 08:06 |
| *** isaacb has quit IRC | 08:15 | |
| *** dpawar has quit IRC | 08:26 | |
| *** hashar has joined #zuul | 08:26 | |
| *** dpawar has joined #zuul | 08:36 | |
| *** isaacb has joined #zuul | 09:10 | |
| *** dpawar has quit IRC | 09:42 | |
| *** dpawar has joined #zuul | 09:58 | |
| *** isaacb_ has joined #zuul | 10:01 | |
| *** isaacb_ has quit IRC | 10:01 | |
| *** dpawar has quit IRC | 10:01 | |
| *** isaacb has quit IRC | 10:04 | |
| *** isaacb has joined #zuul | 10:05 | |
| *** jkilpatr has quit IRC | 10:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: WIP: Add jobs dashboard https://review.openstack.org/466561 | 10:42 |
| *** hashar has quit IRC | 10:59 | |
| *** hashar has joined #zuul | 11:01 | |
| *** jkilpatr has joined #zuul | 11:11 | |
| *** dkranz has joined #zuul | 12:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 12:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets https://review.openstack.org/484000 | 12:42 |
| *** jkilpatr has quit IRC | 12:43 | |
| *** jkilpatr has joined #zuul | 12:43 | |
| Shrews | morning zuul-folk | 12:50 |
| *** amoralej is now known as amoralej|lunch | 12:52 | |
| *** dpawar has joined #zuul | 13:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 13:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_environment to allow users access to shell variables https://review.openstack.org/483935 | 13:10 |
| pabelanger | morning! | 13:13 |
| pabelanger | mordred: the stack at https://review.openstack.org/#/c/484046/ could use a review. Fixes ansible-lint on our roles for openstack-zuul-jobs because of path issues, also adds --syntax-check for quickly testing failures locally | 13:16 |
| pabelanger | remote: https://review.openstack.org/483987 Create openstack-py35 job with upper-constraints | 13:20 |
| pabelanger | mordred: ^ is also ready for review for openstack-py35 job on shade | 13:20 |
| *** dpawar has left #zuul | 13:29 | |
| *** dpawar has joined #zuul | 13:33 | |
| *** amoralej|lunch is now known as amoralej | 13:39 | |
| rbergeron | mornings are terribull | 14:36 |
| *** isaacb has quit IRC | 14:45 | |
| dmsimard | I know I discussed this some while ago but my memory is failing me. Was there going to be anything in Zuul v3 to improve how child jobs might be able to re-use artifacts or data from a parent job ? | 14:53 |
| dmsimard | Right now we sort of hack around this where parent jobs upload the logs at a known location. To do this, we had to override the default LOG_PATH: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul/zuul_functions.py#L17-L33 | 14:54 |
| dmsimard | This way, we can retrieve artifacts from the parent job because the child job can reconstruct the URL to the logs | 14:55 |
| pabelanger | dmsimard: yes, DAG jobs should help with this. Which has landed in zuulv3. We still need to work on the artifact handling | 15:00 |
| dmsimard | pabelanger: What's DAG stand for ? | 15:01 |
| pabelanger | so top level job will build artifact once, then children could all fetch the same artifact | 15:01 |
| pabelanger | Directed Acyclic Graphs | 15:02 |
| dmsimard | Okay, good to know. Thanks. | 15:03 |
| pabelanger | the, parent would build / store artifacts some place (clarkb had an interesting idea the parent job would create a cinder volume), then all children could get artifacts from it | 15:03 |
| pabelanger | however, that is an issue for us with multi clouds | 15:04 |
| pabelanger | and would pin jobs to a single region | 15:04 |
| dmsimard | Right -- so the way we do it sort of makes sense and is not that much of a hack, then, I guess | 15:04 |
| dmsimard | We rely on uploading logs to a known and expected location and then using that expected location in child jobs | 15:04 |
| pabelanger | parent could then setup some ansible variables to pass to children | 15:04 |
| pabelanger | dmsimard: right, passing of data between jobs _should_ be eaiser in zuulv3 | 15:05 |
| jeblair | yeah, we're making progress there. one thing that would really help is the ide of a cleanup job -- one which runs at the end even if its parents fail. that can delete the cinder volume / swift container / whatever. | 15:05 |
| dmsimard | The problem we had to work around was that the default LOG_PATH uses ZUUL_UUID in the log URL which is unknown to the child job. That's why we changed it to ZUUL_REF which is unique to the entire job set. | 15:05 |
| jeblair | that shouldn't be hard to add once we finish the higher priority stuff | 15:05 |
| jeblair | dmsimard: yeah, we'll call it something different, but we'll still have the buildset uuid (the unique part of zuul ref) available for jobs) | 15:06 |
| dmsimard | alright | 15:06 |
| *** isaacb has joined #zuul | 15:56 | |
| *** isaacb has quit IRC | 16:02 | |
| jeblair | jlk: 483597 +3. enjoy vacation! | 16:02 |
| *** hashar is now known as hasharAway | 16:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Simplify _deleteLocalBuild parameters https://review.openstack.org/484413 | 16:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Support UUID as builder identifier https://review.openstack.org/484414 | 16:10 |
| Shrews | jeblair: SpamapS: 484414 was based on an idea you two had several weeks back. Hope that review generates some discussion. | 16:10 |
| SpamapS | Shrews: sounds like trouble. ;) | 16:10 |
| Shrews | i hope not :) | 16:11 |
| tobiash | Shrews: sounds cool, does it make sense to also be able to directly specify the uuid in the config file? | 16:14 |
| *** dpawar has quit IRC | 16:14 | |
| tobiash | Shrews: (thinking of config being an ansible template this might be a use case) | 16:15 |
| jeblair | tobiash, Shrews: the ansible could write out the file contents...? | 16:15 |
| Shrews | tobiash: i hadn't considered that, but might be another good feature to add if others like it | 16:16 |
| tobiash | jeblair: I'm currently deploying zuul/nodepool config via an ansible template | 16:16 |
| tobiash | (which can also include host specific variables) | 16:16 |
| tobiash | Shrews, jeblair: but as we'll (hopefully soon) transitioning to use openshift for deployment I don't know currently how I would add the uuid in this setup then | 16:18 |
| tobiash | maybe just writing the pod id to some file | 16:18 |
| Shrews | pabelanger: also, you might be interested ^^^ since we discussed this in our current infra situation last week | 16:18 |
| pabelanger | Shrews: thanks, will look shortly | 16:19 |
| Shrews | tobiash: we could likely support that as yet another config option that is mutually exclusive to the new one | 16:19 |
| Shrews | tobiash: so either 'builder-id' or 'builder-id-file', but not both | 16:19 |
| jeblair | i think either is fine -- i think the file location is nice because it lets nodepool write the value. there shouldn't be a need to move the value from one host to another, in fact, that's kind of the point. :) | 16:19 |
| tobiash | Shrews, jeblair: that takes me to a similar question, can builders be scaled down without leaking images? | 16:20 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Support dynamic dependent pipeline additions https://review.openstack.org/483597 | 16:20 |
| Shrews | tobiash: probably not. that is probably a topic that should be covered in nodepool operation docs | 16:21 |
| Shrews | (which still need writing) | 16:21 |
| jeblair | tobiash: maybe by pausing the image on the builder to be removed, waiting for it to age out, then removing the config for it. | 16:21 |
| jeblair | Shrews: ^? | 16:21 |
| tobiash | ok, will have to test this once I have an openshift setup | 16:22 |
| Shrews | jeblair: yeah, combined with manual deletion instead of waiting, i think | 16:22 |
| Shrews | either or | 16:22 |
| Shrews | in either case, it involves more than just "stopping the builder" | 16:22 |
| Shrews | :) | 16:22 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update to tox_envlist variable https://review.openstack.org/483934 | 16:24 |
| tobiash | Shrews: so is it possible to delete an image of a dead/crashed/... builder (which is not going to be survived)? | 16:25 |
| Shrews | tobiash: well, yeah, but will take manual steps since the builder obviously cannot clean up after itself in that case | 16:26 |
| jeblair | yeah, we may want to add a helper command for that | 16:26 |
| pabelanger | Shrews: left question about cache file | 16:27 |
| Shrews | *nod* | 16:28 |
| Shrews | pabelanger: responded! | 16:31 |
| jeblair | mordred, pabelanger: i'm blocked on https://review.openstack.org/484073 for the base job move | 16:43 |
| jeblair | mordred, pabelanger: also, moderately less so on https://review.openstack.org/484075 | 16:43 |
| pabelanger | Shrews: jeblair also, added another comment on 484414 | 16:44 |
| pabelanger | looking | 16:44 |
| pabelanger | jeblair: -1 will comment | 16:46 |
| pabelanger | 484075 | 16:46 |
| jeblair | pabelanger: then why didn't you update this job? | 16:49 |
| pabelanger | jeblair: oversight. I thought I checked openstack-zuul-jobs | 16:52 |
| jeblair | pabelanger: i uploaded a new version | 16:52 |
| pabelanger | many places to now look for job changes | 16:52 |
| pabelanger | jeblair: if you are also interested: 483987 for openstack-py35 job | 16:53 |
| mordred | jeblair: +A | 16:53 |
| mordred | jeblair, pabelanger: we should also get this: https://review.openstack.org/#/c/479390/ landed and a restart so that we can start using secrets | 16:53 |
| pabelanger | 484027 is working shade | 16:53 |
| mordred | pabelanger: yes! that stack is next on my list - exciting! | 16:53 |
| pabelanger | so, don't want to block 479390 but maybe some testing in a follow patch? | 16:58 |
| pabelanger | would be helpful to ensure we don't leak things into inventory and secrets.yaml is what we expect | 16:58 |
| jeblair | pabelanger: oh i think i see -- we do test secrets, but you're suggesting an additional test that ensures that they don't show up anywhere except the secrets file, and that it is outside the work directory. | 16:59 |
| pabelanger | jeblair: ya | 17:01 |
| pabelanger | mordred: So, I am starting to think our zuul-jobs: eg http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/playbooks/tox/pre.yaml should not be installing dependencies. The use case would be folks over in redhat that might want to install tox from an RPM. I know we could add logic into the job to decide to install from pip or rpm, but I feel that might be too much business logic in the zuul-jobs | 17:05 |
| pabelanger | playbook. If we made the assumption tox would already be installed, then we can let the user decide how to install it. | 17:05 |
| pabelanger | tox/pre.yaml also assumes that pip is installed | 17:05 |
| pabelanger | I don't think we want it to go though the sets needed to set that up too | 17:06 |
| SpamapS | Shrews: I left some more feedback for you. | 17:06 |
| Shrews | danke | 17:06 |
| mordred | pabelanger: I definitely don't think we should add a ton of logic to do package vs. pip in those base jobs ... | 17:07 |
| mordred | pabelanger: however, I think we could do something similar to the bindep role where we check to see if it's installed on the node and if not install it into a venv in a tempdir perhaps? | 17:08 |
| mordred | pabelanger: OR ... if we keep it like it is there, then folks who want tox from package can include that install in their base job or in their image | 17:10 |
| mordred | pabelanger: this is a good question I like thinking about it :) | 17:10 |
| jeblair | Shrews: i have a suggestion buried in the conversation on line 24; don't want you to miss it. | 17:10 |
| Shrews | jeblair: line 24 of ... ? | 17:12 |
| mordred | pabelanger: so - motivation for installing from my side is that I want people to be able to install zuul, configure it to point to the zuul-jobs repo for a set of base jobs, configure nodepool to return stock distro images from their cloud and have tox-py27 work | 17:12 |
| jeblair | Shrews: configuration.rst in 484414 | 17:12 |
| mordred | pabelanger: obviously balancing that with the ability for deployers to make more specific decisions and for openstack optimizations to work and to be the priority are essential | 17:12 |
| pabelanger | mordred: Right, I think the job should work out of box, like you said. Let me think about how we could do that | 17:13 |
| mordred | pabelanger: but I personally dont' want zuul users to get caught in the crossfire of ubuntu v. redhat v. pip if the thing they want is just to run tox | 17:13 |
| mordred | pabelanger: ++ | 17:13 |
| Shrews | jeblair: oh right. yeah, we both had the same feeling about /etc (i commented there as well, if you missed it) | 17:13 |
| jeblair | Shrews: yep. i didn't want you to miss the idea of just using image-build-dir to contain the uuid file | 17:14 |
| pabelanger | mordred: but one thing I have been defaulting to for roles I build, is if roll uses git for example, I expect git to be installed before you use the role. Same would apply to tox in this case. Let me see how we could do that and keep them working out of box | 17:14 |
| pabelanger | mordred: in fact, let me start with bindep first. Since I already have a role to setup bindep outside of zuul-jobs | 17:16 |
| Shrews | jeblair: yup, thx | 17:16 |
| *** harlowja has joined #zuul | 17:17 | |
| jeblair | mordred: what's the lowdown on dropping the returned json from build log? | 17:32 |
| *** dpawar has joined #zuul | 17:32 | |
| mordred | jeblair: I need to finish up a patch that I'm almost done with | 17:32 |
| jeblair | kk | 17:33 |
| *** dpawar_ has joined #zuul | 17:46 | |
| *** dpawar has quit IRC | 17:47 | |
| jeblair | pabelanger, mordred: oops, my local test was too simple. can you look at https://review.openstack.org/484441 please? | 18:08 |
| pabelanger | +2 | 18:09 |
| *** dpawar_ has left #zuul | 18:09 | |
| mordred | +A | 18:10 |
| *** amoralej is now known as amoralej|off | 19:13 | |
| jeblair | pabelanger, mordred: w00t! https://review.openstack.org/483593 has the base-test job working. after lunch, i'll move it over to be the new base. | 19:20 |
| pabelanger | jeblair: yay | 19:21 |
| *** jkilpatr_ has joined #zuul | 19:25 | |
| *** jkilpatr has quit IRC | 19:27 | |
| mordred | jeblair: woot! | 19:30 |
| *** harlowja has quit IRC | 19:32 | |
| dmsimard | mordred: oi o/ | 19:37 |
| dmsimard | mordred: probably worth looking into adding https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/__init__.py#L76 to the Zuul callback somewhere | 19:38 |
| dmsimard | i.e, https://github.com/openstack/ara/commit/491dd7a142136bdc63c0e7e65ef1d21c3a2f3f86 && https://access.redhat.com/security/cve/CVE-2017-7473 | 19:39 |
| pabelanger | pretty sure we are using _dump_results | 19:41 |
| dmsimard | pabelanger: not afaict https://github.com/openstack-infra/zuul/blob/d8a7dcc2e9e515862f1c29db1390504acb2a400e/zuul/ansible/callback/zuul_stream.py | 19:42 |
| dmsimard | you have a dump_result_dict method which also drops the internal ansible keys but doesn't drop the content if no_log is set | 19:42 |
| pabelanger | dmsimard: propose a patch :) | 19:44 |
| mordred | dmsimard: there should be a patch related to that ... | 19:44 |
| dmsimard | mordred: I remember seeing one in the mqtt callback plugin | 19:44 |
| mordred | yah - it's now directly in the log_message method for us - although I'm finishing up yet-another set of patches there | 19:45 |
| mordred | dmsimard: thanks for the heads up! turns out that's really important :) | 19:47 |
| dmsimard | I can send the patch if you'd like -- just not sure in which shape you'd like | 19:48 |
| mordred | dmsimard: we don't need it at the moment- we take care of it in _log_message | 19:48 |
| dmsimard | mordred: oh, let me double check | 19:48 |
| dmsimard | mordred: you're right :D | 19:49 |
| mordred | dmsimard: I have a todo-list item to refactor that file - it has gotten a bit messy :) | 19:49 |
| dmsimard | mordred: no worries, I was following up on the CVE with infosec and was looking for places it could still be hiding in. | 19:50 |
| pabelanger | dmsimard: we should fix mqtt in openstack-infra, do you mind submitting a patch there? | 20:02 |
| pabelanger | fungi: ^ | 20:03 |
| dmsimard | pabelanger: there's already a patch unless it hasn't merged yet | 20:03 |
| * dmsimard looks | 20:03 | |
| dmsimard | pabelanger: https://review.openstack.org/#/c/461214/ | 20:03 |
| pabelanger | k, we should land that | 20:03 |
| pabelanger | +2 | 20:04 |
| *** jkilpatr_ has quit IRC | 20:05 | |
| fungi | glad to see i've been forward-thinking for once! | 20:08 |
| fungi | that patch is like 2.5 months old. i'd already forgotten i'd written it | 20:08 |
| *** SotK is now known as SotK_ | 20:16 | |
| pabelanger | mordred: jeblair: now that 473764 has landed (zuul.d) did we want to bikeshed about creating zuul-jobs/zuul.d/jobs/tox.yaml vs toplevel zuul.yaml | 20:18 |
| pabelanger | and unittests.yaml for zuul.d | 20:18 |
| *** SotK_ has left #zuul | 20:19 | |
| *** SotK has joined #zuul | 20:19 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: Add support for nodepool testing of openSUSE 42.3 https://review.openstack.org/484476 | 20:19 |
| pabelanger | https://review.openstack.org/#/c/483935 and https://review.openstack.org/#/c/483936/ could use a review. Reworking for openstack-py35 jobs | 20:26 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: Add support for nodepool testing of openSUSE 42.3 https://review.openstack.org/484476 | 20:27 |
| pabelanger | also starts to remove openstack specific settings from zuul-jobs | 20:27 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: Add support for nodepool testing of openSUSE 42.3 https://review.openstack.org/484476 | 20:34 |
| jeblair | pabelanger: let's wait until the repo is a bit bigger before we split the zuul file. also, we need to add support for the split to zuul-sphinx. | 20:35 |
| jeblair | pabelanger: (also, i don't think file-per-job is necessarily a pattern we would want to adopt. something more like "python-jobs.yaml" and "java-jobs.yaml" sounds more useful to me. but again, later) | 20:36 |
| *** jkilpatr has joined #zuul | 20:37 | |
| pabelanger | ack | 20:39 |
| jeblair | remote: https://review.openstack.org/484484 Update base job to content in base-test | 20:40 |
| jeblair | pabelanger, mordred: ^ | 20:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add sample base job https://review.openstack.org/484485 | 20:45 |
| jeblair | pabelanger, mordred: remote: https://review.openstack.org/484487 Remove last remaining roles | 20:48 |
| mordred | jeblair: that fails zuul - do we want to do the dance? or just power it through? | 20:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ensure we load roles for linting https://review.openstack.org/484488 | 20:50 |
| jeblair | mordred: oh, i think it's only failing because it actually needs its dependency to land | 20:50 |
| jeblair | it should work once we actually update the base job | 20:51 |
| jeblair | mordred: actually... let me dig into that real quick and make sure it's running what it should be. | 20:52 |
| jeblair | meantime, i pushed a new PS of 484484 with a minor doc correction | 20:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Include ansible-playbook syntax-check for tox pep8 https://review.openstack.org/484490 | 20:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Rename pep8 to linters for tox https://review.openstack.org/484491 | 20:54 |
| pabelanger | mordred: ^ more linting goodness | 20:54 |
| jeblair | mordred, pabelanger: if you could +2 but not +3 484484 while i double check things, that would be great. i'll +3 it when ready. | 20:55 |
| pabelanger | jeblair: Yup, +2 from me already | 20:55 |
| pabelanger | jeblair: q: does zuul_return need to be the last task? Or could the be move say to first playbook? | 20:56 |
| *** harlowja has joined #zuul | 20:56 | |
| pabelanger | not that I am asking for us to move it :) | 20:56 |
| jeblair | pabelanger: it can be anywhere and can be called multiple times | 20:56 |
| pabelanger | cool | 20:56 |
| Shrews | b 11 | 20:56 |
| Shrews | doh | 20:56 |
| jeblair | Shrews: you sank my battleship! | 20:56 |
| pabelanger | Shrews: bb8 | 20:56 |
| Shrews | wait... this isn't irc bingo? | 20:57 |
| jeblair | mordred always wins because he has the most boards/windows | 20:57 |
| mordred | :) | 20:59 |
| jeblair | mordred: oh! zuul is failing that change because openstack-zuul-roles is no longer a roles repo! | 20:59 |
| pabelanger | jeblair: is it limited to trusted playbooks too? | 20:59 |
| jeblair | pabelanger: nope | 21:00 |
| pabelanger | cool | 21:00 |
| jeblair | however, i think it still should have checked out the current master branch of openstack-zuul-roles which still does have roles... | 21:02 |
| jeblair | oh! nope, that was for the unittest pre playbook, which is speculative | 21:03 |
| jeblair | mordred: so, erm, we should listen to what zuul is telling us here and remove that role repo from that job before we merge the openstack-zuul-roles cleanup. | 21:04 |
| jeblair | mordred: and i'm going to correct myself again: | 21:04 |
| jeblair | mordred: that's still coming from the base job. so we do just need to merge the base job change. | 21:05 |
| mordred | \o/ | 21:05 |
| mordred | jeblair: do you think it's worth adding this to the list of places where debugging the error from zuul could use more verbosity/help? or that it's enough of an edge-case to not be worth it? | 21:06 |
| jeblair | basically -- the base job playbooks are correctly running with the branch tip. the unittest playbooks are running with the speculative change which has no roles. but they *all* run with "roles: openstack-zuul-jobs" because that's in the base job. | 21:06 |
| jeblair | mordred: i used 2 pieces of info to debug this: 1) the actual error: Unable to find role in /tmp/c569ae74cbac4cebbd06e3472f73d65f/ansible/pre_playbook_1/role_0/openstack-zuul-roles. and 2) the debug log telling me which playbooks/roles repo checkouts were prepared in which places. | 21:08 |
| mordred | yah - it's possible that just practice can make that more evident from the actual error | 21:08 |
| jeblair | mordred: certainly surfacing #1 would be helpful (and is on the list -- 2001105) | 21:08 |
| mordred | jeblair: we are also in the position currently of second-guessing zuul issues so that we can validate that it is behaving correctly | 21:09 |
| jeblair | mordred: yes, and that was certainly part of why i looked at #2 | 21:09 |
| mordred | jeblair: ++ | 21:09 |
| jeblair | mordred: i think we'll be exposing some of #2 in due course in build log debugging output. however, that won't help situations like this where we refuse to run a job entirely. | 21:11 |
| mordred | jeblair: yah | 21:11 |
| jeblair | mordred: i think we may just want to make the error we return to the user in #1 as helpful as possible (the exception already has the roles repo it's looking at) and hope that in cases where it crops up it's obvious what the problem is once it's pointed out. cases like *this* should be more rare. | 21:12 |
| *** dkranz has quit IRC | 21:17 | |
| jeblair | pabelanger: hypothesis: when we merged the --die-with-parent change, we "fixed" the ssh control delay by causing the control socket program to die with bubblewrap. | 21:22 |
| jeblair | pabelanger: i filed the story about the ssh control process issue on june 16, and we merged --die-with-parent on june 26, so i think the timing works. | 21:23 |
| pabelanger | ack | 21:25 |
| jeblair | pabelanger: i'd still like to fix it the way we discussed (and wrote in the story) so that we actually take advantage of the control socket across playbook invocations | 21:26 |
| jeblair | https://storyboard.openstack.org/#!/story/2001072 | 21:26 |
| jeblair | mordred: 484487 is green now | 21:42 |
| jeblair | mordred, pabelanger: we can merge that and 484485 | 21:42 |
| pabelanger | looking | 21:42 |
| pabelanger | +2 | 21:44 |
| jeblair | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| *** hasharAway has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create tox_environment to allow users access to shell variables https://review.openstack.org/483935 | 22:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add callback plugin to emit json https://review.openstack.org/484515 | 22:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only output result details on error https://review.openstack.org/484516 | 22:47 |
| mordred | jeblair: ^^ first stab at json output plugin - and then removing results printing from successful tasks | 22:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 22:55 |
| Shrews | with this extra minute, i now have so much more time for activities! | 23:00 |
| pabelanger | mordred: jeblair: Hmm, did something happen to zuul_workspace_root variable? | 23:11 |
| jeblair | pabelanger: yes, i removed it at mordred's suggestion | 23:11 |
| mordred | yah. it's not needed | 23:11 |
| pabelanger | Oh | 23:11 |
| pabelanger | ya, we still have jobs using it | 23:11 |
| pabelanger | let me checkup them | 23:11 |
| pabelanger | cleanup* | 23:12 |
| jeblair | pabelanger: we're just using relative paths and CWD now | 23:12 |
| pabelanger | k | 23:12 |
| pabelanger | actually | 23:13 |
| pabelanger | nevermind | 23:13 |
| jeblair | mordred: both change ^ lgtm | 23:15 |
| mordred | jeblair: I hope you enjoyed my complete cop-out on the hard problem :) | 23:17 |
| jeblair | mordred: which one? :) | 23:18 |
| mordred | jeblair: yes! | 23:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add more information on variables in jobs https://review.openstack.org/484530 | 23:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 23:31 |
| mordred | pabelanger: so - I'm not convinced that's openstack specific code | 23:32 |
| mordred | pabelanger: I think any zuul installation could be reasonably concerned about the size of test output being too giant | 23:33 |
| pabelanger | mordred: right, I can see that. I was thinking maybe it should be a check in unittests | 23:35 |
| pabelanger | I'm actually thinking maybe we should have a role to check filesize, fail if too large, then gzip | 23:35 |
| pabelanger | which gets called before we synchonize logs | 23:36 |
| mordred | ++ | 23:37 |
| mordred | pabelanger: we also need $something in the streamer itself | 23:37 |
| pabelanger | but, the ansible code did work: http://logs.openstack.org/20/484520/1/check/openstack-py35/aaebb1a/job-output.txt | 23:37 |
| mordred | pabelanger: because the logs can completely get away from us when there's an error which causes all tests to fail and collect ALL of their logs into the subunit stream :) | 23:37 |
| pabelanger | mind you, it wasn't 50MB | 23:37 |
| mordred | pabelanger: :) | 23:38 |
| pabelanger | mordred: looks like some wonkyness on our logger | 23:38 |
| pabelanger | oh, maybe that is the missing pretty json stuff | 23:39 |
| mordred | pabelanger: where? gotta timestamp? | 23:40 |
| pabelanger | mordred: 2017-07-17 23:35:34.996100 | 23:41 |
| pabelanger | http://logs.openstack.org/20/484520/1/check/openstack-py35/468d6d1/job-output.txt is a better example | 23:41 |
| pabelanger | 2017-07-17 23:24:48.309549 | 23:42 |
| pabelanger | failed task was successful | 23:42 |
| pabelanger | but we still outputted error msg | 23:42 |
| mordred | pabelanger: nod. that's likely a bug in the streamer - I should be able to reproduce that easily | 23:45 |
| mordred | pabelanger: AH - I see why | 23:46 |
| mordred | pabelanger: it's because that msg is part of teh input argument to the task which we are emitting int he task banner | 23:46 |
| pabelanger | Ah | 23:46 |
| mordred | pabelanger: so it's telling you that you ran "Fail if subunit is larger then 50MB." with the msg of "testrepository.subunit was > 50 MB of uncomp" ... | 23:46 |
| mordred | that's definitely a thing I'm working on cleaning up righ tnow - so that's a good test case, thanks | 23:47 |
| pabelanger | honestly, I am not much of a fan of output vars in the banner. I kinda like how default ansible does it today :) | 23:47 |
| mordred | pabelanger: yah - that's where I'm wanting to get back to - and why the new json output thing with an eye towards smart display of that | 23:48 |
| pabelanger | okay cool! I'll hold off on complaining then :D | 23:49 |
| mordred | no - please complain! it's easy to miss things :) | 23:49 |
| pabelanger | okay, I think I'm going to convert https://review.openstack.org/#/c/484518/ into a role to check filesize, compress things | 23:51 |
| pabelanger | it is passing | 23:51 |
| pabelanger | then expose some parameters to the job to allow the user to configure them | 23:52 |
| jeblair | pabelanger, mordred: do we care whether subunit files are of a certain size? i think we only care that job logs are of a certain size. so maybe this should be a base job role? | 23:52 |
| pabelanger | jeblair: do we care in openstack land about it? | 23:53 |
| pabelanger | I mean, could we remove the file check today? | 23:53 |
| pabelanger | from jenkins scripts | 23:53 |
| jeblair | pabelanger: oh we certainly care. i'm saying i don't think we care that *unittest* jobs don't get too big, what we actually care about is that any job doesn't get too big | 23:55 |
| mordred | jeblair, pabelanger: I definitely think it should be a base job role if we have it | 23:56 |
| jeblair | with zuulv3, we're in a good position to say "okay, just copy the build log, not all this other stuff" if the other stuff is too big | 23:56 |
| mordred | but I think there's "does this job produce too much content overall" and "does this job produce a console output that is too large" | 23:56 |
| mordred | yah | 23:57 |
| jeblair | i guess there's an argument to be made that it's still useful to restrict unit test jobs to 50mb even if we want to restrict all jobs to, say, 100mb. because the pathological case for unit tests is smaller than the one for other jobs. | 23:57 |
| jeblair | in which case, maybe we have both? | 23:58 |
| pabelanger | Right, I that was my logic for starting with openstack-zuul-jobs first, we likely only care in openstack-infra about that specific file atm. but, maybe the role to check filesize is in zuul-jobs | 23:58 |
| jeblair | pabelanger: i am 100% certain we, in openstack infra, care about restricting every single job on the system to not upload more than X MB. | 23:58 |
| pabelanger | right | 23:59 |
| jeblair | pabelanger: i do not know what that value is, but i know that we want to set it so that we do not get into the case we are in now where we have projects uploading gigantic log files and not even knowing. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!