| jeblair | so, in other words, from my pov, the base job version of this which implements the global limit is something we have to do. if folks *also* want to further limit unit test jobs, that's fine with me but i don't feel as strongly about it. | 00:00 |
|---|---|---|
| pabelanger | so, here was the step for the playbook I am writing | 00:00 |
| jeblair | pabelanger: the reason we only have the unittest limit is historical -- it was written when the only jobs we had were unit test jobs. | 00:00 |
| pabelanger | this might be easier to talk on pbx server | 00:02 |
| jeblair | pabelanger: maybe tomorrow then? it's past eod for me (and i reckon mordred too) | 00:02 |
| pabelanger | yup, that works | 00:03 |
| mordred | I'll be on a plane tomorrow | 00:07 |
| mordred | but also it is eod | 00:07 |
| mordred | happy to do more irc chatting - or if y'all want to pbx it without me that's cool | 00:07 |
| pabelanger | sure, let me think about what I need to type out in the morning | 00:08 |
| mordred | pabelanger, jeblair: the main thing from my end is that I'd love for us to _not_ have an openstack-py27 job at all - I'd like to avoid it if we can - even if that means putting somethig in the base job for now that's not _stricly_ applicable but that either wouldn't hurt or that we might take a todo-list item go add support somewhere for | 00:09 |
| mordred | so if we can, I'd like for the basic jobs like tox-py27 to only have an openstack version if it's just flat unavoidable | 00:09 |
| pabelanger | right, but adding things into base job means something like devstack would also get tox variables | 00:10 |
| mordred | our docs jobs is a good exapmle - the way we build docs doesn't map to anywhere else | 00:10 |
| mordred | pabelanger: well, by "base" here I mean zuul-jobs/tox | 00:10 |
| pabelanger | mordred: right, I'm really trying to resist the need for if/else logic in the zuul-jobs. | 00:12 |
| pabelanger | I think if we go down that path, they might grow pretty big and complex to manage | 00:12 |
| mordred | yah - and I agree with that sentiment in general | 00:13 |
| pabelanger | where if we added it into openstack-py27 for example, then we only need to worry about openstack things | 00:13 |
| mordred | however, I think a tox-py27 job that has detection of .testrepository, nose and py.test artifacts and can do all the right things with them is a good tox-py27 job | 00:13 |
| pabelanger | So, a good exersise would be to have bonnyci or tobiash try and use our tox role. Then, see what needs to be added and what it would look like into zuul-jobs | 00:14 |
| mordred | so some amount of complexity in some of these base jobs is a net win | 00:14 |
| pabelanger | Ya, I can see that | 00:14 |
| pabelanger | maybe I should be starting with check_sudo_usage | 00:16 |
| mordred | pabelanger: (also, I think these are the hard ones - translating the definitely-openstack stuff is piece of cake :) ) | 00:16 |
| pabelanger | _shrugs_ | 00:16 |
| pabelanger | mordred: ya, it is a little struggle atm :) | 00:16 |
| mordred | pabelanger: ++ | 00:17 |
| mordred | pabelanger: hopefully only a few of them will be ugly like this though | 00:17 |
| pabelanger | Ya, having a non openstack zuulv3 would be helpful. :D | 00:18 |
| pabelanger | k, EOD for me too | 00:19 |
| mordred | pabelanger: ++ | 00:34 |
| *** harlowja has quit IRC | 01:34 | |
| *** patrickeast_ has joined #zuul | 02:06 | |
| *** mattclay_ has joined #zuul | 02:07 | |
| *** toabctl_ has joined #zuul | 02:13 | |
| *** patrickeast has quit IRC | 02:13 | |
| *** toabctl has quit IRC | 02:13 | |
| *** rcarrillocruz has quit IRC | 02:13 | |
| *** mattclay has quit IRC | 02:13 | |
| *** patrickeast_ is now known as patrickeast | 02:14 | |
| *** toabctl_ is now known as toabctl | 02:14 | |
| *** mattclay_ is now known as mattclay | 02:14 | |
| *** rcarrillocruz has joined #zuul | 02:19 | |
| *** timrc has quit IRC | 02:45 | |
| *** timrc has joined #zuul | 02:52 | |
| *** timrc has quit IRC | 02:56 | |
| *** timrc has joined #zuul | 02:58 | |
| *** harlowja has joined #zuul | 03:40 | |
| *** harlowja has quit IRC | 04:54 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add tenant column to the buildset reporter table https://review.openstack.org/484256 | 05:33 |
| *** hashar has joined #zuul | 06:55 | |
| *** amoralej|off is now known as amoralej | 07:44 | |
| *** isaacb has joined #zuul | 07:45 | |
| *** isaacb_ has joined #zuul | 07:56 | |
| *** isaacb has quit IRC | 07:59 | |
| tristanC | pabelanger: mordred: is it the openstack-infra/zuul-jobs that would be re-usable? | 08:28 |
| *** isaacb_ has quit IRC | 08:48 | |
| SpamapS | tristanC: yes. | 09:25 |
| *** isaacb_ has joined #zuul | 10:04 | |
| *** xinliang has quit IRC | 10:33 | |
| *** xinliang has joined #zuul | 10:46 | |
| *** xinliang has quit IRC | 10:46 | |
| *** xinliang has joined #zuul | 10:46 | |
| *** jkilpatr has quit IRC | 10:46 | |
| *** amoralej is now known as amoralej|lunch | 11:05 | |
| *** jkilpatr has joined #zuul | 11:07 | |
| *** amoralej|lunch is now known as amoralej | 12:24 | |
| *** isaacb_ has quit IRC | 12:59 | |
| *** dkranz has joined #zuul | 13:05 | |
| *** hashar is now known as hasharAway | 13:15 | |
| *** persia__ is now known as persia | 14:01 | |
| pabelanger | morning | 14:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Rework tox -e linters https://review.openstack.org/484836 | 14:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Support UUID as builder identifier https://review.openstack.org/484414 | 15:04 |
| mordred | morning pabelanger ! | 15:07 |
| pabelanger | o/ | 15:07 |
| Shrews | pabelanger: so, in the interest of dropping py2 support from nodepool, what would it take to transition our nodepool builders in production to py3? | 15:16 |
| pabelanger | Shrews: I don't think it would be much work, we could either repurpose nb03 / nb04 or just bring up nb01 / nb02 under python3. We are already running xenial now | 15:19 |
| pabelanger | we need to update puppet-nodepool, but that should be straight-forward too | 15:19 |
| pabelanger | TL;DR: we could do it in a 1/2 I think | 15:19 |
| pabelanger | 1/2 day* | 15:19 |
| Shrews | pabelanger: and probably do 1 at a time, yeah? | 15:20 |
| pabelanger | if we do that, we should merge feature/zuulv3 to master too | 15:20 |
| pabelanger | Shrews: right | 15:20 |
| pabelanger | we can stop 1 builder now | 15:20 |
| pabelanger | to force the other to only build images | 15:20 |
| Shrews | pabelanger: merge feature/zuulv3 TO master? or the other way around? | 15:21 |
| pabelanger | ya, we should merge master to feature/zuulv3 first I guess, then back to master | 15:21 |
| pabelanger | but nb03 / nb04 are using master branch today | 15:22 |
| Shrews | launchers are pinned to a tagged release though, correct? | 15:23 |
| jeblair | pabelanger: we can't merge zuulv3 to master -- we're still running old nodepool master. | 15:23 |
| pabelanger | jeblair: oh, right. | 15:23 |
| pabelanger | so, would we consider running nodepool-builders from feature/zuulv3 in openstack-infra? | 15:23 |
| Shrews | we should do a master -> feature/zuulv3 at some point though | 15:24 |
| pabelanger | Shrews: nl01 is running from feature/zuulv3 | 15:24 |
| Shrews | pabelanger: right, but that's non-production | 15:24 |
| pabelanger | ya | 15:24 |
| jeblair | pabelanger: just cherry-pick the builder related changes to master | 15:24 |
| pabelanger | Shrews: which launchers are you referring too about pined | 15:25 |
| pabelanger | jeblair: I don't think that works in this case? Shrews was asking about dropping py27 support | 15:25 |
| jeblair | oh, that's probably major changes to the v2 launcher. | 15:26 |
| jeblair | but Shrews only asked about production builders | 15:26 |
| jeblair | which may be less work | 15:26 |
| * Shrews confused now | 15:27 | |
| jeblair | yep. i don't know what we're talking about. | 15:27 |
| Shrews | if we remove v2 support, that affects both production (builders there are using v3), and our dev environment | 15:27 |
| jeblair | Shrews: no -- production builders are running master | 15:28 |
| jeblair | so we can remove py27 support from feature/zuulv3 without affecting production launchers or builders | 15:28 |
| Shrews | jeblair: oh, we merged the builder changes back to master, didn't we? | 15:28 |
| jeblair | Shrews: yes | 15:28 |
| Shrews | ah, k. right. so yes, only dev environment is affected then | 15:29 |
| Shrews | which means only nl01 needs migrating, i think? | 15:29 |
| jeblair | yep | 15:30 |
| Shrews | that's easier | 15:31 |
| Shrews | pabelanger: If you want to point me to the things that need updating, I'm happy to attempt that. Not well versed in puppet, but can take a stab | 15:34 |
| pabelanger | Shrews: sure, we need to add python3 support to http://git.openstack.org/cgit/openstack-infra/puppet-nodepool you can see what we did in http://git.openstack.org/cgit/openstack-infra/puppet-zuul. And system-config will then use python3 http://git.openstack.org/cgit/openstack-infra/system-config | 15:35 |
| Shrews | pabelanger: cool thx. let me see if i can make sense of it | 15:36 |
| Shrews | ah, i think i see how the pieces fit | 15:46 |
| *** bhavik1 has joined #zuul | 15:46 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add more information on variables in jobs https://review.openstack.org/484530 | 15:52 |
| *** hasharAway has quit IRC | 15:59 | |
| *** hashar has joined #zuul | 15:59 | |
| pabelanger | mordred: so, if I understand right, you do not want to do: https://review.openstack.org/#/c/483936 | 16:00 |
| jeblair | mordred, pabelanger: what do we need to move https://review.openstack.org/479390 forward? should we add that test now? do we need more reviews? | 16:01 |
| pabelanger | mordred: eg: remove NOSE vars from tox role | 16:01 |
| pabelanger | jeblair: mordred: I don't want to block it, but think tests would be helpful. I don't mind +3 if we want to do a follow up | 16:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add sample base job https://review.openstack.org/484485 | 16:06 |
| *** bhavik1 has quit IRC | 16:25 | |
| *** hashar is now known as hasharDinner | 16:30 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add callback plugin to emit json https://review.openstack.org/484515 | 16:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only output result details on error https://review.openstack.org/484516 | 16:34 |
| jeblair | i'm taking another stab at the implied role change (where we automatically try to add the job's own repo as a zuul role repo) | 16:54 |
| pabelanger | jeblair: okay, since mordred is likely on a metal tube. Do you mind chatting a bit about the subunit file size? Right now, that lives in tox role. I think that is too specific to openstack right now. So, my thought was to move that into openstack-zuul-jobs. https://review.openstack.org/#/c/484518/ I actually think, this could be made into a role, that allows to check filesize of uncompressed data, error if | 17:05 |
| pabelanger | large, and gzip if not. Which might be useful to live in zuul-jobs as a role | 17:05 |
| pabelanger | jeblair: but right now I am blocked a little as I don't know which path to move forward on. The ansible code is written and works | 17:05 |
| * mordred waves from tube - I'm fine putting things in openstack-zuul-jobs for now so that we don't spin too much on this while I'm less reachable - but I also think pabelanger and jeblair chatting it through higher bandwidth could also be useful and whatever y'all come up with works for me | 17:08 | |
| pabelanger | mordred: right, we might have this dance for a bit between openstack-zuul-jobs and zuul-jobs for a while longer, but I'd think landing things in openstack-zuul-jobs first makes more sense, and I feel once in zuul-jobs (and people are using it) it will be harder to remove playbooks / roles out of zuul-jobs. | 17:11 |
| jeblair | mordred, pabelanger: okay, i took a look at that change. i'm guessing the openstack specific parts are: a) should a job fail on subunit too large? b) what should the threshold be? | 17:12 |
| jeblair | is that a good analysis? is "should it be gzipped?" also an openstackism, or is that universal? | 17:13 |
| pabelanger | jeblair: right to a and b. | 17:13 |
| pabelanger | to be, gzip should be handled at the log publisher setp | 17:14 |
| pabelanger | step* | 17:14 |
| pabelanger | otherwise, we might have a lot of duplicate code gzipping things in roles | 17:14 |
| jeblair | pabelanger: yeah, i think i agree with that. easiest to just gzip everything before upload, and i believe mordred has a change to do that. so let's discard that for now. that leaves a and b. | 17:15 |
| pabelanger | jeblair: sure, I'll look for the gzip code on review.o.o | 17:16 |
| jeblair | mordred: so how can we make a and b suitable for everyone in zuul-jobs? do we look for site-local variables for that, and if not found, default the behavior to "off"? | 17:16 |
| jeblair | pabelanger: oh, https://review.openstack.org/483611 was the change i was thinking of, but it's console log only. | 17:16 |
| mordred | jeblair: yes. OR - we omit it completely and deal with job content size differently | 17:16 |
| jeblair | mordred: right, yes, we should back up and ask ourselves whether we want a subunit size limit. | 17:17 |
| mordred | ++ | 17:17 |
| jeblair | or just a job size limit. | 17:17 |
| mordred | like, I don't think we specifically care about subunit size, as much as we care that jobs don't eat too much disk | 17:17 |
| mordred | I could be wrong about that though | 17:18 |
| jeblair | personally, i want a job size limit and don't care about subunit. if someone stands up and says infra wants a subunit size limit, i'm fine with adding it. | 17:18 |
| jeblair | fungi, clarkb: ^ ? | 17:18 |
| pabelanger | didn't we say yesterday that openstack jobs would care if subunit was 50MB? | 17:19 |
| clarkb | I think youay want arbitrary file limit as well as aggregate limit | 17:19 |
| clarkb | byt not necessarily a subunit specific thing | 17:19 |
| mordred | clarkb: do we specifically care about subunut size ourselves? | 17:20 |
| pabelanger | I can only think openstack-qa might? | 17:20 |
| mordred | (it's in our job now, but if we had job-size-limit would we care about subunit size limit?) | 17:20 |
| clarkb | mordred: I dont know that we do, opensta k health might though | 17:21 |
| jeblair | oshealth doesn't store the attachments, does it? | 17:21 |
| pabelanger | I don't think zuul-jobs would care about subunit files size, but we should support it for some openstack project job. That was the assumption I went with starting to work on it | 17:21 |
| fungi | summarizing the question: we have run-tox.sh right now limiting the maximum size of subunit files, and wondering whether that should be a general role vs something specific to openstack community tox jobs? | 17:24 |
| *** harlowja has joined #zuul | 17:24 | |
| jeblair | fungi: yes, but -- we're backing up a step and asking whether it's even required at all | 17:24 |
| jeblair | storytime | 17:24 |
| SpamapS | jeblair: subunit2sql discards attachments IIRC | 17:24 |
| jeblair | we added the 50mb limit 6 years ago when the only thing we were running was nova unit tests under jenkins, and someone did something silly and they went haywire | 17:24 |
| jeblair | some things *might have changed* since then | 17:25 |
| jeblair | as an operator of openstack's ci system, all i really care about is not running out of disk space | 17:25 |
| fungi | jeblair: thanks! i was in the process of trying to pickaxe through git history to see when/why it was added myself | 17:25 |
| jeblair | my inclination would be to say "all jobs can store up to X mb" and leave it at that. | 17:26 |
| jeblair | i think we need *that* regardless | 17:26 |
| fungi | yes, we'd already discussed wanting that feature | 17:26 |
| jeblair | so the first question is: is that sufficient for us, or do we *also* want something like the 50mb limit for subunit files. | 17:26 |
| pabelanger | So, easy mode is no and remove it | 17:26 |
| jeblair | my answer is i think the global limit is sufficient, but i'm happy for us to have the subunit limit as well if others see value. | 17:27 |
| fungi | so i think given the circumstances of the subunit cap addition history, it's reasonable to say the new aggregate log size cap solves the original problem in a more clean and generally-applicable fashion | 17:27 |
| jeblair | SpamapS: ok, so the overall size is probably not a big concern to subunit2sql (attachments tend to be where that comes from) | 17:28 |
| jeblair | pabelanger, mordred: okay, so way forward is don't worry about subunit size for now. just do an overall size check as part of the log upload. also gzip at the log upload as well. | 17:30 |
| jeblair | pabelanger, mordred: er... proposal ^ :) | 17:30 |
| pabelanger | okay, I can work on that | 17:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 17:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 17:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 17:44 |
| pabelanger | jeblair: mordred: ^ removes file check, moves gzip into fetch-testr-output | 17:45 |
| pabelanger | next up will be to make a subunit2html role | 17:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 17:53 |
| *** amoralej is now known as amoralej|off | 18:11 | |
| *** bhavik1 has joined #zuul | 18:13 | |
| *** deep-book-gk_ has joined #zuul | 18:30 | |
| *** deep-book-gk_ has left #zuul | 18:33 | |
| *** tobiash_ has joined #zuul | 18:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 18:34 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Write secrets into their own file, not into inventory https://review.openstack.org/479390 | 18:34 |
| pabelanger | jeblair: mordred: I am guess we don't care if a tox jobs uses sudo in zuul-jobs but we'd still like it for openstack-infra? | 18:35 |
| *** bhavik1 has quit IRC | 18:36 | |
| mordred | pabelanger: I vote for blocking sudo in both - this is a unittest job we're writing, I think we get to suggest some best practices while doing it :) | 18:37 |
| SpamapS | Heh.. story #2000773 ... still the best way to destroy your FireFox performance. | 18:38 |
| pabelanger | mordred: right, we should be able to control that in our playbook for the unit test, since we'd use becomes: true / false for sudo actions now | 18:38 |
| pabelanger | unless somebody adds sudo commands into tox.ini | 18:40 |
| pabelanger | then, the job should fail | 18:40 |
| mordred | ++ | 18:40 |
| mordred | yes. and if they want to do things with sudo - they can write another job to do that :) | 18:40 |
| pabelanger | ya | 18:41 |
| mordred | but I think being opinionated about sudo in tox.ini being bad is a gift we can give to the world ;) | 18:41 |
| pabelanger | mordred: basically, trying to see if we need to write jenkins-sudo-grep.sh right now as ansible books for the tox job. | 18:41 |
| pabelanger | to me, sudo is disabled by default and if they use sudo, the job will just fail | 18:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add unittest for secrets data https://review.openstack.org/484911 | 18:42 |
| mordred | pabelanger: yah - that also seems like a holdover from a time before we had disable sudo | 18:42 |
| mordred | pabelanger: this is fun - we haven't cleaned house in a while | 18:43 |
| pabelanger | ++ | 18:43 |
| mordred | jeblair, pabelanger: ^^ unit test for secrets.yaml | 18:44 |
| mordred | SpamapS: you like tests | 18:44 |
| SpamapS | yesiree | 18:45 |
| SpamapS | jeblair: so having finally caught up on the openstack running zuulv3 transition plans in that etherpad... I don't see much in https://storyboard.openstack.org/#!/board/41 beyond the devstack-gate roles refactoring that needs to get done before that. | 18:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove check_sudo_usage logic from tox https://review.openstack.org/484916 | 18:47 |
| SpamapS | So I wonder if we should flip our plan, leave zuulv3 as "stuff to say zuulv3 is ready for outside consumption" and make a new tag that is "stuff we need to do before Denver" | 18:47 |
| SpamapS | mordred: I especially like adding assertions to existing tests. :) | 18:52 |
| SpamapS | says more about coverage IMO | 18:52 |
| mordred | \o/ | 18:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 18:58 |
| SpamapS | mordred: fail :( | 19:03 |
| *** hasharDinner is now known as hashar | 19:07 | |
| mordred | SpamapS: yah - I forgot - the live tests don't keep all those files around - patch coming | 19:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 19:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add unittest for secrets data https://review.openstack.org/484911 | 19:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fail early if people attempt to add zuul vars or secrets https://review.openstack.org/484000 | 19:15 |
| SpamapS | so... | 19:21 |
| SpamapS | back some time ago | 19:21 |
| SpamapS | we had a plan to run control masters outside the bwrap | 19:21 |
| SpamapS | I actually think that's going to be super problematic. | 19:22 |
| SpamapS | Mainly because usually we let Ansible run SSH for us. | 19:22 |
| SpamapS | and thus, we get hostnames, users, ports, etc., from the inventory we build for ansible. | 19:22 |
| SpamapS | so, I think I've come up with a plan | 19:23 |
| SpamapS | which is to run this inside the bwrap to shutdown controlmasters: 'ansible -m ping -i {inventory} -e ansible_ssh_command="ssh -O exit" all' | 19:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 19:32 |
| mordred | SpamapS: as a subprocess essentially at the end of the thing? makes sense to me (ansible '*' -m ping ...) | 19:32 |
| SpamapS | mordred: yeah | 19:33 |
| SpamapS | it doesn't seem to work actually | 19:33 |
| mordred | boo | 19:38 |
| SpamapS | oh bah, it uses execve on that value | 19:38 |
| SpamapS | and it's not -e ansible_ssh_command but env var ANSIBLE_SSH_EXECUTABLE | 19:38 |
| SpamapS | so have to use a wrapper script | 19:38 |
| SpamapS | (lame) | 19:38 |
| SpamapS | yeah works with a wrapper | 19:39 |
| SpamapS | though ansible exits with an error since the command doesn't actually ssh to the remote host | 19:40 |
| SpamapS | oh hm | 19:42 |
| SpamapS | there are some mechanics deep in the ssh connection plugin to do this | 19:42 |
| SpamapS | what's a "meta task" ?? | 19:42 |
| SpamapS | oh nice! there's a meta module | 19:43 |
| SpamapS | that can influence the connection plugin | 19:43 |
| SpamapS | and has a 'reset_connection' | 19:43 |
| * SpamapS hopes | 19:43 | |
| SpamapS | so maybe we can just pop that into a playbook that we always run after | 19:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 19:44 |
| mordred | SpamapS: maybe last task of the base job? | 19:49 |
| SpamapS | yeah | 19:50 |
| SpamapS | well | 19:50 |
| SpamapS | no | 19:50 |
| SpamapS | because some playbooks run in and out of different bubblewraps | 19:50 |
| SpamapS | it has to be basically another playbook | 19:50 |
| SpamapS | that we run after every one finishes | 19:50 |
| mordred | ah. we need a playbook baked in to zuul that we run after each playbok | 19:50 |
| mordred | nod | 19:50 |
| SpamapS | it sort of works tho | 19:51 |
| mordred | SpamapS: that's super meta | 19:51 |
| mordred | totally | 19:51 |
| SpamapS | except it 'splodes with a TypeError | 19:51 |
| * SpamapS tries updating ansible | 19:51 | |
| mordred | SpamapS: SOOOOOOOOOOO - while you're considering things | 19:51 |
| mordred | SpamapS: (I promise, you're going to love this one) | 19:52 |
| SpamapS | hah, 2.3.0.0 exploded with a TypeError | 19:52 |
| SpamapS | 2.3.1.0 explodes with an IndexError | 19:52 |
| mordred | SpamapS: v2_playbook_on_stats in zuul/ansible/callback/zuul_stream.py is the last event ansible-playbook triggers at the end of a playbook run | 19:52 |
| SpamapS | actually we could probably do it with our evil action plugins | 19:53 |
| mordred | SpamapS: and has access to the inventory | 19:53 |
| mordred | well- the callback plugin is the thing that runs in the right context | 19:53 |
| SpamapS | ah hm | 19:53 |
| mordred | action plugins don't know "the playbook is over" - but the callback plugin does | 19:53 |
| mordred | SpamapS: we could, in fact, write a third callback plugin that ONLY does the ssh reset and add it to the list | 19:54 |
| mordred | so that zuul_stream doens't become any more un-understandable | 19:54 |
| SpamapS | yeah | 19:55 |
| SpamapS | I think that may make a lot more sense. | 19:55 |
| SpamapS | mordred: un-understandable reduces to derstandable | 19:55 |
| mordred | ++ | 19:55 |
| mordred | :) | 19:55 |
| SpamapS | oh this is dumb | 19:56 |
| SpamapS | so it uses -O stop | 19:56 |
| SpamapS | but not -O exit | 19:56 |
| mordred | hah | 19:56 |
| SpamapS | so the control master disconnects from the remote end | 19:56 |
| SpamapS | but does not actually exit | 19:56 |
| SpamapS | >:| | 19:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 19:56 |
| mordred | I bet that's the more 'normal' thing to want? | 19:56 |
| SpamapS | well yeah, it would help with things like having a bunch of phantom users on boxes | 19:57 |
| SpamapS | and not exiting means you can reuse things like loaded SSH keys | 19:57 |
| mordred | pabelanger: ooh! can I make sugestoinon that last patch? (I don't think I would have thought of us if you hadn't written that one) | 19:57 |
| SpamapS | but >:| | 19:57 |
| SpamapS | however | 19:57 |
| SpamapS | I think our callback might actually be able to reach in and just do the -O exit | 19:57 |
| mordred | ++ | 19:57 |
| * SpamapS will play with that | 19:57 | |
| pabelanger | mordred: most welcome | 19:58 |
| mordred | pabelanger: instead of post-task in playbook - make the pip freeze a handler that gets triggered from the tasks | 19:58 |
| mordred | pabelanger: that way it always runs at the end - but the role is still self-contained | 19:58 |
| mordred | pabelanger: oh- wait | 19:58 |
| mordred | pabelanger: NEVERMIND - your way is better | 19:58 |
| mordred | pabelanger: because your way the tox role can be used outside of zuul - where that freeze is less likely to be super useful | 19:59 |
| mordred | otoh - your thing does put it into the .otx/logs dir - so it's not like it would HURT | 20:00 |
| mordred | pabelanger: sorry - I may have geeked out slightly too much on that patch :) | 20:00 |
| mordred | pabelanger: another thought - while I'm thinking about it WAY too much ... | 20:00 |
| pabelanger | mordred: ya, I think this will be useful to share with devstack also. So trying to think about how we'd share it there too | 20:00 |
| mordred | well - my next random thought might be helpful there ... | 20:01 |
| jeblair | SpamapS: yeah, i think your ptg tag is a good idea. i'm about to start reading scrollback on the control socket thing now. back in a minute. :) | 20:02 |
| pabelanger | interesting | 20:04 |
| pabelanger | finger://ze01.openstack.org/d7bc75a315754c4bae4b4636c3f13f51 | 20:04 |
| pabelanger | that job finished almost 20mins ago | 20:04 |
| pabelanger | but we can still access info via finger | 20:04 |
| jeblair | SpamapS: i think i don't understand the problem with the plan outlined in 2001072. you said we let ansible run ssh for us so we get stuff from the inventory. i believe we will still do that. the proposal in the story is not to actually ssh anywhere, but rather, just to start the control socket process -- same as the agent process. | 20:05 |
| jeblair | SpamapS: i'm under the impression we can start a control socket process without actually connecting to any hosts. | 20:06 |
| pabelanger | mordred: jeblair what are your thoughts about maybe adding a zuul-executor group into our inventory file, so we can references delegate_to: zuul-executor over localhost. My logic, is so we can see which host we run task on in job-output.txt, over localhost: http://logs.openstack.org/17/484917/4/check/tox-linters/577433f/job-output.txt | 20:07 |
| mordred | pabelanger: sorry - lost reception | 20:07 |
| pabelanger | in our case, it would be ze01.openstack.org, not localhost any more | 20:08 |
| mordred | write it as a role that does pip-freeze and sets the output as a fact - and that can take n optional path to a virtualenv - and in that rule, actually write the freeze as a little python module that imports pip and runs the freeze method and returns the data in a fact variable | 20:08 |
| mordred | that way you can control which virtualenv or global it's dumping just with ansible_python_interpreter | 20:08 |
| mordred | pabelanger: WELL - the problem with doing that generally is that makes writing playbooks with hosts: all harder | 20:08 |
| pabelanger | mordred: ya, I was actually thinking of patching upstream pip task to support freeze, then we get all that for free | 20:09 |
| mordred | pabelanger: and I think could make it harder for 'normal' users to safely write jobs | 20:09 |
| mordred | pabelanger: ++ | 20:09 |
| jeblair | pabelanger: the hostname is in zuul vars | 20:09 |
| mordred | pabelanger: I think that's a great idea - and we can carry a local module in the meantime | 20:09 |
| *** tobiash_ has quit IRC | 20:11 | |
| pabelanger | mordred: jeblair: sure, was just somthing I noticed looking at logs. it makes sense that zuul-jobs would have localhost = zuul-executor group, since they only run on zuul. But other jobs could just have localhost | 20:11 |
| jeblair | pabelanger: i don't understand that. | 20:12 |
| *** tobiash_ has joined #zuul | 20:12 | |
| *** tobiash_ has quit IRC | 20:13 | |
| mordred | OH | 20:13 |
| mordred | I do | 20:13 |
| mordred | I think we can (and should) solve that differently ... in fact, I've been considering that same problem while hacking on zuul-stream | 20:14 |
| mordred | jeblair: the issue is that the logs report local actions as having run on "localhost" - but the logical thing for some of those tasks is "this is something that ran on theexecutor" | 20:15 |
| mordred | lemme find an eample of where this has bothered me too | 20:15 |
| jeblair | that i got | 20:15 |
| jeblair | what i did not understand is: "it makes sense that zuul-jobs would have localhost = zuul-executor group, since they only run on zuul. But other jobs could just have localhost" | 20:16 |
| jeblair | pabelanger, mordred: can one of you explain that using different words? | 20:16 |
| pabelanger | that was me replying to: WELL - the problem with doing that generally is that makes writing playbooks with hosts: all harder | 20:16 |
| mordred | yes | 20:16 |
| jeblair | yep. i understand that too. | 20:16 |
| mordred | jeblair: different words coming ... | 20:16 |
| pabelanger | if I understood right, somebody with their own playbook as a job would delegate_to: localhost. Where I am suggest using delegate_to: zuul-executor in zuul-jobs | 20:17 |
| pabelanger | both would do the same thing | 20:17 |
| mordred | right - so that in targetted tasks we can mark a task as "running on zuul-executor" when that's what we want to be logged | 20:17 |
| mordred | but so that we don't just do a wholesale mapping of localhost -> zuul-executor which would confuse humans who are running their own ansible things | 20:18 |
| mordred | pabelanger: I think the biggest issue with any of the ways of doing the thing you're suggesting is that it'll inject localhost into all | 20:19 |
| jeblair | i don't think any of that changes the "hosts: all" situation. i think having "hosts: all" mean, well, all of the hosts in the nodeset is very convenient and intuitive. | 20:19 |
| jeblair | most job authors shouldn't actually have to think about the executor. | 20:19 |
| mordred | right | 20:19 |
| *** jkilpatr has quit IRC | 20:20 | |
| mordred | jeblair: what it changes is that if we added an alias into the inventory so that _we_ could write tasks that explicitly 'run on the executor' that we want labelled that way ... | 20:20 |
| mordred | it has the un-desired side-benefit of adding localhost to 'all' and having all not mean "all the hosts in teh nodeset" anymore | 20:20 |
| mordred | we benefit currently from magic in the inventory which adds localhost to the inventory but NOT to the all group | 20:21 |
| mordred | but thatonly works if localhost is not explicitly referenced anywhere in teh inventory | 20:21 |
| jeblair | also, we went through this last month: https://review.openstack.org/476081 | 20:21 |
| mordred | jeblair: yes. exactly - this is why I think the thing pabelanger is suggesting as a solution is a thing we should not do - but, I understand thething that is irking him from logs as it has also irked me | 20:22 |
| mordred | so I'm mostly now noodling to see if there is diffrent possible solution we could employ | 20:22 |
| mordred | that would not break anything or confuse people | 20:23 |
| jeblair | mordred: in mean time, want to abandon 081? | 20:23 |
| mordred | yes | 20:23 |
| mordred | done | 20:23 |
| mordred | oh - speaking of ... bcoca has been landing the new pluggable inventory sources for 2.4 already | 20:24 |
| mordred | I need to look at the API there and see what, if anyhting, we might need to be aware of | 20:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 20:25 |
| pabelanger | jeblair: mordred: right, but with 081 if we create zuul-executor group, ze01.o.o ansible_connection=localhost that would just create a single entry and localhost should still work | 20:26 |
| mordred | pabelanger: that's not the problem | 20:27 |
| mordred | pabelanger: it's that because of the way the default inventory works, adding that would _add_ localhost to the all group - which means hosts: all roles: -tox would start to tryto run tox on the executor too | 20:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 20:28 |
| mordred | and it's important that we be able to do hosts: all - becaue toerhwise playbooks will have a hard time with node-override variants | 20:28 |
| mordred | pabelanger: so we have a happy state currently where all only includes the hosts in the nodeset and not localhost | 20:28 |
| pabelanger | mordred: Oh, right. I see what you are saying now. We all + !localhost | 20:28 |
| mordred | ++ | 20:28 |
| pabelanger | Hmm | 20:29 |
| mordred | pabelanger: I think we'l be better off handling this in the logging layer | 20:29 |
| pabelanger | mordred: Ya | 20:30 |
| mordred | pabelanger: since the problem is how the logs are presented | 20:30 |
| mordred | pabelanger: I'll keep it in my brain as I keep poking at log cleanups and let you know if I find a non-crazy way | 20:30 |
| pabelanger | cool | 20:30 |
| jeblair | i'm taking 2001105 and 2001106 | 20:30 |
| mordred | kk | 20:31 |
| jeblair | (report executor errors to user and don't retry) | 20:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 20:31 |
| jeblair | i suspect their solutions are connected so i'll do both at once | 20:31 |
| *** jkilpatr has joined #zuul | 20:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 20:37 |
| pabelanger | jeblair: mordred: I think keep-jobs is still enabled | 20:40 |
| pabelanger | that is why fingur URLs still work | 20:40 |
| jeblair | pabelanger: probably so. i'm done, i'll turn it off and cleanup. | 20:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 20:50 |
| *** dkranz has quit IRC | 20:51 | |
| pabelanger | mordred: so, facts don't persist over playbook runs, to with our tox-linters job we are missing tox_envlist being setup. I think I'll have to refactor that and have it work like tox-py35 | 20:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Ansiblify pip freeze logic https://review.openstack.org/484917 | 20:54 |
| pabelanger | cool | 20:59 |
| pabelanger | http://logs.openstack.org/17/484917/10/check/openstack-doc-build/9eded10/tox/pip-freeze.txt | 20:59 |
| jeblair | lovely | 21:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ansiblify pip freeze logic https://review.openstack.org/484917 | 21:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox-pep8 jobs https://review.openstack.org/484948 | 21:04 |
| pabelanger | that should make linters pass | 21:04 |
| pabelanger | but creates a new problem | 21:04 |
| pabelanger | need to think a minute more about it | 21:04 |
| pabelanger | mordred: jeblair: it would be helpful to land the following: https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul-jobs+topic:ansible-lint | 21:09 |
| SpamapS | jeblair: I was under that impression too, but I think it is false unfortunately. | 21:16 |
| SpamapS | jeblair: Unless I've missed something, you need to pass hostname args to ssh every time. | 21:17 |
| mordred | pabelanger: looking | 21:17 |
| mordred | pabelanger: (what's the new problem?) | 21:17 |
| SpamapS | jeblair: and that actually makes sense. the hostname is used as a hash for the control socket path. | 21:17 |
| SpamapS | well, host+port+user is | 21:18 |
| pabelanger | mordred: basically https://review.openstack.org/#/c/484948/. having tox-linters do 2 envlists and facts. I mean, we can maybe add some logic to handle that, write facts to disk, but I created a new job for now. | 21:19 |
| pabelanger | mordred: maybe we just add tox-pep8 to openstack ? | 21:19 |
| mordred | pabelanger: yah - for today - I think it would be great to get tox-linters doing the thing you're wanting to make it do | 21:20 |
| mordred | pabelanger: also - I wonder if ansible-test has any helper things for doing what we do in those find commands | 21:20 |
| pabelanger | oh, maybe. I haven't looked there | 21:21 |
| pabelanger | I quickly looked at ansible-review a while back | 21:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Return executor errors to user https://review.openstack.org/484953 | 21:21 |
| jeblair | SpamapS: oh. :( well, we think that since we added '--die-with-parent' to bubblewrap, we are no longer seeing the delayed exit issue. presumably bubblewrap is cleaning up appropriately. | 21:26 |
| jeblair | SpamapS: that turns this into an optimization problem -- it would be nice to have the control socket persist across playbook runs. but we're no longer seeing the significant delays which made it seem like a requirement earlier. | 21:27 |
| jeblair | mordred, tobiash: implicit role is green now: https://review.openstack.org/482726 | 21:27 |
| SpamapS | jeblair: we can still set our own controlpath and such. But it does feel a little bit like a layer violation to start trying to match ansible's ssh args. | 21:28 |
| SpamapS | jeblair: we can, btw, use -N and not run any _commands_ on the remote host. | 21:28 |
| mordred | jeblair: woot! | 21:28 |
| SpamapS | But yeah, the mux is actually pretty stupid, and basically just hooks up clients who are using the same controlpath to the host that the master was already connected to. | 21:29 |
| mordred | I love tox-py35-on-zuul btw | 21:29 |
| mordred | it just asnwered a question I was going to ask pabelanger :) | 21:29 |
| jeblair | pabelanger has been replaced by zuul | 21:29 |
| SpamapS | jeblair: anyway, I think this one goes in the "nice to have some day" bin. How about we drop the zuulv3 tag from it? | 21:30 |
| SpamapS | I'll add my analysis to the description. | 21:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Ensure we load roles for linting https://review.openstack.org/484488 | 21:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Include ansible-playbook syntax-check for tox pep8 https://review.openstack.org/484490 | 21:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename pep8 to linters for tox https://review.openstack.org/484491 | 21:30 |
| jeblair | SpamapS: yeah, given the complexities, i don't think we need to push on it now. | 21:30 |
| jeblair | SpamapS: thanks for digging in | 21:31 |
| jeblair | mordred, pabelanger: https://review.openstack.org/484953 executor errors is also green | 21:32 |
| mordred | jeblair, SpamapS: ANOTHER thing we could consider ... | 21:35 |
| mordred | which the ansible core team mentioned to us a while back but we did not do because it didn't solve _that_ problem | 21:35 |
| mordred | is pluggable connection strategies | 21:35 |
| mordred | or pluggable connection types | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove export commands from tox based roles https://review.openstack.org/483936 | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove subunit file size check for tox role https://review.openstack.org/484519 | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove check_sudo_usage logic from tox https://review.openstack.org/484916 | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ansiblify pip freeze logic https://review.openstack.org/484917 | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox-pep8 jobs https://review.openstack.org/484948 | 21:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Compress testrepository.subunit in fetch-testr-output https://review.openstack.org/484896 | 21:36 |
| mordred | the connection controls the actual transport layer and the eecution strategy plugin controls how ansible uses the connnection plugin AIUI | 21:37 |
| mordred | SpamapS: mostly mentioning becauesit might be another place to mention in the writeup of a place we could hook in to the management of this stuff | 21:38 |
| pabelanger | jeblair: +2 | 21:38 |
| SpamapS | mordred: indeed, I think the right place to do this might be in the ssh connection plugin itself. | 21:40 |
| mordred | jeblair: feature request re: implicit roles | 21:40 |
| SpamapS | hm | 21:40 |
| SpamapS | I wonder if we already have this. | 21:40 |
| mordred | jeblair: on top of what you have, when you do the name inferrance - it is also a 'standard' thing for ansible galaxy to understand that a repo called ansible-role-foo contains a rolenamed foo | 21:41 |
| mordred | jeblair: so stripping that prefix might also be a friendly thing for us to do | 21:42 |
| SpamapS | the ControlPath by default is $HOME/.ansible/cp/$HASH_OF_THINGS | 21:42 |
| SpamapS | so if we just bind mounted that dir in............. | 21:42 |
| SpamapS | but no, we run everything in bwrap now | 21:43 |
| SpamapS | so even if a trusted pre ran first, its ssh control master would get killed because of --die-with-parent | 21:43 |
| jeblair | SpamapS: yeah, and we're successfully (yay) killing all subprocs. yeah that. | 21:43 |
| SpamapS | I'm happy just leaving this until we get into optimization cycles. | 21:43 |
| mordred | jeblair: "Galaxy strips <em>(ansible[-_.+]*)*(role[-_.+]*)*</em> from the beginning of the name." (found the docs on what it strips) | 21:43 |
| jeblair | SpamapS: yeah, i think that puts us right back at "ping all hosts" outside of the bwrap context. | 21:44 |
| mordred | https://github.com/ansible/galaxy/blob/9258039e3fd357ab70e742cd0cbaba3277dd8197/galaxy/templates/account/role_add.html#L17 for the record | 21:44 |
| SpamapS | How many bwraps do you all think we will run on the most common jobs? 3? 4? That's how many extra SSH connection setups we'll start doing. | 21:44 |
| mordred | jeblair, SpamapS: ++ | 21:44 |
| mordred | 7 or 8 at least | 21:45 |
| pabelanger | jeblair: you can override that however in the requirements.yaml file, if I remember right | 21:45 |
| pabelanger | mordred: ^sorry | 21:45 |
| mordred | base/pre, unittest/pre, tox/pre, run, tox/post, unittest/post, base/post | 21:45 |
| mordred | pabelanger: yes you can - but that's consume-side | 21:45 |
| mordred | and we suport that | 21:45 |
| mordred | but for implicit 'import git repo into galaxy and expose it to people' galaxy helpfully strips those prefixes | 21:46 |
| pabelanger | I've been thinking maybe we also support that in zuul.yaml, for a single role I cann it ansible-role-foo as my git repo, but want zuul to install it to disk as openstack.foo | 21:46 |
| mordred | pabelanger: you can already! done :) | 21:46 |
| pabelanger | Oh | 21:46 |
| SpamapS | mordred: yeah, so each of those will have a 3-4 second startup spike to setup a new SSH connection. | 21:46 |
| pabelanger | nice | 21:46 |
| pabelanger | mordred: didn't know that | 21:46 |
| mordred | SpamapS: yah - let's live until we don't :) | 21:47 |
| mordred | jeblair: oh - actually - I was just noodling - but we honestly may want to think about the implied name a little ... | 21:47 |
| SpamapS | https://storyboard.openstack.org/#!/story/2001072 <-- zuulv3 tag removed. Analysis added. | 21:47 |
| jeblair | pabelanger: see the 'roles' section in https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html | 21:48 |
| mordred | jeblair: https://galaxy.ansible.com/openstack/cloud-launcher/ is in galaxy - it gets installed as 'openstack.cloud-launcher' if you install it - and it's from the openstack/ansible-role-cloud-launcher repo | 21:48 |
| pabelanger | jeblair: Yay, thanks | 21:49 |
| jeblair | mordred: why does it get installed as 'openstack.cloud-launcher'? | 21:49 |
| mordred | jeblair: so there are two customary transforms performed by galaxy to get to a default role name | 21:49 |
| pabelanger | mordred: Wow, who did that | 21:49 |
| pabelanger | ansible-role-diskimage-builder is there | 21:49 |
| SpamapS | adam_g: https://storyboard.openstack.org/#!/story/2000879 <-- you still looking at this? | 21:49 |
| pabelanger | cool | 21:49 |
| mordred | jeblair: that's how they deal with namespacing of published roles - and because they're github-centric, they adopted the org prefix thing | 21:50 |
| mordred | jeblair: obviosly we don't want to encode github org prefixes in our stuff | 21:50 |
| jeblair | mordred: right, but that's the *galaxy* prefix. i don't think we can make any assumptions about git path prefixes. | 21:51 |
| *** hashar has quit IRC | 21:51 | |
| mordred | BUT - we probably want to consider two things a) what zuul's relationship to this is,should or should not be b) as infra/openstack how would we LIKE for galaxy to deal with our repos since they do not 'live' on github butthey assume all roles hav ea prefix | 21:52 |
| SpamapS | pabelanger: https://storyboard.openstack.org/#!/story/2000789 is for using SSL for gearman. Isn't that completed? | 21:52 |
| jeblair | mordred: do we need to solve b) now, because we've discussed this before, and the conversation involves patching galaxy to support non-github repos. | 21:53 |
| jeblair | mordred: i'd rather defer that from this time and channel if possible. | 21:53 |
| jeblair | mordred: sorry, that first was intended to be a question | 21:53 |
| pabelanger | SpamapS: it is! | 21:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ansiblify pip freeze logic https://review.openstack.org/484917 | 21:53 |
| * SpamapS is still pretty suspicious about galaxy's usefulness in most cases. :-P | 21:53 | |
| SpamapS | pabelanger: sweet, I'll mark it as such | 21:53 |
| pabelanger | SpamapS: yay | 21:54 |
| mordred | jeblair:I agree - I mention it because if we want to provide any prefix mapping at all to be similar to galaxy and ry to meet expectatoins- I don't think we want to hard-encode anyhting related to "repo prefix" in to zuul at all | 21:54 |
| SpamapS | woot, https://storyboard.openstack.org/#!/story/2000897 also just wasn't properly annotated, and so I've closed that too | 21:55 |
| * SpamapS is crushing it today | 21:55 | |
| mordred | jeblair: so while we don't nee to solve the mechanism - it might be useful to think about how we might want to express such a mapping to zuul, if at all | 21:55 |
| SpamapS | why write code when you can just mark the bugs irrelevant? | 21:55 |
| mordred | SpamapS: +100 | 21:55 |
| SpamapS | mordred: "Make zuul_stream callback plugin more robust": https://storyboard.openstack.org/#!/story/2001085 ... can we .. live without that? | 21:56 |
| SpamapS | Kinda feels like we can. | 21:57 |
| SpamapS | "If you're not embarrassed by your first release" ... | 21:57 |
| mordred | SpamapS: 4738 is we can live without | 21:57 |
| mordred | SpamapS: that's just a reminder to follow up with ansibleupstream on our discussion | 21:58 |
| mordred | SpamapS: actually - you kinow what - we shoul dhave a board/list of things to track with ansible core over time | 21:58 |
| SpamapS | Agree. zuul-ansible tag? :) | 21:58 |
| mordred | SpamapS: for instance, I'm loking at the inventory plugin interface from 2.4 - I don't think it matters to us- but it's a task that might spawn async work | 21:59 |
| mordred | ++ | 21:59 |
| jeblair | mordred: most of what we're looking at here are role collection repos where the implied name doesn't matter anyway. i'm not sure we have any use cases at the moment where the implied name does matter? | 21:59 |
| SpamapS | If we ever have to rename the project.. Zanzibar would be fun. :) | 21:59 |
| mordred | jeblair: we can certainly cross the bridge when we hit it - I think when we import ansible-role-cloud-launcher is when we'll have a specific use case to see how it feels | 22:00 |
| SpamapS | mordred: ok, so I'm going to drop zuulv3 from that story, and add zuul-ansible to it, and any other stories which are about interfacing upstream with Ansible. | 22:00 |
| mordred | SpamapS: ++ | 22:00 |
| jeblair | mordred: are we going to have any playbooks run by zuul that use cloud-launcher? | 22:01 |
| mordred | jeblair: maybe? | 22:01 |
| mordred | jeblair: but let's put a pin in it for now - I agree, it's not urgent today - sorry for the noise | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 22:04 |
| jeblair | mordred: i'd already written that much at least ^ :) | 22:04 |
| mordred | woot | 22:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Ansiblify pip freeze logic https://review.openstack.org/484917 | 22:10 |
| pabelanger | mordred: jeblair: So, if we do that by default, something like https://review.openstack.org/#/c/484526/ will be interesting. Since we have a bindep role for zuul-jobs | 22:12 |
| pabelanger | I know we talked about namspacing zuul-jobs roles | 22:13 |
| mordred | pabelanger: that is, in fact, an excellent example of a single-role repo that might want to be run by zuul and that might also want to be published to galaxy | 22:13 |
| mordred | pabelanger: although when I'm not a plane I'd like to voice-chat with you about bindep end-to-end story | 22:14 |
| pabelanger | mordred: ya, I think to work around conflict we could have name: openstack.bindep for zuul.yaml job | 22:14 |
| mordred | pabelanger: that one is multi-faceted and I think each of us has 1/2 to 2/3 of the facets - so we need to figure out the overlap and also the missing :) | 22:14 |
| mordred | pabelanger: totally. turns out we can be explicit there | 22:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Return executor errors to user https://review.openstack.org/484953 | 22:15 |
| pabelanger | mordred: agree, that role just installs bindep from multi sources, and that is it. our zuul-job bindep actually runs bindep. I think one could depend on the other | 22:15 |
| pabelanger | mordred: or we be more explict in role name: eg: ansible-role-install-bindep | 22:15 |
| pabelanger | but yes, once you are off plane | 22:16 |
| mordred | pabelanger: ++ exactly - because I think using bindep to install packages is a great story we can start telling folks | 22:17 |
| SpamapS | holy cow.. ansible has.... a lot of open issues | 22:18 |
| mordred | pabelanger: oh - what if we just had the bindep role be both, but had a "install: true" option (or something) so that it'll install bindep if-needed, but without that will just run it as our zuul-jobs rle does? | 22:18 |
| SpamapS | I just threw this on the pile: https://github.com/ansible/ansible/issues/27016 :-P | 22:18 |
| pabelanger | mordred: maybe? I'm on the fense about merging them into a single role. I like the idea of a dependency however. That way each role can be purpose specific | 22:21 |
| pabelanger | mordred: but it is something I'd like to experiment with | 22:21 |
| pabelanger | mordred: that's why I proposed the project-config change to see what it would look like as a role dependency | 22:21 |
| mordred | SpamapS: fwiw - they moved a debug line from afer thePopen to before it in devel | 22:23 |
| mordred | SpamapS: before line 816 of lib/ansible/plugins/connection/ssh.py they added: display.vvv(u'sending stop: %s' % cmd) | 22:24 |
| mordred | SpamapS: which was after the communicate in 2.3 | 22:24 |
| pabelanger | mordred: jeblair: okay, so here is something crazy. Apparently tox will pip freeze today: http://logs.openstack.org/48/484948/2/check/tox-linters/18091d5/tox/linters-1.log.txt | 22:24 |
| pabelanger | mordred: jeblair: so, I guess we don't need to write anything in ansible atm | 22:25 |
| pabelanger | unless we want to use pbr freeze | 22:25 |
| pabelanger | but not sure the difference | 22:25 |
| mordred | pbr freeze isn't superuseful in this contenxt | 22:26 |
| pabelanger | http://tox.readthedocs.io/en/latest/config.html#confval-list_dependencies_command | 22:26 |
| pabelanger | pip freeze is the default it seems in 2.4 | 22:26 |
| pabelanger | also much easier for jobs to configure that if they want PBR | 22:27 |
| mordred | pabelanger: awesome. let's stop doing it ourselves then | 22:27 |
| pabelanger | mordred: ++ | 22:27 |
| mordred | \o/ | 22:27 |
| SpamapS | mordred: yeah but the IndexError happens because cmd == [] | 22:28 |
| SpamapS | I should try it on master | 22:28 |
| SpamapS | or devel I guess | 22:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove pip freeze logic https://review.openstack.org/484917 | 22:28 |
| mordred | SpamapS: oh - I wonder if it's a 3.5 bug | 22:29 |
| mordred | SpamapS: cmd = map(to_bytes, self._build_command(self._play_context.ssh_executable, '-O', 'stop', self.host) | 22:29 |
| mordred | SpamapS: like - what if map(to_bytes is failing badly there | 22:29 |
| mordred | well - I say that- but you shouldn't be even running the command if controlpersist is False, and it should not be True if cmd is empty | 22:30 |
| mordred | (see _persistence_controls) | 22:30 |
| mordred | SpamapS: you have nerd-sniped me | 22:31 |
| jeblair | mordred: abandon https://review.openstack.org/479446 ? | 22:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add sample base job https://review.openstack.org/484485 | 22:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Require tox_envlist for tox role https://review.openstack.org/484975 | 22:38 |
| pabelanger | mordred: jeblair: ^ that stack is ready for some feedback | 22:38 |
| SpamapS | mordred: totally ;) anyway, tried on devel, still present | 22:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 22:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove ansible-role from implied role names https://review.openstack.org/484962 | 22:44 |
| jeblair | pabelanger: zuul has provided negative feedback on 484975 | 22:48 |
| jeblair | i'm going to restart zuulv3.o.o to pick up our recent fixes | 22:50 |
| jeblair | done | 22:51 |
| SpamapS | mordred: you know what it is? subprocess doesn't take an iterator. It takes a _list_ | 22:52 |
| SpamapS | oh hm no, it actually will convert it to a list above, hm | 22:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Require tox_envlist for tox role https://review.openstack.org/484975 | 22:55 |
| SpamapS | mordred: to end the nerd snipe, if I turn it into a list in the reset() method, subprocess works, but then we get a new fun problem: | 23:02 |
| SpamapS | ERROR! Cannot reset connection: | 23:02 |
| SpamapS | b'Control socket connect(/home/clint/.ansible/cp/276d6f8b70): No such file or directory\r\n' | 23:02 |
| jeblair | mordred: abandon https://review.openstack.org/479253 ? | 23:03 |
| * SpamapS lets it go like Elsa | 23:03 | |
| jeblair | SpamapS: how many times have you seen Elsa let it go? | 23:04 |
| * jeblair assumes something like maxint | 23:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Require tox_envlist for tox role https://review.openstack.org/484975 | 23:06 |
| pabelanger | hey, I see a job-output.json file :) | 23:08 |
| jeblair | neat. ff doesn't like it: SyntaxError: JSON.parse: expected property name or '}' at line 464 column 18 of the JSON data | 23:09 |
| jeblair | http://logs.openstack.org/75/484975/2/check/tox-py35-on-zuul/d1780cf/job-output.json | 23:09 |
| SpamapS | jeblair: we're certainly way past 16-bits | 23:09 |
| SpamapS | and that's just with boys.. who got tired of it already.. ;) | 23:10 |
| pabelanger | jeblair: loaded okay in chrome | 23:11 |
| jeblair | oh that's weird. now it's fine... | 23:11 |
| jeblair | shrug | 23:11 |
| jeblair | mordred, pabelanger: the log on http://logs.openstack.org/75/484975/2/check/tox-py35-on-zuul/d1780cf/job-output.txt is *much* better, but i think it can still be improved | 23:14 |
| pabelanger | ++ | 23:14 |
| mordred | jeblair: YES! | 23:15 |
| mordred | for one the blank line before the OK line isn't necessary | 23:15 |
| jeblair | mordred, pabelanger: i think we can actually get rid of some of the whitespace now... -- like exactly that, mordred. :) | 23:15 |
| mordred | jeblair: also - I think we can be slightly smarter with task banners ... | 23:16 |
| jeblair | mordred: how so? | 23:16 |
| pabelanger | +1 for smaller task banners | 23:16 |
| mordred | hrm. no -nevermind - my idea was bad | 23:16 |
| mordred | jeblair: I reallykind of want to be able to mark shell tasks as streaming or non-streaming | 23:17 |
| mordred | I mena - I don't actually | 23:17 |
| mordred | it wouldn't work - byt my display brain wants to be able to go back in time for many of these tiny commands | 23:17 |
| pabelanger | I still prefer the ansible default task banner: PLAY [Bootstrap node.] ********************************************************* for example | 23:17 |
| pabelanger | but agree whitespace before okay can be removed | 23:18 |
| mordred | pabelanger: is it the stars that you like? | 23:18 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: github: prevent getRepoPermission to raise AttributeError https://review.openstack.org/475368 | 23:18 |
| mordred | pabelanger: or the lack of the phase/playbook info? | 23:18 |
| mordred | or both? | 23:18 |
| pabelanger | mordred: lack of variable info mostly. But stars to break it up a bit. | 23:19 |
| jeblair | pabelanger: what do you mean by 'variable info'? | 23:20 |
| pabelanger | TASK [add-build-sshkey : Distribute it to all nodes user={{ ansible_ssh_user }}, state=present, key={{ lookup('file', zuul_temp_ssh_key + '.pub') }}] | 23:20 |
| pabelanger | that seems overloaded to me | 23:21 |
| mordred | yah. sorry - I'd meant to have removed those already | 23:21 |
| SpamapS | the stars do break things up visually | 23:25 |
| SpamapS | but I'm not sure if thats just habitually comfortable, or truly important | 23:25 |
| SpamapS | I'm kind of at the point where I just want ARA pages. | 23:26 |
| jeblair | SpamapS: i'm happy we'll be able to fill that desire while having nice text logs. :) | 23:26 |
| SpamapS | yeah | 23:27 |
| SpamapS | text for the greppin, ARA for the lookin | 23:27 |
| * dmsimard ARA senses are tingling | 23:27 | |
| SpamapS | Honestly, these look great. | 23:27 |
| jeblair | we'll make them better. :) | 23:28 |
| jeblair | when it comes to text layout, some of us may be... detail oriented. :) | 23:28 |
| dmsimard | You just know jeblair will make an ansibletty or an aratty thing anyway :) | 23:28 |
| jeblair | dmsimard: i wasn't even the first person to write a tui for zuul status. :) | 23:29 |
| dmsimard | There's a text interface to zuul status ?? | 23:29 |
| mordred | pabelanger, jeblair: what about the space between play and task? | 23:30 |
| jeblair | dmsimard: https://github.com/harlowja/gerrit_view/blob/master/README.rst#czuul | 23:31 |
| jeblair | (that helped push me toward urwid for gertty) | 23:31 |
| pabelanger | mordred: I think that is fine | 23:31 |
| mordred | dmsimard: writing up some thoughts on zuul text-logging, ara and tristanC's dashboard is on my todo list - I will also have some questions for you - probably not this week, but probably next week | 23:31 |
| pabelanger | mordred: something like http://paste.openstack.org/show/615789/ ? (ignore star for now) | 23:32 |
| mordred | pabelanger: k. I'm going to split the playbook part of the banner to its own line- then trim down the task lines | 23:32 |
| pabelanger | for whitespacing | 23:32 |
| dmsimard | jeblair: that zuul UI is nice! | 23:33 |
| dmsimard | Definitely going to try it out. | 23:33 |
| jeblair | mordred: i'd probably drop the space between play and task. but i think it's more important to drop the intra-task whitespace first. then i'll probably have a stronger opinion on play->task. | 23:33 |
| dmsimard | mordred: sure | 23:34 |
| pabelanger | mordred: hmm ansible-playbook --check-syntax is failing because it things zuul_stream is a typo :( | 23:38 |
| pabelanger | will have to figure out how to fix it | 23:38 |
| pabelanger | I'll play with it more in the morning | 23:40 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!