| mnaser | Shrews things should be okay now | 00:47 |
|---|---|---|
| mnaser | fun ops graphs - https://i.imgur.com/0LSHHb1.png - router psu deteriorating till failure (but still managed to get enough voltage to report SNMP) | 00:47 |
| *** jkilpatr_ has quit IRC | 01:37 | |
| *** zhuli has quit IRC | 02:22 | |
| *** zhuli has joined #zuul | 02:22 | |
| dmsimard | what, obscure scenarios ? We think about those first ! | 02:27 |
| dmsimard | jeblair: for all the negative press zuul v3 gets, we should pin those positive "testimonials" somewhere :p | 02:27 |
| *** jhesketh has quit IRC | 02:35 | |
| mrhillsman | agreed | 02:36 |
| *** cinerama has quit IRC | 02:36 | |
| mrhillsman | coming in not know anything it is not too bad of a task to get somewhat up to speed and get things working | 02:37 |
| *** jhesketh has joined #zuul | 02:38 | |
| *** cinerama has joined #zuul | 02:41 | |
| *** bhavik1 has joined #zuul | 03:54 | |
| *** bhavik1 has quit IRC | 04:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 05:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 05:08 |
| tobiash | tristanC: I just hit the rebase button so your stack is linear and I can put something on top of it | 05:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route https://review.openstack.org/503268 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 05:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Adapt relative streaming urls to tenant scope https://review.openstack.org/512483 | 05:18 |
| tobiash | tristanC: ^ had to rebase your whole stack due to merge conflict (and added a change on top), hope that's ok for you | 05:19 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Adapt relative streaming urls to tenant scope https://review.openstack.org/512483 | 05:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add support for result data in child jobs https://review.openstack.org/504808 | 06:21 |
| tobiash | jeblair: fixed your result data change ^ | 06:22 |
| *** hashar has joined #zuul | 07:53 | |
| *** electrofelix has joined #zuul | 08:51 | |
| *** sambetts|afk is now known as sambetts | 09:20 | |
| *** smyers has quit IRC | 09:54 | |
| *** smyers has joined #zuul | 09:56 | |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** hashar has quit IRC | 10:33 | |
| *** openstackgerrit has joined #zuul | 10:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle tox environments via tox config https://review.openstack.org/512448 | 10:57 |
| *** jkilpatr has joined #zuul | 11:06 | |
| *** zhuli has quit IRC | 11:11 | |
| *** zhuli has joined #zuul | 11:12 | |
| odyssey4me | so, giving some thought to using zuul for ansible role/playbook testing a question has come up | 11:12 |
| *** zhuli has quit IRC | 11:12 | |
| odyssey4me | if a role/playbook is written with one version of ansible in mind, and zuul has some unknown version of ansible - how do we make them all get along? | 11:12 |
| *** zhuli has joined #zuul | 11:12 | |
| odyssey4me | ansible versions are quite different, and at this stage I can see us having a rough time trying to use zuul's ansible to run any of our roles consistently forever, as we have per branch ansible version support | 11:14 |
| *** jkilpatr has quit IRC | 11:27 | |
| *** jkilpatr has joined #zuul | 11:27 | |
| Shrews | mnaser: yes, seems to be doing much better since last night. thanks. | 12:01 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 12:31 |
| dmsimard | odyssey4me: It's something I have thought about as well, the obvious workaround is to install and run ansible within your job but there would be great benefits to running "natively". The thing is Zuul (executor) itself has a significant amount of Ansible code (plugins, modules, callbacks) which makes it hard to support different versions simultaneously. | 12:43 |
| dmsimard | I don't think we expose the ansible version in zuul other than in the ara report right now, maybe putting a bit more visibility on it could be a first step | 12:47 |
| *** dkranz has joined #zuul | 12:49 | |
| *** pabelanger has quit IRC | 13:01 | |
| *** pabelanger has joined #zuul | 13:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Migrate legacy jobs https://review.openstack.org/512611 | 13:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Migrate legacy jobs https://review.openstack.org/512611 | 13:15 |
| Shrews | jeblair: mordred: I'm a bit confused on what to do with the nodepool jobs (e.g. ^^^) since we intend to merge the features branch back into master at some point. Should I duplicate some of the work from 512611 into both branches? Or maybe we should go ahead with the merge and get rid of the feature branch? | 13:33 |
| mordred | Shrews: that's a great question | 13:36 |
| pabelanger | +1 for merge to master. will make rolling out new builders a little easier too | 13:37 |
| mordred | Shrews: btw - if you feel like playing - there's a new shiny devstack base job - https://review.openstack.org/#/c/500365/29/.zuul.yaml examples of use there <-- | 13:37 |
| Shrews | mordred: ++ | 13:38 |
| mordred | Shrews: however - you can define all of the jobs in master and just have a project: stanze .zuul.yaml in the feature branch that uses them | 13:38 |
| mordred | Shrews: so maybe it's a good opportunity to play with that even though it's not STRICTLY necessary - you know, since we've got a good opportunity to flesh out bugs and whatnot | 13:39 |
| Shrews | mordred: wouldn't that make the merge back to master more difficult? | 13:39 |
| mordred | Shrews: btw -im rechecking that shade change | 13:39 |
| mordred | Shrews: nah - we can always null-merge the .zuul.yaml file | 13:40 |
| Shrews | mordred: i sort of feel like it's time we did the merge anyway. if we do that, i could just change master. but i'll defer to the actual person(s) doing the merge | 13:41 |
| mordred | Shrews: I'm game to merge nodepool feature back to master myself - but I think we need to make sure folks using puppet-openstackci aren't going to all of a suden get upgraded | 13:42 |
| tobiash | nodepoolv3 is incompatible to zuulv2 so shouldn't we merge zuul and nodepool to master at the same time? | 13:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle tox environments via tox config https://review.openstack.org/512448 | 13:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Make executor hostname configurable https://review.openstack.org/512629 | 13:57 |
| mordred | tobiash: yah - that's my main reason to be hesitant | 13:58 |
| Shrews | mordred: i suppose i could move that job migration change to master, and just remove the .zuul.yaml from features/zuulv3 and that should work, yeah? | 14:06 |
| Shrews | going to try that and find out | 14:10 |
| mordred | Shrews: sort of ... you still need a .zuul.yaml in feature/zuulv3 that lists the jobs to run - jobs in the master branch get an implied branch matcher ... | 14:15 |
| mordred | Shrews: I think jeblair wrote an updated piece of doc about this yesterday ... | 14:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:15 |
| mordred | Shrews: https://docs.openstack.org/infra/manual/zuulv3.html#stable-branches | 14:16 |
| mordred | Shrews: it's written about stable branches - but it holds for all branches | 14:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update bindep role defaults to be more sane https://review.openstack.org/512639 | 14:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 14:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 14:29 |
| Shrews | sorry for the noise. trying some stuff :/ | 14:29 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add support for project-templates https://review.openstack.org/512449 | 14:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 14:36 |
| mnaser | awesome work jeblair ... changes to zuul yaml are *so* fast :) | 14:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 14:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: New client command for printing autohold requests https://review.openstack.org/512360 | 14:54 |
| mnaser | moving jobs to non voting without infra, yay for successful use cases | 14:55 |
| pabelanger | +1 | 14:55 |
| pabelanger | I think it is awesome that a non infra person wrote a publisher to docker, all in repo, and got it working | 14:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:56 |
| pabelanger | without having to land code in project-config | 14:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 14:59 |
| jlk | pabelanger: can you explain why https://review.openstack.org/#/c/512458/ was made? | 15:18 |
| pabelanger | jlk: to workaround that we don't have access to ansible facts on executors | 15:19 |
| pabelanger | so ansible_user_dir is undefined | 15:19 |
| jlk | ah, that wasn't clear from the commit message | 15:20 |
| pabelanger | jlk: yah, it merged before I could update it per your comment | 15:20 |
| jlk | gotcha | 15:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 15:27 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 15:38 |
| Shrews | cool. pabelanger, we should probably restart launchers when convenient to get that ^^^. i don't think there's a huge hurry though | 15:41 |
| pabelanger | wfm | 15:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Several executor threading fixes https://review.openstack.org/512673 | 15:45 |
| *** sambetts is now known as sambetts|afk | 16:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Periodic refname, newrev and oldrev legacy vars https://review.openstack.org/512701 | 16:30 |
| *** bhavik1 has joined #zuul | 16:31 | |
| *** bhavik1 has quit IRC | 16:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Several executor threading fixes https://review.openstack.org/512673 | 16:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't return POST_FAILURE when aborting post playbooks https://review.openstack.org/512712 | 16:49 |
| dmsimard | jeblair, mordred: odyssey4me had an interesting question earlier this morning that I also wonder about: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2017-10-17.log.html#t2017-10-17T11:12:12 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't return POST_FAILURE when aborting post playbooks https://review.openstack.org/512712 | 17:01 |
| jeblair | clarkb, tobiash: ^ | 17:01 |
| clarkb | ty | 17:01 |
| tobiash | lgtm | 17:03 |
| jeblair | dmsimard, odyssey4me: yes, mordred and i were talking about this recently too. i think we would like to support that better. at the moment, that would probably mean that we would take on more work to support more versions of ansible simultaneously. it may be feasible, but not something to promise lightly. probably something to discuss in the coming weeks. | 17:04 |
| jeblair | Shrews: ^ | 17:04 |
| odyssey4me | ja | 17:04 |
| dmsimard | yeah, I think we want to support that, the question is more how to do it | 17:05 |
| dmsimard | i.e, install ansible (zuul-executor ?) *inside* the bubblewrap at a specified version ? something else ? | 17:06 |
| dmsimard | ¯\_(ツ)_/¯ | 17:06 |
| jeblair | yeah, there are questions about installation, our plugins, and executor behavior | 17:06 |
| dmsimard | Maintaining the Ansible code we have in Zuul to work across different versions of Ansible is definitely challenging to say the least. Just for ARA, which pulls a much smaller surface, is not trivial | 17:07 |
| dmsimard | I end up having to maintain differences between 2.2, 2.3 and 2.4 right now and I already can't wait until 2.4 is EOL already to do things differently with what's coming in 2.5 | 17:08 |
| jeblair | dmsimard: yes, though hopefully minor release upgrades would be less work... | 17:08 |
| jeblair | oh here's a question | 17:08 |
| dmsimard | ansible doesn't follow semver | 17:08 |
| dmsimard | 2.3 to 2.4 doesn't guarantee you anything | 17:08 |
| dmsimard | 2.3.0 to 2.3.1, is probably fine | 17:08 |
| jeblair | odyssey4me: do you need, say, multiple 2.3 releases (ie, 2.3.1, 2.3.2), or would "latest 2.3" and "latest 2.4" be sufficient? | 17:09 |
| dmsimard | Typically (as an ansible end user), it's based on major releases like 2.3 and 2.4. Dot releases are usually backwards compat bug fixes. | 17:10 |
| jeblair | dmsimard: for a fun 2.3 -> 2.4 executor behavior change, see this, which i wrote yesterday: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n1406 | 17:10 |
| dmsimard | jeblair: oh, it's much more ugly for ara: https://github.com/openstack/ara/blob/master/ara/config.py#L26-L100 | 17:11 |
| mordred | yah - this is an area I totally want to support more better - but also - WOW | 17:11 |
| mordred | that said - hopefully starting with 2.4 -> 2.5 we can provide some feedback on whether or not changes might break things ... | 17:12 |
| pabelanger | yah, once we start reporting against ansible/ansible, our unit testing for zuul should start to expose issues | 17:13 |
| pabelanger | then we just import unit testing with more coverage | 17:13 |
| odyssey4me | jeblair we're typically testing against the latest of a minor, so 2.4.x/2.3.x - we need to use it outside of infra, so we'd need a constraints file to reference on a sha as the 'reference' version that was tested | 17:14 |
| *** weshay|ruck is now known as weshay|afk_see_r | 17:14 | |
| odyssey4me | yeah, minor revs for ansible are bad when it comes to underlying changes, and often breaking behaviour | 17:15 |
| odyssey4me | as they add more tests for things they never knew existed it's getting better though | 17:15 |
| dmsimard | oh yeah, sometimes it's not "regressions", it's "fixing unintended usage" or stuff like that :) | 17:23 |
| jlk | those are features, they're bugs. | 17:27 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Be explicit about legacy var additions/omissions https://review.openstack.org/512769 | 18:02 |
| kklimonda | how will zuul-executor behave if gearman connection is lost while it's executing a job? Will it keep trying to report it back, or just drop it? | 18:17 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Be explicit about legacy var additions/omissions https://review.openstack.org/512769 | 18:29 |
| fungi | kklimonda: i believe we (somewhat recently) fixed it so that the scheduler will reschedule builds from an executor which disconnects, but i don't know what the executor will do after that (pretty sure it won't be able to get the obsoleted builds reported back to gerrit in the end at least) | 18:31 |
| fungi | kklimonda: that was mostly in the interest of making hard/unanticipated shutdown of executors less disruptive to end users | 18:32 |
| pabelanger | I think executors just pass results down into python-gear and that is about it | 18:33 |
| kklimonda | fungi: is there a timeout before scheduler marks executor as down, or does it do it as soon as it gets to it? | 18:34 |
| * fungi finds the change | 18:36 | |
| fungi | https://review.openstack.org/510223 Always retry jobs on executor disconnect | 18:37 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Several executor threading fixes https://review.openstack.org/512673 | 18:37 |
| fungi | kklimonda: so it's if gear reports DISCONNECT as the result, i think | 18:38 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't return POST_FAILURE when aborting post playbooks https://review.openstack.org/512712 | 18:39 |
| *** electrofelix has quit IRC | 18:39 | |
| kklimonda | mhm, thanks - we've been having some network issues recently, and it seems zuulv3 is not going to magically fix those ;) | 18:39 |
| fungi | kklimonda: meaning it's less up to the zuul scheduler, more up to the gearman service it's relying on | 18:39 |
| fungi | kklimonda: if you're having issues with connectivity between your control plane services (e.g. executor to gearman or in v2 launcher/jenkins master to gearman) then i don't expect v3 will necessarily make things any more robust there | 18:41 |
| pabelanger | fungi: right, but I think when we move to zookeeper, we'll have a little more robustness? If I understand how locks are working with nodepool nodes | 18:42 |
| fungi | yeah, once we can replace gearman in that part of the architecture, perhaps | 18:43 |
| fungi | but that's not v3 as it exists today/for release | 18:43 |
| pabelanger | yah | 18:44 |
| fungi | more of a 3.x or 4.0 | 18:44 |
| pabelanger | zuulv4, hype! | 18:44 |
| tristanC | tobiash: no worries, thanks for the rebase. I'll be afk for a couple more week | 19:30 |
| tobiash | tristanC: my deployment now includes your stack and it looks great :) | 19:32 |
| tristanC | tobiash: nice! does the zuul-web randomly return 500s or is it consistent? | 19:34 |
| dmsimard | a wild tristanC has appeared ! Hi :D | 19:34 |
| tristanC | i mean, on the status page or the builds list, the controllers sometime fail for no reason | 19:34 |
| tristanC | dmsimard: hello :-) | 19:35 |
| tobiash | tristanC: didn't notice 500s so far | 19:35 |
| tristanC | great, then I suspect an incorrect aiohttp version or something on my side | 19:36 |
| tobiash | tristanC: have it working since today, but I'll observe this | 19:37 |
| *** weshay|afk_see_r is now known as weshay|ruck | 19:42 | |
| tristanC | tobiash: thanks, the code in await blocks may need a try/except since there was no traceback/error message in logs | 19:46 |
| tristanC | i'll have a fresh look in 2 weeks | 19:47 |
| tobiash | k, will check this if I notice 500s | 19:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 19:51 |
| *** dkranz has quit IRC | 19:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 19:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:01 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:36 |
| dmsimard | mordred: just noticed something in zuul_stream for action plugins: http://logs.openstack.org/54/509254/4/infra-check/zuul-stream-functional/8ae19d1/stream-files/stream-job-output.txt | 20:38 |
| dmsimard | search for "2017-10-04 13:04:44.434081 | TASK [Output a debug sentence]" | 20:38 |
| dmsimard | Notice how the action plugin (debug) runs on both nodes and they are not prefixed by the node ? | 20:39 |
| dmsimard | I'll try and hunt that down, found it while looking for another bug :/ | 20:40 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Migrate legacy jobs https://review.openstack.org/512637 | 20:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update jobs for features/zuulv3 branch https://review.openstack.org/512642 | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: upload-pypi: make repository url optional https://review.openstack.org/512862 | 21:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Add a projects_by_name view of zuul.projects https://review.openstack.org/512868 | 21:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't set the build result to ABORTED https://review.openstack.org/512883 | 22:14 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: upload-pypi: make repository url optional https://review.openstack.org/512862 | 22:27 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update bindep role defaults to be more sane https://review.openstack.org/512639 | 23:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't set the build result to ABORTED https://review.openstack.org/512883 | 23:19 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
