| pabelanger | mordred: corvus: for the morning, if we wanted to make a setting in ansible.cfg configurable, how would we best do that? Maybe even per playbook. Thinking of comment from tobiash https://review.openstack.org/#/c/512130/1/zuul/executor/server.py | 01:08 |
|---|---|---|
| *** JasonCL has quit IRC | 01:48 | |
| *** JasonCL has joined #zuul | 01:49 | |
| *** JasonCL has quit IRC | 02:03 | |
| *** JasonCL has joined #zuul | 02:03 | |
| *** JasonCL has quit IRC | 02:13 | |
| *** JasonCL has joined #zuul | 02:15 | |
| *** JasonCL has quit IRC | 02:22 | |
| *** yolanda has quit IRC | 02:22 | |
| *** JasonCL has joined #zuul | 02:23 | |
| *** JasonCL has quit IRC | 02:23 | |
| *** JasonCL has joined #zuul | 02:24 | |
| *** JasonCL has quit IRC | 02:24 | |
| *** rlandy|bbl is now known as rlandy | 02:58 | |
| *** harlowja has quit IRC | 03:21 | |
| *** rlandy has quit IRC | 03:31 | |
| *** openstackgerrit has quit IRC | 03:33 | |
| *** harlowja has joined #zuul | 04:28 | |
| *** harlowja has quit IRC | 05:17 | |
| *** xinliang has quit IRC | 06:09 | |
| *** xinliang has joined #zuul | 06:22 | |
| *** xinliang has quit IRC | 06:22 | |
| *** xinliang has joined #zuul | 06:22 | |
| *** threestrands_ has quit IRC | 07:02 | |
| *** openstackgerrit has joined #zuul | 07:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Default max pool resources to math.inf https://review.openstack.org/537776 | 07:26 |
| *** jpena|off is now known as jpena | 08:44 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 09:37 |
| *** electrofelix has joined #zuul | 10:01 | |
| *** jpena is now known as jpena|off | 10:33 | |
| *** jpena|off is now known as jpena | 10:33 | |
| *** Wei_Liu has joined #zuul | 10:39 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 10:41 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 11:11 |
| *** Wei_Liu has quit IRC | 11:13 | |
| *** sshnaidm_ has joined #zuul | 11:39 | |
| *** sshnaidm has quit IRC | 11:41 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: /{tenant}/projects.json routes https://review.openstack.org/537870 | 12:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 12:06 |
| *** jkilpatr has quit IRC | 12:08 | |
| *** sshnaidm_ has quit IRC | 12:38 | |
| *** jkilpatr has joined #zuul | 12:46 | |
| *** jkilpatr has quit IRC | 12:47 | |
| *** jkilpatr has joined #zuul | 12:47 | |
| *** sshnaidm_ has joined #zuul | 12:54 | |
| *** JasonCL has joined #zuul | 12:57 | |
| *** sshnaidm_ is now known as sshnaidm | 13:11 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 13:28 |
| *** yolanda has joined #zuul | 13:34 | |
| *** rlandy has joined #zuul | 13:37 | |
| *** weshay|rover is now known as weshay|ruck | 13:58 | |
| *** dmellado has quit IRC | 14:32 | |
| *** jpena|lunch is now known as jpena | 14:35 | |
| *** dmellado has joined #zuul | 14:39 | |
| *** dmellado has quit IRC | 14:39 | |
| *** sshnaidm_ has joined #zuul | 14:41 | |
| *** dmellado has joined #zuul | 14:41 | |
| *** dmellado has quit IRC | 14:42 | |
| *** dmellado has joined #zuul | 14:43 | |
| *** sshnaidm has quit IRC | 14:44 | |
| *** dmellado has quit IRC | 14:44 | |
| *** dmellado has joined #zuul | 14:53 | |
| *** dmellado has quit IRC | 14:53 | |
| *** dmellado has joined #zuul | 15:01 | |
| *** dmellado has quit IRC | 15:01 | |
| *** dmellado has joined #zuul | 15:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 15:03 |
| *** dmellado has quit IRC | 15:03 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Handle missing request during a decline. https://review.openstack.org/537932 | 15:09 |
| *** sshnaidm_ is now known as sshnaidm | 15:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 15:17 |
| *** dmellado has joined #zuul | 15:19 | |
| *** dmellado has quit IRC | 15:25 | |
| *** dmellado has joined #zuul | 15:26 | |
| *** dmellado has quit IRC | 15:27 | |
| *** dmellado has joined #zuul | 15:29 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Stop the PoolWorker thread when max-servers is 0 https://review.openstack.org/537942 | 15:44 |
| dmsimard | Shrews: finding a couple of cool things from that nl02 issue eh | 15:45 |
| Shrews | a couple | 15:46 |
| *** yolanda has quit IRC | 15:47 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Allow for max-servers less than 0 https://review.openstack.org/537946 | 15:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Allow for max-servers less than 0 https://review.openstack.org/537946 | 15:55 |
| corvus | pabelanger: i don't think we need to make that option configurable; at least not now. let's wait for someone to say why they need some other value that doesn't work for everyone. | 15:56 |
| pabelanger | corvus: okay, I'll bring that patch online today. But should help with UNREACHABLE errors we sometime gets from nodes | 15:57 |
| gundalow | Could we disable https://github.com/ansible/ansible/pull/35315#issuecomment-360392787 | 16:08 |
| dmsimard | mordred: ^ | 16:09 |
| gundalow | mordred: What's triggering that notification? | 16:10 |
| dmsimard | gundalow: if I had to guess, there's probably a "Depends-On" somewhere | 16:13 |
| mordred | gundalow: oh! that should not be happening at all | 16:13 |
| mordred | gundalow: terribly sorry, let me investigate | 16:13 |
| dmsimard | I wonder if it would be possible for that "merge failure" message to print what's the source (i.e, the parent change) | 16:13 |
| dmsimard | corvus: ^ | 16:14 |
| gundalow | mattclay: FYI, discussing Zuul notifications of merge conflicts | 16:15 |
| mordred | gundalow, mattclay: well - there's 2 different issues here | 16:15 |
| gundalow | mordred: haha, like all good bugs :) | 16:15 |
| mordred | one is that zuul should not be reporting ANYTHING on those - and I now see the issue, it's a config problem on our side, fix coming | 16:16 |
| mordred | the second is, when such an error does happen and it's appropriate to report, how can we improve the message | 16:16 |
| dmsimard | mordred: +1 | 16:17 |
| gundalow | cool | 16:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enabled ssh retries for ansible https://review.openstack.org/537953 | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:17 |
| mordred | gundalow, dmsimard: remote: https://review.openstack.org/537955 Stop spamming ansible/ansible | 16:18 |
| dmsimard | +2 | 16:19 |
| mordred | pabelanger, corvus: ^^ we're reporting on changes to ansible/ansible that we are not supposed to be reporting on (and also performing unneeded merge-check on every ansible PR) | 16:19 |
| pabelanger | mordred: +3 | 16:20 |
| pabelanger | mordred: corvus: dmsimard: 537953 might be of interest, enables ssh retires in ansible to help with networking issue we see on jobs | 16:21 |
| mordred | pabelanger: oh - I need to update the validation for project-config that's going to bomb out on system-required not being there | 16:21 |
| pabelanger | mordred: ack | 16:22 |
| dmsimard | mordred: oh right the job will fail otherwise, good point | 16:22 |
| mordred | dmsimard, pabelanger: updated | 16:22 |
| pabelanger | done | 16:23 |
| corvus | mordred: have links to the output? | 16:25 |
| corvus | oh sorry, i see now | 16:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:26 |
| *** bhavik1 has joined #zuul | 16:27 | |
| corvus | mordred, gundalow: the merge failure itself was caused by an interrupted network connection to github -- considering what we know about github availability, we may need to make fetching refs a bit more robust. | 16:36 |
| dmsimard | good catch | 16:37 |
| corvus | (obviously reporting on that was in error, but i didn't want to waste the opportunity to learn from it) | 16:37 |
| corvus | dmsimard: and yes, we should be able to improve the merge failure reports | 16:38 |
| mordred | corvus: kind of reminds me of one of the thoughts (that didn't wind up being the issue) when we were looking at the deleted ref thing the other day ... | 16:39 |
| mordred | corvus: which is that if we're fetching a ref that we've been told to fetch from another part of the system, we should be able to be more aggressive with retries, since we should be able to know that the thing we're trying to fetch does exist | 16:40 |
| corvus | mordred: indeed -- we know this should work, so this is is something we can do a tight retry on -- try N times with 60s sleep on failure sort of thing. | 16:40 |
| mordred | corvus: otoh - the issue you mentioned around the delete issue still holds - how do we find actual permanant issues in that case | 16:40 |
| mordred | corvus: yah | 16:41 |
| *** bhavik1 has quit IRC | 16:41 | |
| gundalow | corvus: ah, nice | 16:44 |
| gundalow | I wonder if Ansibullbot has work arounds for that | 16:44 |
| gundalow | hum, though ansibullbot pulls status, rather than using webhooks, so not sure | 16:45 |
| corvus | and this was on an actual git fetch, not an api call, so probably hitting different github systems | 16:45 |
| *** jkilpatr has quit IRC | 16:54 | |
| *** jkilpatr has joined #zuul | 16:56 | |
| *** electrofelix has quit IRC | 17:06 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Only upload logs when jobs fail https://review.openstack.org/537929 | 17:07 |
| *** yolanda has joined #zuul | 17:15 | |
| tobiash | mordred: do you have a plan for ^^? | 17:31 |
| mordred | tobiash: a plan for making that patch go away? yes, both a short term and a medium term | 17:32 |
| tobiash | Yes ;) | 17:33 |
| mordred | tobiash: short term is that we can revert it as soon as our current crisis is averted - medium term I'd like to take another stab at moving to swift, so I think that'll be a new upload-logs-to-swift role (or something like that) - and I'm working on putting together a proposal for what that might look like | 17:34 |
| mordred | also shorter-medium term we should perhaps think of a way to introduce a safety valve in the role so that an admin can switch to "only upload logs on failures" in a similar crisis without needing to patch upload-logs in zuul-jobs | 17:35 |
| mordred | (us editing a role in zuul-jobs because of an openstack production incident is obviously not the way we want to treat content in that repo) | 17:36 |
| tobiash | mordred: just wanted to suggest that ;) | 17:37 |
| mordred | tobiash: ++ | 17:38 |
| mordred | yah - there's several lessons to learn from today | 17:38 |
| * tobiash reads the backlog | 17:39 | |
| tobiash | Also what I observed is that failed cat jobs are not retried which can prevent startup of the scheduler | 17:41 |
| mordred | tobiash: most of the relevant discussion is actually in #openstack-infra-incident | 17:42 |
| tobiash | Yah, currently reading that | 17:43 |
| corvus | oh, i did not realize that we merged an openstack change to zuul-jobs :( | 17:44 |
| corvus | is someone working on reverting that? | 17:45 |
| corvus | like, i think we need to revert it before the end of the crisis, because we've just caused crises for other folks | 17:45 |
| corvus | what we could probably do is add in a site-variable toggle for that | 17:45 |
| corvus | mordred, pabelanger, dmsimard: ^ do any of you want to work on that change right now? | 17:46 |
| mordred | corvus: yah - I'll work on that now | 17:46 |
| dmsimard | oh damn, we should have probably toggled this through a project-config playbook | 17:47 |
| dmsimard | mordred: are you going to hide that logic under a boolean or something ? | 17:47 |
| mordred | dmsimard: yah | 17:47 |
| *** weshay|ruck is now known as weshay|ruck|brb | 18:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:10 |
| mordred | dmsimard, corvus, tobiash: ^^ check me on my boolean logic - I went through a few more complex iterations on it with double-negatives and whatnot but I *think* that's correct | 18:11 |
| *** jkilpatr has quit IRC | 18:14 | |
| tobiash | mordred: I think you need a | default(true) on the site variable | 18:14 |
| mordred | tobiash: I stuck it in to https://review.openstack.org/#/c/537986/1/roles/upload-logs/vars/main.yaml ... but I could also just do a default on it there if you think that'll be more readable | 18:15 |
| tobiash | mordred: sorry, overlooked the default | 18:15 |
| tobiash | Wait, you put it into vars? | 18:16 |
| tobiash | mordred: vars in the vars folder are constants and cannot be overwritten | 18:16 |
| tobiash | You need to put that into the defaults folder | 18:17 |
| mordred | tobiash: except by vars on the command line, right? or - well, let's just do default() | 18:17 |
| tobiash | Don't know the exact order but I thought the vars folder has the highest precedence | 18:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:18 |
| * tobiash rereads docs about that | 18:18 | |
| mordred | tobiash: maybe just doing default() like you originally suggested is clearer ^^? | 18:18 |
| tobiash | mordred: you're right that extravars win, but I wouldn't count on that | 18:22 |
| dmsimard | mordred: I'd use defaults/main.yml instead of vars/ | 18:22 |
| dmsimard | In summary precedence is defaults -> vars -> extra_vars | 18:22 |
| tobiash | And many more in between ;) | 18:23 |
| dmsimard | tobiash: summary :) | 18:23 |
| dmsimard | extra_vars even prevents set_fact tasks from applying properly, like the set_fact task will run, won't fail but won't actually set the variable if there's an extra_var for that var | 18:23 |
| dmsimard | I was bitten by that more than once before realizing what was going on | 18:24 |
| tobiash | mordred: do you really set a default to zuul-success? | 18:24 |
| tobiash | Really want | 18:25 |
| * tobiash apologizes for bad typing on mobile phone | 18:25 | |
| *** jkilpatr has joined #zuul | 18:26 | |
| mordred | tobiash: I only did that as a cargo-cult from roles/emit-ara-html/tasks/main.yaml ... | 18:27 |
| tobiash | mordred: maybe it's better to crash ansible if zuul success is not defined instead of silently defaulting to false | 18:27 |
| *** weshay|ruck|brb is now known as weshay | 18:27 | |
| mordred | tobiash: wfm | 18:28 |
| corvus | it will only be defined for post playbooks. so omitting the default is a way of enforcing that's only used in post. | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:28 |
| tobiash | Yes that was my thought. Do we want that? | 18:29 |
| clarkb | tobiash: its ok I am super bad at typing on mobile phone | 18:29 |
| tobiash | corvus, mordred I'm fine with both, just wanted that we think about these two options | 18:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:38 |
| mordred | tobiash, corvus, dmsimard: updated ^^ to match the logic that's currently in emit-ara-html | 18:38 |
| mordred | pabelanger: ^^ you too | 18:39 |
| corvus | mordred: does the failure string equality work if the var is not defined? | 18:40 |
| corvus | mordred: oh, it short circuits in that case, yeah? | 18:40 |
| tobiash | It should | 18:41 |
| mordred | yah. should short-circuit | 18:41 |
| mordred | also, I copy-pastad that logic from emit-ara-html, so I'm fairly confident in it | 18:42 |
| tobiash | mordred: and next ps with docs? ;) | 18:42 |
| mordred | tobiash: so demanding ... :) | 18:42 |
| tobiash | mordred: I also don't like writing docs but that's important ;) | 18:44 |
| mordred | tobiash: agree | 18:44 |
| *** jpena is now known as jpena|off | 18:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 18:46 |
| mordred | tobiash: this time with docs | 18:46 |
| *** sshnaidm has quit IRC | 18:46 | |
| tobiash | mordred: beautiful | 18:48 |
| *** myoung is now known as myoung|biab | 18:58 | |
| dmsimard | mordred: comment on https://review.openstack.org/#/c/537986 | 19:10 |
| *** myoung|biab is now known as myoung | 19:20 | |
| Shrews | mordred: is https://review.openstack.org/536863 just for a testing? or did you mean for that to merge? unfortunately, the logs are gone so i can't see the output without rechecking it | 19:21 |
| mordred | Shrews: it's a real patch - but I think we're gonna wait until post-release to mess with the reno/stestr stack | 19:22 |
| Shrews | mordred: oh. we should WIP or -2 those then | 19:23 |
| mordred | yah - just did - zuul got a -2 already - missed nodepool | 19:23 |
| mordred | sorry for confusion | 19:23 |
| Shrews | silly mordred. i'm always confused | 19:24 |
| mordred | heh | 19:24 |
| *** myoung is now known as myoung|cheesebur | 19:34 | |
| *** myoung|cheesebur is now known as myoung|food | 19:34 | |
| *** harlowja has joined #zuul | 19:44 | |
| *** myoung|food is now known as myoung | 20:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zone-zuul-ci.org master: DNM Testing that log disabling works https://review.openstack.org/538020 | 20:21 |
| mordred | gundalow: the fix to stop spamming you landed, you should see no more such comments. sorry again | 20:22 |
| *** jkilpatr has quit IRC | 20:22 | |
| pabelanger | config-validate for nodepool seems to started raising an exception: http://logs.openstack.org/91/537991/1/check/project-config-nodepool/07312ef/job-output.txt.gz#_2018-01-25_19_46_27_210846 | 20:30 |
| gundalow | mordred: Thanks for the quick fix. It's all good. Also as long as the restriction to OpenStack is working then that's fine. | 20:33 |
| gundalow | Guess as some point we should sync up, though will let you decide when it makes sense to do that | 20:33 |
| pabelanger | okay, looks like python2 / python3 issue | 20:36 |
| mordred | gundalow: yes indeed! we're getting quite close to it being a good time for that | 20:37 |
| *** rlandy is now known as rlandy|brb | 20:47 | |
| *** myoung is now known as myoung|pto | 20:53 | |
| *** sshnaidm has joined #zuul | 21:00 | |
| *** rlandy|brb is now known as rlandy | 21:12 | |
| *** jkilpatr has joined #zuul | 21:20 | |
| *** sshnaidm is now known as sshnaidm|off | 21:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 21:33 |
| corvus | tristanC: do you know what would be involved in updating the url when you enter filters into the dashboard? | 22:07 |
| corvus | i want folks to be able to share deep-links to the search results they're seeing | 22:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:04 |
| *** rlandy is now known as rlandy|biab | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add .zuul.yaml https://review.openstack.org/538068 | 23:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add .zuul.yaml https://review.openstack.org/538068 | 23:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:40 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Add .zuul.yaml https://review.openstack.org/538068 | 23:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!