| *** isaacb has joined #zuul | 06:44 | |
| *** isaacb has quit IRC | 07:48 | |
| mordred | dmsimard|off: well - we already save the inventory file ourselves and consider that safe from a zuul pov - I'm assuming that means the original files themselves and not necessarily any additional vars passed in as -e@foo.yaml yeah? | 13:19 |
|---|---|---|
| dmsimard|off | mordred: yeah, the actual files | 13:20 |
| mordred | I do not think it would break anything for us at least | 13:21 |
| dmsimard|off | Actually it's kind of scary, callbacks even have access to decrypted vault content and the vault password :/ | 13:21 |
| mordred | dmsimard|off: yah - callbacks have to be trusted | 13:22 |
| mordred | dmsimard|off: btw - http://logs.openstack.org/24/495424/2/check/tox-linters/ff8cbf9/ara/ | 13:22 |
| dmsimard|off | Sweet! | 13:22 |
| dmsimard|off | So, anyway, what I meant to say is that I might default collection to true and we'll need to toggle it off | 13:23 |
| mordred | I don't think we need to toggle it off - we dont put any sensitive data into inventory or hostvar/groupvar files | 13:24 |
| dmsimard|off | You mentioned that extra vars file with secrets though, I haven't tested one of those but I suspect it'd be picked up too | 13:25 |
| mordred | yah - if you collect extra vars files passed via -e@ then that would be epicly bad | 13:25 |
| dmsimard|off | right, so basically there is a list of all the files (and their parsed contents, strings, lists, dicts) cached in the play/playbook context | 13:27 |
| dmsimard|off | tl;dr, ara will pick up all the files provided by cache.keys(). toggling that off entirely probably doesn't make a lot of sense so I guess the blacklist would need to be pattern/regex based | 13:29 |
| dmsimard|off | http://paste.openstack.org/raw/618853/ | 13:30 |
| dmsimard|off | that's the playbook scope, in the play scope there's even a (fully decrypted) vault file | 13:31 |
| pabelanger | hey, ara data | 13:32 |
| dmsimard|off | pabelanger: good morning :) | 13:33 |
| pabelanger | dmsimard|off: feature request! Sort plays by timestamp, they are reverse sorted now :) | 13:34 |
| dmsimard|off | pabelanger: yeah, I've been asked that before -- can't please everyone so I'll have to make it configurable | 13:34 |
| dmsimard|off | pabelanger: otherwise the people who prefer the current sorting will whine :) | 13:34 |
| pabelanger | we also likely need to add full path to playbook location, make it easier to know which job pre.yaml comes from | 13:35 |
| pabelanger | dmsimard|off: :) | 13:35 |
| dmsimard|off | pabelanger: the full path to playbook is available if you hover the playbook file name and if you click on it | 13:35 |
| dmsimard|off | (also if you expand the files tab) | 13:36 |
| pabelanger | Ya, but I have to click links :) | 13:36 |
| pabelanger | but ya, looks good | 13:38 |
| dmsimard|off | pabelanger: playbook paths tend to be long at best though, we used to display it but it just looked clunky (too long) or wasn't useful (too truncated) | 13:38 |
| dmsimard|off | pabelanger: open to ideas though | 13:38 |
| dmsimard|off | pabelanger: what are you interested in the full path for ? | 13:38 |
| pabelanger | http://logs.openstack.org/24/495424/2/check/tox-linters/ff8cbf9/ara/ for example, look at the 3 pre.yaml files | 13:39 |
| dmsimard|off | oh, because there can be different pre.yaml files | 13:39 |
| pabelanger | at first glance, that is a little confusing | 13:39 |
| pabelanger | however they are actually, from bottom up: git.openstack.org/openstack-infra/project-config/playbooks/base-test/pre.yaml git.openstack.org/openstack-infra/zuul-jobs/playbooks/unittests/pre.yaml git.openstack.org/openstack-infra/zuul-jobs/playbooks/tox/pre.yaml | 13:40 |
| dmsimard|off | I guess we could at a minimum put the playbook basedir | 13:40 |
| pabelanger | like you said, you can find the info by deep links | 13:41 |
| dmsimard|off | without sacrificing ux | 13:41 |
| pabelanger | extra_vars Parameter not saved by ARA due to configuration | 13:42 |
| pabelanger | so, what data do you see there, if it was saved? | 13:42 |
| pabelanger | the contents of extra_vars files? | 13:42 |
| dmsimard|off | pabelanger: no, it would be the actual string, like "-e @file.yml" | 13:43 |
| pabelanger | okay, cool | 13:43 |
| dmsimard|off | pabelanger: the contents is retrieved another way, and that's what I was discussing with mordred just now | 13:43 |
| mordred | yah - the file_cache | 13:43 |
| pabelanger | Ya, that what I thought so | 13:43 |
| pabelanger | so, we need to be super careful not to leak that :) | 13:43 |
| dmsimard|off | pabelanger: in 1.0 ara will pick up like everything that's loaded by ansible so I'll implement a blacklisting mechanism, probably pattern/regex based | 13:44 |
| dmsimard|off | pabelanger: ara even has access to fully decrypted vault files so.. | 13:44 |
| mordred | well, we don't use vault for anything, so that's ok :) | 13:44 |
| dmsimard|off | you'd find that this would probably be CVE worthy for some folks though :p | 13:45 |
| * mordred hasn't figured out how to make vault useful tbh | 13:45 | |
| pabelanger | looking at the files tab is interesting, should be helpful for people to understand out directory structure | 13:45 |
| mordred | it requires a password on invocation - which would imply people running ansible by hand, which seems crazy to me | 13:45 |
| mordred | pabelanger: ++ | 13:45 |
| dmsimard|off | mordred: the password can be in a file | 13:45 |
| pabelanger | dmsimard|off: do you server up facts some place? I see Gathering Facts task, but no results. Is that intentional? | 13:46 |
| pabelanger | serve* | 13:46 |
| dmsimard|off | pabelanger: in the hosts tab, hover or click on the hosts | 13:47 |
| pabelanger | okay cool, localhost is empty. | 13:47 |
| pabelanger | There is no host statistics for this playbook. | 13:47 |
| pabelanger | same with static.o.o | 13:47 |
| dmsimard|off | pabelanger: post-logs ? | 13:47 |
| pabelanger | ya | 13:48 |
| dmsimard|off | pabelanger: yeah sort of chicken and egg, looks like you're exporting as part of that playbook so the playbook hasn't finished yet | 13:48 |
| dmsimard|off | pabelanger: so it means that ara didn't run the v2_playbook_on_stats callback event for that playbook | 13:48 |
| mordred | well - it did, just not before we exported the html | 13:49 |
| dmsimard|off | pabelanger: see the icon on the left, it says it's incomplete | 13:49 |
| mordred | the end of post-logs will always be missing | 13:49 |
| pabelanger | Ya, that | 13:49 |
| pabelanger | otherwise, we'd need to move that into zuul, like we did for ssh-agent | 13:49 |
| dmsimard|off | ah I see, yeah, well.. not much ara can do about that unless an external process (ex: bash after the last ansible-playbook command) generates the html | 13:49 |
| mordred | yah - I think the benefit of having log publication be ansible content and therefor flexible by installation outweighs the gathering the logging of commands that do the logging | 13:50 |
| pabelanger | mordred: dmsimard|off: Looks good, at first glance I don't see any issues with the data ARA is generating. Everything there is what I would have expected to be displayed | 13:51 |
| mordred | pabelanger: I agree | 13:51 |
| dmsimard|off | pabelanger: so I was mentioning that there'll be more files in 1.0 than there are now -- stuff like meta/defaults/handlers files for roles for example, or hostvars and groupvars files | 13:52 |
| pabelanger | dmsimard|off: maybe at PTG we can start looking at moving js bit to CND, so dedup some of the data on logs.o.o | 13:52 |
| dmsimard|off | I'll also add host groups (which group a host is in), as well as what roles were invoked during that playbook | 13:53 |
| pabelanger | wouldn't be much work to start logs.o.o/ara/html-bit, or something. Then we'd maybe have a cfg file setting to use that path in html | 13:53 |
| dmsimard|off | pabelanger: yeah, we can definitely hack on that | 13:54 |
| dmsimard|off | The js/css/font files for ara are not negligible in size | 13:54 |
| dmsimard|off | btw where is the post-logs code ? | 13:55 |
| pabelanger | project-config | 13:55 |
| pabelanger | trusted playbooks | 13:55 |
| mordred | the roles are in git.openstack.org/openstack-infra/zuul-jobs/roles | 13:55 |
| pabelanger | dmsimard|off: I think that is from base-test | 13:56 |
| dmsimard|off | got it, thanks, sending a quick patch | 13:56 |
| mordred | yah - we're rolling out a bunch of changes to that right now | 13:56 |
| mordred | dmsimard|off: look at http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base-test/post-logs.yaml | 13:56 |
| mordred | and/or the end of this stack: https://review.openstack.org/#/c/495464/1 | 13:57 |
| mordred | speaking of - pabelanger, if you have a sec, I'd love ot land ^^ that stack | 13:57 |
| pabelanger | +3 | 13:59 |
| mordred | pabelanger: I think with those and 2 related project-config patches we'll be good to go with this and ready for the base job patch | 13:59 |
| pabelanger | mordred: approved them too | 14:01 |
| mordred | woot! | 14:02 |
| pabelanger | Exciting! | 14:02 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Document and update fileserver roles https://review.openstack.org/494291 | 14:02 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add zuul_return call into upload-logs role https://review.openstack.org/495461 | 14:03 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Copy inventory as part of validate host https://review.openstack.org/495464 | 14:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add support for gzipping static ARA reports https://review.openstack.org/495551 | 14:16 |
| dmsimard|off | mordred, pabelanger: fyi https://review.openstack.org/#/q/topic:ara-gzip | 14:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove bogus post_tasks line from upload-logs https://review.openstack.org/495553 | 14:18 |
| mordred | dmsimard|off: sweet | 14:18 |
| mordred | pabelanger: ^^ we landed an oops | 14:18 |
| mordred | :) | 14:18 |
| mordred | dmsimard|off: patch looks great - left two small suggestions that should be easy fixes | 14:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove bogus post_tasks line from upload-logs https://review.openstack.org/495553 | 14:25 |
| pabelanger | mordred: oops | 14:26 |
| mordred | pabelanger: I force-landed it. the syntax error broke the gate job. | 14:26 |
| pabelanger | k | 14:26 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add support for gzipping static ARA reports https://review.openstack.org/495551 | 14:38 |
| mordred | dmsimard|off: +2 | 14:43 |
| mordred | pabelanger: woot! https://review.openstack.org/#/c/495424/ ran with all the patches applied and it all looks good | 15:02 |
| dmsimard|off | lol, TFW you think maybe someone else figured out a solution but in the end it's the same hack :( https://github.com/openstack-infra/zuul/blob/7dfd7cae7a7c4c618ef6b987994fe779fb33933b/zuul/ansible/callback/zuul_stream.py#L492 | 15:03 |
| * dmsimard|off asks #ansible-devel | 15:04 | |
| pabelanger | mordred: Yay | 15:04 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Retrieve filtered list of hosts for a task, not all hosts https://review.openstack.org/495561 | 16:09 |
| dmsimard|off | turns out I found a solution ^ | 16:09 |
| dmsimard|off | (when figuring out the same issue in ara..) | 16:09 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Retrieve filtered list of hosts for a task, not all hosts https://review.openstack.org/495561 | 16:10 |
| dmsimard|off | btw have you started testing with ansible 2.4 ? | 16:18 |
| *** yolanda has quit IRC | 19:15 | |
| *** yolanda has joined #zuul | 19:18 | |
| mordred | dmsimard|off: not yet - it's on the TDL | 19:27 |
| mordred | SpamapS: https://review.openstack.org/#/c/495440 tiny #notaminusone nit | 19:57 |
| mordred | jeblair: if you happen to look in IRC - there's a weirdness with the base job (I merged the patch along with a fix that was immediately apparent) ... | 21:40 |
| mordred | jeblair: NEVERMIND I believe I see the issue | 21:41 |
| mordred | jeblair: tl;dr - openstack-publish-tarball removal job landed, but still had a reference to openstack-publish-tarball as a base job for release-openstack-python - this causes the config in zuul.yaml in project-config to be invalid - which, if I'm understanding correctly, means that zuul did not update the config from the config contained in that file | 21:45 |
| mordred | jeblair: however, the patch to add the secret to base came after that - so zuul hasn't been able to update its config to match what's in the project-config repo | 21:45 |
| mordred | this won't happen in "normal" operation as v3 would be the one responsible for landing changes, not v2, so the syntax checks would be binding | 21:46 |
| mordred | jeblair: I believe landing https://review.openstack.org/#/c/495659/ will allow v3 to be able to appropriately read the config from project-config and apply it, which will update the base job definition to include the secret as is currently in the file | 21:47 |
| mordred | pabelanger: you may also find the above interesting | 22:09 |
| mordred | yay that fixed it | 22:13 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!