| dmsimard | svg: I'm EST, let's chat sometime tomorrow :) | 00:16 |
|---|---|---|
| *** etienne has quit IRC | 03:10 | |
| *** dbpiv has joined #ara | 03:47 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #ara | 04:33 | |
| *** TKersten has joined #ara | 05:40 | |
| svg | OK, that makes a 6h difference. Your morning is my afternoon. | 08:34 |
| *** gvincent has quit IRC | 08:38 | |
| *** gvincent has joined #ara | 08:39 | |
| *** etienne has joined #ara | 13:28 | |
| *** TKersten has left #ara | 14:18 | |
| dmsimard | svg: o/ wanna chat ? | 14:48 |
| svg | dmsimard, good timing, yes | 14:49 |
| dmsimard | so just to set the table from a workflow perspective | 14:51 |
| dmsimard | tl;dr of object relationships: https://ara.readthedocs.io/en/latest/api-documentation.html#relationship-between-objects | 14:51 |
| dmsimard | so, for example, creating a play assumes that a playbook has been created and we have the playbook ID because a play is a "child" of a playbook | 14:52 |
| svg | yes, saw that happening in the code | 14:53 |
| dmsimard | similarly for tasks (requires playbook id and play id) and results (requires playbook id, play id, task id and host id) | 14:53 |
| dmsimard | I think there are improvement opportunities with the current callback without necessarily doing a full-on "fork" | 14:54 |
| svg | what's the difference between playbook-file and task-file? | 14:54 |
| dmsimard | there's a file for the playbook (say, playbook.yml) and for tasks (say, roles/foo/tasks/main.yml) | 14:55 |
| svg | ok, so I wrongly assumed those were all uploaded the same | 14:56 |
| svg | the starting part where it uploads all the yaml files, is also quite time-consuming. | 14:56 |
| dmsimard | improvement opportunities I see at a high level: 1) file creation can be threaded/async, by this point the callback has already created the playbook and we don't /need/ to do one file at a time 2) host stats update at the end (v2_playbook_on_stats) can be made threaded/async as well 3) _load_result ( | 14:57 |
| dmsimard | https://github.com/ansible-community/ara/blob/master/ara/plugins/callback/ara_default.py#L431 ) can probably be made threaded/async as well | 14:57 |
| dmsimard | I have a very naive and largely untested WIP patch implementing threading for _load_result here https://review.opendev.org/#/c/749378/1/ara/plugins/callback/ara_default.py | 15:02 |
| dmsimard | I'm not personally very experienced with threads and async things so very much open to feedback :D | 15:03 |
| dmsimard | could try adding the file creation in there and see what happens | 15:03 |
| svg | My Python's a bit rusty, and never worked on doing threaded/async stuff. You have any specifics on how to implement that? (just thinking out load:) IIRC, there's an async version of requests; or would it be an option to implement that directly in ara.client? | 15:04 |
| svg | is that patch available on github? I could test it. | 15:06 |
| *** openstackgerrit has joined #ara | 15:11 | |
| openstackgerrit | David Moreau Simard proposed recordsansible/ara master: Testing: callback threading https://review.opendev.org/749378 | 15:11 |
| dmsimard | it's not on github but you can check out the patch ^ from the git repository by running: git fetch https://review.opendev.org/recordsansible/ara refs/changes/78/749378/2 && git checkout FETCH_HEAD (it's at the top right in gerrit under "download") | 15:12 |
| svg | got it | 15:18 |
| openstackgerrit | David Moreau Simard proposed recordsansible/ara master: Testing: callback threading https://review.opendev.org/749378 | 15:20 |
| dmsimard | ^ added host facts to threading cause they can be async too | 15:20 |
| dmsimard | going to try with/without the patch with --check on my production and see what it looks like | 15:23 |
| dmsimard | just from tailing nginx logs, it does feel more async-y | 15:27 |
| svg | first comparison (with version 2 of your patch) - 119s wheras before that patch IO got ~156s | 15:29 |
| svg | those 119s compare with a test I did on a remote vm, which was local to the deployed nodes | 15:29 |
| dmsimard | did you pull in the host facts patchset too ? -- git fetch https://review.opendev.org/recordsansible/ara refs/changes/78/749378/3 && git checkout FETCH_HEAD | 15:30 |
| dmsimard | s/did you/can you/ | 15:31 |
| dmsimard | but hey, shaving ~40s with a relatively simple patch ain't terrible | 15:31 |
| dmsimard | doing a bit of a larger-scale test here, will report back when I have data | 15:32 |
| svg | I did now, and with patch 3 I get now 123s - basically no difference (but facts are cached in this setup, so I suppose it's normal since the setup module would not run again here) | 15:32 |
| svg | yes, that is definitely a good start | 15:32 |
| svg | btw, just to be sure, in the virtualenv where I run ansible, I update ara with a simple | 15:36 |
| svg | pip install /home/serge/src/ansible-community/ara | 15:36 |
| dmsimard | yup | 15:36 |
| svg | normally, instructions say to install ara[server] iirc - I'm not sure what that does differently? the [server] part that is | 15:36 |
| dmsimard | "pip install ara" installs only the Ansible plugins (like the callback) as well as the API clients -- ara[server] does that and also pulls in the API server dependencies | 15:37 |
| dmsimard | the API server dependencies aren't required if you're not using the offline API client or if you don't plan on running the API server | 15:38 |
| dmsimard | so the base package is very small, actually | 15:38 |
| svg | ok | 15:40 |
| svg | shouldn't the self.log.debug statement show with ANSIBLE_DEBUG=1? | 15:43 |
| svg | or they don't show up because it's async in another thread? | 15:44 |
| svg | when running my firsdt play, there's around 20 seconds of logging nothing (and IIRC that's where it uploads yaml?) | 15:45 |
| dmsimard | try with ARA_DEBUG=true and ARA_LOG_LEVEL=DEBUG -- but it might require server dependencies to be installed because of https://github.com/ansible-community/ara/issues/112 | 15:45 |
| dmsimard | yeah that's probably the part where the files are uploaded | 15:45 |
| dmsimard | there's a tradeoff for file uploads -- files are stored uniquely in the database (checksum'd a bit like git) but the callback doesn't know if the file has already been uploaded before uploading it.. the callback /could/ do an API call first to check if the checksum already exists but then it's a second API call to upload the file if it hasn't been | 15:47 |
| dmsimard | created yet | 15:47 |
| svg | can both happen in a separate thread? | 15:48 |
| dmsimard | but then again, we still need to create a file object so it's part of the playbook -- though the *content* of the file can reference something that already exists | 15:48 |
| svg | yeah, can't seem to make it log, even when installing [server] | 15:49 |
| dmsimard | should work, looks like this: http://paste.openstack.org/show/798998/ | 15:52 |
| dmsimard | got data with threads, testing without threads now | 15:56 |
| svg | afk for coffee update, brb | 15:56 |
| dmsimard | I got largely the same performance (literally a second of difference) with and without threads :/ | 16:10 |
| svg | local client or http? | 16:11 |
| dmsimard | 22 hosts, 74 plays, 2754 results, 398 files in 14m51s | 16:11 |
| dmsimard | http to a server between 10 and 20ms away, backed by mysql | 16:11 |
| dmsimard | there's probably a margin of error (sample count being 1 with and 1 without) but it didn't seem like it made a huge difference -- going to try recording locally with and without and see what happens | 16:12 |
| svg | of course, threading & python might not be the best option here | 16:13 |
| dmsimard | don't really have a choice for python, it's the interface provided by ansible :p | 16:15 |
| dmsimard | the rationale for threading is to avoid ansible getting blocked on things that should be non-blocking | 16:15 |
| dmsimard | otherwise ansible waits until the callback hook completes to continue | 16:15 |
| dmsimard | like, the callback should return asap so ansible can continue and then post whatever it needs to post in the background | 16:16 |
| dmsimard | btw what version of python and ansible are you testing with ? the runs I did were with ansible 2.7 and python 3.7.9 | 16:17 |
| dmsimard | I'll try with 2.9 and python 3.8 out of curiosity after I'm done with this current roud | 16:17 |
| svg | 3.8.6 | 16:21 |
| dmsimard | svg: and what version of Ansible ? | 16:25 |
| svg | 2.9.12 | 16:27 |
| dmsimard | so I tested against a local API server with the http client and it brought down the duration from 14m51s to 8m30s but there is still no meaningful difference between with or without threads... trying newer version of ansible and python now | 16:46 |
| svg | I'm off for the day. Be sure to let me know if I should test anything. Thank you for looking into this. | 17:05 |
| dmsimard | sure, I'll finish gathering data and put it up in the issue: https://github.com/ansible-community/ara/issues/171 | 17:08 |
| dmsimard | I think it'd be worth having a synthetic benchmark playbook so we can easily test different versions of ansible/python -- don't have that kind of flexibility in production (unfortunately) | 17:10 |
| dmsimard | no meaningful difference with 2.9.12 and 3.8.9 either :/ | 17:33 |
| dmsimard | 2.9.14* | 17:33 |
| *** sshnaidm is now known as sshnaidm|afk | 18:24 | |
| *** mgariepy has quit IRC | 23:24 | |
| *** mgariepy has joined #ara | 23:34 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!