| *** harlowja has quit IRC | 01:50 | |
| *** harlowja has joined #ara | 02:31 | |
| *** bcoca has quit IRC | 02:39 | |
| *** harlowja has quit IRC | 03:55 | |
| *** leucos has quit IRC | 04:55 | |
| *** leucos has joined #ara | 07:26 | |
| *** resmo has joined #ara | 08:10 | |
| *** leucos has quit IRC | 10:25 | |
| *** cliles_ has joined #ara | 10:48 | |
| *** cliles has quit IRC | 10:48 | |
| *** tbielawa has joined #ara | 12:46 | |
| *** bcoca has joined #ara | 13:19 | |
| *** bcoca has joined #ara | 13:19 | |
| *** Phinnay has joined #ara | 13:40 | |
| Phinnay | hey there | 13:41 |
|---|---|---|
| Phinnay | getting a weird error with the ara_write plugin - i have it in a playbook recording info from a set of VM's | 13:42 |
| Phinnay | the first VM is recorded just fine and any subsequent ones fail with the error: | 13:42 |
| Phinnay | [WARNING]: Failure using method (v2_runner_on_failed) in callback plugin (<ansible.plugins.callback.log_ara.CallbackModule object at 0x7f4deb3e35d0>): (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))") | 13:42 |
| Phinnay | another clue is further into the failure message: | 13:43 |
| Phinnay | "msg": "Data not recorded in ARA: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back | 13:43 |
| Phinnay | im not doing anything earth shattering with ara_record, just passing it a key and some json as a value | 13:46 |
| Phinnay | it *seems* like something is not closing or flushing properly when ara goes to move data to SQL | 13:52 |
| *** resmo has quit IRC | 13:59 | |
| dmsimard | Phinnay: hi | 14:02 |
| Phinnay | hullo | 14:02 |
| dmsimard | Just to get some background/context, how are you using it ? In a playbook ? a role ? against a single host ? many ? | 14:03 |
| Phinnay | in a playbook against numerous hosts | 14:04 |
| Phinnay | https://imgur.com/a/Nvi4A | 14:08 |
| Phinnay | the playbook in question | 14:08 |
| Phinnay | https://imgur.com/i620yKj | 14:13 |
| Phinnay | some run output | 14:13 |
| Phinnay | in this case only the VM in green passed the rest of them failed | 14:13 |
| Phinnay | all with the same failure message | 14:13 |
| dmsimard | Phinnay: hey, I'm not forgetting you but I'm on multiple threads right now | 14:37 |
| Phinnay | no worries, i appreciate you looking into this. | 14:38 |
| dmsimard | Phinnay: I can sort of point you in the direction I would look in ? | 14:38 |
| dmsimard | I honestly haven't tested ara_record against multiple hosts in the sense that it's really an action plugin (not an ansible module) and those are only ever executed locally on the control node (where ansible-playbook is run from) | 14:39 |
| dmsimard | In my usage of ara record, either I would record things in a play scoped to localhost or I would delegate to localhost | 14:39 |
| dmsimard | ultimately, we can probably chat about your use case and settle on something that works | 14:40 |
| Phinnay | alright, if im using it in a boneheaded way thats an acceptable answer ;p but having the ability to record data per node in this fashion has value. | 14:43 |
| Phinnay | as long as i can record per host items into ARA in some way it'd be great to explore options | 14:45 |
| dmsimard | yeah, I mean, ultimately it's still some sort of rudimentary key/value store but as long as keys are unique (which I feel like you're doing) it should work | 14:50 |
| Phinnay | right | 14:52 |
| Phinnay | the key is the node name and the data is a json array containing all the updates needed for the node | 14:52 |
| Phinnay | there will be duplicate data in the json but key will be unique | 14:52 |
| dmsimard | is that "MySQL server has gone away" error reproducible all the time ? | 14:55 |
| Phinnay | yes, each time i've run a playbook where it writes multiple VM's to ARA it gives that error | 14:56 |
| Phinnay | its random-ish what VM succeeds but its always one of the first few | 14:56 |
| dmsimard | Phinnay: can you try something ? It's totally a stab in the dark but worth trying | 15:00 |
| Phinnay | sure! | 15:00 |
| dmsimard | Phinnay: so there's this patch here that has had some amount of success with MySQL users but it's not merged yet for a mix of good and bad reasons: https://review.openstack.org/#/c/524427/ | 15:02 |
| dmsimard | Phinnay: could you install ara from source with that patch ? something like https://ara-community.slack.com/files/U6U5DKJ59/F9WA3RFHB/test_unmerged_patch.sh | 15:03 |
| dmsimard | (not sure if that slack link works publicly? it's a snippet I gave someone the other day) | 15:03 |
| Phinnay | i do not have access to that slack site | 15:04 |
| dmsimard | Phinnay: okay, here: http://paste.openstack.org/show/718303/ | 15:04 |
| Phinnay | ok i ran each of those lines as root, no errors | 15:07 |
| Phinnay | try running the playbook now? | 15:08 |
| dmsimard | Phinnay: so ara is installed as a system-wide package ? | 15:09 |
| Phinnay | correct | 15:09 |
| dmsimard | ok | 15:10 |
| dmsimard | that patch by itself is feature disabled, you need to add a config | 15:10 |
| dmsimard | sec | 15:10 |
| dmsimard | SpamapS: do you know what settings you're putting for the sql alchemy pool things in ara ? | 15:10 |
| ara-slack | <sethrei> Does ara have to be installed on the same server as the machine you are running ansible on ? I was trying to figure out how to have a separate web/db server from the machine I run ansible on. | 15:11 |
| dmsimard | @sethrei: nope, there's a FAQ that explains exactly that: https://ara.readthedocs.io/en/latest/faq.html#can-ansible-with-ara-run-on-a-different-server-than-the-web-application | 15:11 |
| ara-slack | <sethrei> sweet | 15:11 |
| dmsimard | @sethrei: long story short, your components just need to "agree" on where the database is located and have access to it | 15:12 |
| ara-slack | <sethrei> @dmsimard ahhhhhh ... that means I need to expose the port on my docker container ... gotcha | 15:13 |
| Phinnay | okay give me a second got a phone call | 15:14 |
| ara-slack | <sethrei> @dmsimard does the flask app need be running on both hosts ? | 15:28 |
| ara-slack | <dmsimard> The flask app is the web application server, it only reads from the database | 15:30 |
| ara-slack | <dmsimard> It's not required for the callback or the CLI (as long as each are configured to point to the right database) | 15:31 |
| ara-slack | <sethrei> @dmsimard okay ... I'm just trying to debug. Thanks | 15:31 |
| ara-slack | <sethrei> @dmsimard and a simple `ansible localhost -m setup ` should work ? | 15:33 |
| ara-slack | <dmsimard> Actually no, adhoc commands aren't recorded by ARA (yet) but it's a planned feature | 15:33 |
| ara-slack | <dmsimard> You can probably tell if your configuration is taken into account by checking if there's a ~/.ara/ansible.sqlite file | 15:34 |
| ara-slack | <dmsimard> If there's a database there, you're probably not using mysql | 15:34 |
| ara-slack | <sethrei> @dmsimard ahhh | 15:34 |
| ara-slack | <sethrei> @dmsimard I'm actually using Postgres ... but good to know adhoc doesn't work yet | 15:37 |
| ara-slack | <dmsimard> Postgre should work fine too | 15:37 |
| ara-slack | <dmsimard> There was a patch for postgre that wasn't merged though, would be nice if you could actually vouch for it (I don't use postgre) | 15:38 |
| ara-slack | <dmsimard> Oh nm that was mysql | 15:38 |
| ara-slack | <sethrei> @dmsimard I'm not able to connect yet ... so don't know | 15:39 |
| ara-slack | <sethrei> @dmsimard and there is no need to set `callback_whitelist` and `stdout_callback` ? | 15:45 |
| dmsimard | @sethrei: nope, don't need to set either of those | 15:46 |
| ara-slack | <sethrei> okay ... | 15:46 |
| dmsimard | @sethrei: you should have at a minimum two config parameters | 15:46 |
| dmsimard | For where you run Ansible from (the "control" node, can be your laptop or whatever) you need callback_plugins = <path to ara's plugins callbacks directory> under [defaults] in ansible.cfg | 15:47 |
| dmsimard | and then you also need something like "database = postgresql+psycopg2://user:password@dbserver:5432/ara" under the [ara] block | 15:48 |
| dmsimard | the web application server also needs to know about this database, depending on how you run the webapp this can be done through env variables, ansible.cfg or even SetEnv apache directives (if running with mod_wsgi) | 15:49 |
| dmsimard | The configuration is loaded in the exact same way as ansible, same order of precedence and everything | 15:49 |
| Phinnay | @dmsimard ok im back - confused as to what I have to do to enable the feature on the patch | 15:50 |
| dmsimard | @sethrei: so typically you'll have $PWD/ansible.cfg, ~/ansible.cfg (I think?), /etc/ansible/ansible.cfg and then environment variables have priority over any of those config | 15:50 |
| Phinnay | mysql is installed on the same PC as ARA/Ansible there is nothing remote | 15:50 |
| dmsimard | Phinnay: yeah, I was asking SpamapS if he could share his configuration for those sqlalchemy pool settings | 15:51 |
| ara-slack | <dmsimard> @harlowja ^ if you're around | 15:51 |
| ara-slack | <harlowja> File uploaded https://ara-community.slack.com/files/U6V9WRRTM/FA0ANPV0T/-.txt / https://slack-files.com/T6VAB05L7-FA0ANPV0T-6ad6ce43fb - our template for ansible.cfg.j2 | 15:52 |
| ara-slack | <dmsimard> @harlowja no cows? you're no fun | 15:53 |
| ara-slack | <harlowja> lol | 15:53 |
| Phinnay | lol what is the cows thing about | 15:53 |
| dmsimard | Phinnay: by default ansible gives the results with cowsay :) | 15:54 |
| dmsimard | https://michaelheap.com/cowsay-and-ansible/ | 15:54 |
| Phinnay | OH haha | 15:54 |
| dmsimard | Phinnay: do you configure ansible/ara with an ansible.cfg file ? | 15:54 |
| Phinnay | yes i do | 15:54 |
| dmsimard | Phinnay: ok, try adding this to your [ara] block: http://paste.openstack.org/raw/718309/ | 15:54 |
| dmsimard | need to afk for lunch, I'll be back later.. let me know if that seems to improve things for you ? | 15:55 |
| Phinnay | sure thing, im rerunning the playbook now | 15:56 |
| Phinnay | ok that had the most success... | 16:03 |
| Phinnay | https://imgur.com/hTNa6sD | 16:03 |
| Phinnay | but weird error: | 16:04 |
| Phinnay | u"Duplicate entry 'dcfffd09-0112-440b-bcd6-9a44e27c1e2e-Common-PRD-VM1' for key 'playbook_id'") [SQL: u'INSERT INTO hosts (id, playbook_id, name) VALUES (%(id)s, %(playbook_id)s, %(name)s)'] [parameters: {'playbook_id': u'dcfffd09-0112-440b-bcd6-9a44e27c1e2e', 'id': '473e8ff2-7773-4b05-a275-c8bbc9daa24a', 'name': u'Common-PRD-VM1'}] (Background on this error at: http://sqlalche.me/e/gkpj) | 16:04 |
| Phinnay | nothing in the playbook would duplicate that... | 16:04 |
| SpamapS | dmsimard: I'll looks | 16:04 |
| SpamapS | look even | 16:04 |
| SpamapS | ugh.. slack<->irc bridges are so.. weird. | 16:05 |
| dmsimard | SpamapS: harlowja gave me what I was looking for | 16:11 |
| dmsimard | Phinnay: that error is foreign to me | 16:16 |
| Phinnay | same! | 16:16 |
| Phinnay | no idea what would duplicate or why | 16:16 |
| dmsimard | Phinnay: tl;dr, there can only be one host named "something" for a playbook. Do you run multiple plays ? Default ansible strategy (i.e, not free or serial) ? | 16:17 |
| Phinnay | i understand that, there are no duplicate host names | 16:21 |
| Phinnay | what is odd though is it's trying to insert "dcfffd09-0112-440b-bcd6-9a44e27c1e2e-Common-PRD-VM1" into a column that does not contain data that looks like that | 16:22 |
| *** tbielawa is now known as tbielawa|lunch | 16:23 | |
| dmsimard | that's likely the name of the key | 16:23 |
| dmsimard | (the foreign key) | 16:23 |
| dmsimard | there's a foreign key from hosts.id (playbook_id) to playbooks.id | 16:24 |
| dmsimard | Phinnay: does that reproduce every time ? | 16:24 |
| ara-slack | <sethrei> @dmsimard thanks for your help, I fixed my docker container and it works. | 16:25 |
| dmsimard | @sethrei: yay \o/ | 16:26 |
| ara-slack | <sethrei> @dmsimard this will help me with all my 2.5 testing | 16:28 |
| ara-slack | <dmsimard> I haven't had the chance to try 2.5 all that much since it's come out, let me know if you notice anything weird in regards to ara :slightly_smiling_face: | 16:29 |
| ara-slack | <dmsimard> I know there's some unit test regressions but it seems to be a unit test bug rather than an ara (or ansible) bug. | 16:30 |
| Phinnay | i ran the playbook again and got different SQL errors | 16:30 |
| ara-slack | <sethrei> cool | 16:30 |
| Phinnay | https://imgur.com/RJAiPk5 | 16:30 |
| dmsimard | Phinnay: different? :/ | 16:30 |
| Phinnay | its the worst when the error changes o.x | 16:32 |
| dmsimard | Phinnay: Very weird. Do you have the ability to easily start the database from scratch ? Just in case there's some weird inconsistency with the database from the previous failed transactions... | 16:33 |
| Phinnay | ran the playbook 4x now and it hasnt bawked at duplicate keys in playbook_id, maybe the random ID generator got reset when the patch got applied? | 16:33 |
| dmsimard | Odd timing because I spent a good part of the weekend standing up better automated integration testing with mysql/postgresql that'll run in the CI whenever we send a patch | 16:34 |
| dmsimard | Phinnay: what version of ara was installed ? latest I suppose ? | 16:34 |
| Phinnay | but the next 3x playbook runs have been consistently erroring the same as shown in https://imgur.com/RJAiPk5 | 16:34 |
| dmsimard | Phinnay: can you run with -vvv ? that should expose the actual stack traces | 16:35 |
| Phinnay | sure | 16:35 |
| Phinnay | and yeah, i can blow away the DB if needed | 16:35 |
| dmsimard | let's try -vvv first | 16:36 |
| Phinnay | the ARA version on the website shows ARA 0.14.7.dev4 | 16:36 |
| dmsimard | yeah but what version was installed originally ? | 16:37 |
| dmsimard | if you didn't specify a version when installing it, it was the latest | 16:37 |
| Phinnay | yes, i did not specify a version | 16:38 |
| dmsimard | ok | 16:38 |
| *** sshnaidm is now known as sshnaidm|bbl | 16:39 | |
| Phinnay | https://pastebin.com/2GxCvdsc | 16:45 |
| Phinnay | showing the exception | 16:46 |
| Phinnay | i also see " "msg": "Data not recorded in ARA: (pymysql.err.OperationalError) (2006, \"MySQL server has gone away (error(32, 'Broken pipe'))\") [SQL: u'SELECT data.id AS data_id, data.playbook_id AS data_p$ }" but those follow after the one shown on pastebin | 16:47 |
| dmsimard | Phinnay: is that trace related to the first error we're seeing ? the one about the object not returning rows ? | 16:49 |
| dmsimard | also -- I'm curious if you'd run into any of those issues with the default sqlite backend | 16:52 |
| dmsimard | it would help isolate where the problem could be | 16:52 |
| Phinnay | https://jpst.it/1dczc | 16:58 |
| Phinnay | is the full output of ansible playbook -vvv > log.txt | 16:58 |
| Phinnay | i have not tried the default SQLlite backend, can give that a go | 16:59 |
| dmsimard | Phinnay: I'm mostly curious if we're dealing with a mysql specific issue or an ara_record specific issue (or something else entirely) | 17:01 |
| Phinnay | right | 17:02 |
| Phinnay | is there a process to switch back to sqllite or just comment out the database= line in ansible.cfg and restart apache | 17:06 |
| Phinnay | doesnt look like it :/ error cant find the sqlite file | 17:28 |
| *** tbielawa|lunch is now known as tbielawa | 17:39 | |
| Phinnay | ok in the mean time, attached a clean mySQL db and ran the playbook again - same errors | 17:42 |
| dmsimard | Ok, I don't have a solution for you right now until I have the time to understand what's going on | 17:43 |
| dmsimard | Using sqlite would be helpful to narrow down where to look | 17:44 |
| Phinnay | no problem, do you know how to switch back to sqllite? not sure how to generate the db | 17:44 |
| dmsimard | Just commenting out the database config should automatically create ~/.ara/ansible.sqlite | 17:45 |
| dmsimard | Unless you have a dir config | 17:45 |
| dmsimard | Basically dir defaults to ~/.ara and the database to $dir/ansible.sqlite | 17:46 |
| Phinnay | alrighty | 17:46 |
| Phinnay | ran with no errors on sqlite | 17:51 |
| Phinnay | all records populated as expected | 17:51 |
| Phinnay | :D | 17:51 |
| dmsimard | hmm | 18:03 |
| *** tbielawa has quit IRC | 18:47 | |
| *** jlozadad[m] has quit IRC | 18:47 | |
| *** DrWaluigi has quit IRC | 18:51 | |
| *** DrWaluigi has joined #ara | 18:52 | |
| *** tbielawa has joined #ara | 18:53 | |
| *** jlozadad[m] has joined #ara | 18:54 | |
| *** harlowja has joined #ara | 19:23 | |
| harlowja | Phinnay https://gist.github.com/harlowja/9c8dcfeb4697b11781126db1fab19a4c | 19:24 |
| harlowja | pretty sure u need to use this patch and lower SQLALCHEMY_POOL_TIMEOUT | 19:24 |
| Phinnay | hrmm, ill try it, give me a bit | 19:24 |
| harlowja | though u may then hit https://imgur.com/a/1ocsj and memory gets absorbed | 19:25 |
| * harlowja trying to see what may affect that | 19:25 | |
| dmsimard | harlowja: yeah actually I already had him try that patch | 19:27 |
| ara-slack | <harlowja> kk | 19:27 |
| harlowja | setting SQLALCHEMY_POOL_TIMEOUT below when your mysql server will drop the connection is helpful :) | 19:27 |
| harlowja | though i may try mysql-python soon | 19:28 |
| *** paulfantom has quit IRC | 19:33 | |
| *** paulfantom has joined #ara | 19:33 | |
| Phinnay | @harlowja my mysql timeout is set to 120seconds | 19:38 |
| Phinnay | pool_timeout is set to 60seconds | 19:39 |
| *** tbielawa has quit IRC | 20:14 | |
| harlowja | kk | 20:43 |
| *** sshnaidm|bbl is now known as sshnaidm | 21:09 | |
| *** bcoca has quit IRC | 23:00 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!