Friday, 2019-01-25

*** Arsenick has joined #ara01:52
dmsimardWork in progress to bootstrap an Ansible role for installing ara-server: https://github.com/dmsimard/ansible-role-ara-server03:01
Arsenickdmsimard: That's a good news! What's different than the one in the openstack repo ?03:41
dmsimardArsenick: the one in the openstack repo is targetted at 0.x03:41
dmsimardI looked at what it would look like if I ported it to 1.0 and it didn't suit very well03:41
dmsimardRobert De Bock has a great Ansible role for ARA too: https://github.com/robertdebock/ansible-role-ara03:42
Arsenickok, I'm lookinf forward to test the new one then, we are using ara since your presentation at the rhug Quebec city ;)03:43
dmsimardArsenick: we have live demos at https://api.demo.recordsansible.org/api/v1/ and https://web.demo.recordsansible.org/03:44
dmsimardit doesn't look like much but most of the hard work is done03:44
dmsimard:)03:44
dmsimardthis new ara-server role is actually part of re-deploying api.demo from scratch after every commit03:45
Arsenicknice I didn't knew ara was on track for 1.0 soon, I might have missed the news!03:46
dmsimardwe'll be consumer of the role in a lot of ways so hopefully this will be a strong feedback loop03:46
dmsimardArsenick: part one https://ara.recordsansible.org/blog/2019/01/16/introducing-new-projects-for-the-upcoming-1.0-release/03:47
ArsenickIt's like a new world coming here, I've never found that blog nor getara.org in the topic lol.. I've always landed in the documentation or in the openstack github while searching for ara03:49
ArsenickThanks for the link03:49
dmsimardoh, getara.org is the old domain03:49
dmsimardbut it's still on the stickers so I'll just keep it and it's redirected03:50
dmsimardjust noticed it's not redirected right though03:50
ArsenickHehe yup!03:50
*** ChanServ sets mode: +o dmsimard03:50
*** dmsimard changes topic to "ARA Records Ansible: Making playbook runs easier to visualize, understand and troubleshoot | https://ara.recordsansible.org | Git: https://github.com/openstack/ara | Changelog: https://github.com/openstack/ara/releases | This channel is logged: http://eavesdrop.openstack.org/irclogs/%23ara/""03:51
*** dmsimard sets mode: -o dmsimard03:52
Arsenickbtw, I have a question for you. I'd like to know if it's possible to easily modify the way the playbook name is displayed Ex. smoke.yml in the web demo ? Here's a bit of context, ARA is installed on VSTS agents where our pipeline deploy the infra in azure and then run playbooks to configure them, but the agents are used by differents person and different pipeline, so the playbook smoke.yaml could be called by 3 different person/pipeline and it's a bit hard03:56
Arsenick to differentiate which one is mine. For now I'm using ara_record to set few value but it would be awesome if there was a way to display this in the name or anywhere else.. I don't know if my explanation is clear enough.. ;) I can say it in french that would probably be easier to understand! hah03:56
Arsenickor a global searchbox or a filter something along those line03:56
dmsimardArsenick: so there's going to be a few things that will be able to help for that in 1.003:56
Arsenicksweet03:57
dmsimardyou can give playbook names at runtime with an ansible variable (ara_playbook_name)03:58
dmsimardin the web client, the file path would probably be displayed if the playbook was not given a name03:59
Arsenickoh, I'll try with ara_playbook_name that could do the trick for now03:59
dmsimardthe other feature that might help is labels03:59
dmsimardbut I'm still thinking about the client side implementation04:00
dmsimardthe general idea is that you can tag playbooks with labels -- any label04:00
dmsimardso then it makes those labels searchable in the API (and web client)04:00
dmsimardthis is what it might look like: https://gist.github.com/dmsimard/de4a1dc51136e73054fb5f3515ac754b04:01
ArsenickI think that would be perfect for our usecase, that way I could tag with the username who launched the release, the name of the pipeline etc.. and if they are searchable we got everything!04:02
dmsimardArsenick: yes.04:02
dmsimard\o/04:02
Arsenickhehe04:02
dmsimardthere will be something similar for hosts04:02
dmsimardletting you search results for a specific host across multiple playbook runs04:03
dmsimardI need to write moar docs04:03
Arsenickyeah we all know that feeling04:04
Arsenick;)04:04
Arsenickanyway congrats for the project it's really usefull and well made! Keep up the good work04:07
ArsenickI'll try to stay here and help testing new releases, I'm really not a good dev so can't help on the code but I'll be happy to help whenever I can04:07
dmsimardI'm trying to make it easy to try out in different ways both for small and *larger* deployments04:08
dmsimardWe still have a few features to land but I've shifted a lot of focus to better integration testing and the other things around that, the roles, etc.04:09
dmsimardBut by doing that, I'm also writing tools that people can use to get started04:10
ArsenickGreat, I hope you'll come back show that to us at the RHUG once it's ready04:12
dmsimardyes, in fact I'm working with Michael to start organizing the next meetups04:12
ArsenickGood news! I can't remember you were from mtl  or qc ?04:12
dmsimardI've done both, haha04:12
dmsimardI've presented at the Ansible meetup at the Microsoft office from Quebec04:13
dmsimardand I've presented at several meetups in Montreal :p04:13
Arsenickyeah that's where I was, but I just can't recall if you are living in qc or mtl04:13
dmsimardMontreal :)04:14
dmsimardI drove to QC for the meetup haha04:14
Arsenickit's always great to find people working on foss in qc04:14
Arsenickok nice, it was great, you "sold" it to us ;)04:14
dmsimardselling free software \o/04:18
Arsenickhaha04:18
dmsimardit's actually a common misconception that red hat sells free software04:18
ArsenickYeah a lot of people need to read more about redhat and think about what it has done/give to the human race since the beginning..04:21
ArsenickEnough for tonight, see ya. Thanks again for the work you are doing! gnight04:22
dmsimard++04:23
*** etienne has joined #ara05:05
*** herald85 has joined #ara07:58
*** gvincent has quit IRC08:49
*** gvincent has joined #ara09:22
*** ssbarnea|bkp2 has joined #ara09:38
*** ssbarnea|rover has quit IRC09:39
*** gnupyx has joined #ara09:40
*** ssbarnea|bkp2 has quit IRC09:59
*** ssbarnea|rover has joined #ara10:00
*** ssbarnea|rover has quit IRC10:01
*** ssbarnea has joined #ara10:02
*** ssbarnea|bkp2 has joined #ara10:06
*** ssbarnea has quit IRC10:07
dmsimardapollo13, gvincent: In the context of updating api.demo automatically, I'm working on bootstrapping an Ansible role for ara-server: https://github.com/dmsimard/ansible-role-ara-server14:36
dmsimardhas a narrow set of supported deployment options (i.e, gunicorn and nginx) but we can add support for other things like uwsgi, mod_wsgi, etc. later on14:37
dmsimardI eventually want to use the role for integration testing as well :D14:37
gvincenthttpd + mod_wsgi or gunicorn IMO14:38
dmsimardgvincent: yes, the general idea is to support the most popular deployment configurations in the role so we can use the role to test these deployment configurations14:47
dmsimardWe need to test mysql and postgre too14:47
gvincenthum14:47
gvincentI disagree but ok :D14:48
Jmainguyneeds more oracle database14:51
Jmainguylook how popular they are14:51
dmsimardgvincent: what are you disagreeing on ?15:05
gvincentto support multiple storage backend15:06
dmsimardgvincent: it's supported out of the box by django, why not ?15:06
dmsimardalso, sqlite works well but has real limitations -- especially with concurrent writes15:06
gvincentwhy not? because it will introduce some extra work15:07
gvincentI prefer to choose one (postgresql) and test correctly with this backend15:07
dmsimard0.x supports mysql and postgre through sqlalchemy15:08
dmsimardbut isn't well tested15:08
gvincentbut as I said I'm ok with this, I just prefer one well tested instead of 2/3 backends not tested15:08
dmsimardgvincent: the cost of testing mysql is writing a job that sets up mysql and ara to use mysql15:08
dmsimard0.x has a hook for testing postgre -- it's designed to use the postgresql docker container15:09
gvincentand in the future some feature only avaialble on postgresql could not be used15:09
gvincentbut if you are motivated to support mysql and postgres ;D15:10
gvincentlet's go15:10
Jmainguysqlite is awesome15:12
dmsimardJmainguy: ++15:13
gvincentyeah until you want to do 2 writes in the same time15:13
dmsimardwe've scaled ara to >1 000 000 playbooks a month with sqlite in the openstack ci infrastructure :p15:13
JmainguyI had some problems writing an app to do that with sqlite, but eventually got past it15:15
Jmainguyhttps://github.com/Jmainguy/survey_hail a texting survey app thing I wrote, which stored texts in sqlite15:16
Jmainguyjust got to configure your app to not try writing 2x things at once =)15:17
Jmainguywait for sqlite to be writable and try then15:18
Jmainguydmsimard: gvincent what do you believe is the target market / expected architecture for most consumers of ARA?15:18
Jmainguyvms, containers, old reliable centos, or brand new fedora, full cluster of mysql / postgres, or sqlite15:19
gvincentJmainguy, dmsimard can answer this one, I don't know15:20
dmsimardJmainguy: right now we're testing against f29 and ubuntu bionic mostly because it brings us enough coverage without dragging our velocity down into attempting to support too many things out of the gate15:20
dmsimardwe're targetting python >=3.615:21
dmsimardso it does exclude a lot of things (CentOS 7/RHEL 7 doesn't ship python3, for example)15:21
dmsimardI suspect that we will have a client and callback implementation that is supported on py2 but only when using a remote API server -- py2 wouldn't work with the offline client.15:22
Jmainguyseems reasonable15:23
dmsimardIn terms of architecture, the "default" deployment is very similar to ara 0.x15:23
dmsimardYou don't need to start an API server -- you pip install ara-server and ara-plugins, set the callback in Ansible and you're ready to start recording things locally on a sqlite database15:24
Jmainguynice15:27
dmsimardIf users need more than that,  either because they're aggregating data from multiple locations or they need more performance/throughput, then we can start talking about a proper gunicorn/nginx setup or a apache/mod_wsgi setup.. maybe a mysql, maybe a postgre, etc15:27
Jmainguyfor my personal use case, where I would be hosting it in openshift, and having clients connect to it from laptops and vms15:27
JmainguyI would use gunicorn and mysql if needed, but if sqlite and gunicorn worked I would use that15:28
Jmainguysince those seem easier to containerize for my use case15:28
dmsimardBefore releasing, I want to run some benchmarks15:28
dmsimardto compare 0.x against 1.015:28
dmsimardbut also to benchmark different deployment scenarios in 1.015:28
*** etienne has quit IRC15:29
dmsimardJmainguy: when using a remote API (or database) server, the main concern I have is the impact of latency on playbook perforamnce15:29
dmsimardwhich is also why I tend to recommend using sqlite for as long as possible (to avoid the roundtrip to a remote database server)15:30
Jmainguymakes sense15:30
Jmainguyyeah i dont have a need for a full cluster of databases for ARA, if the remote api worked great with sqlite, I would definitly just use that mdoel15:31
Jmainguywould be much easier for me to deploy that way as well15:31
dmsimardthe one thing that is admittedly different in 1.0 is that the web interface doesn't come built-in15:32
dmsimardI mean, when you do "pip install ara" to install 0.x, it's a single package with everything in it (backend, frontend, callback, plugins, etc)15:32
dmsimardThe web interface isn't hard to run locally, it's like npm install/npm start but since it is stateless, it does requires an API server to connect to15:33
dmsimardfor example, https://web.demo.recordsansible.org/ is actually pulling real data from https://api.demo.recordsansible.org/15:34
*** TimNZ has quit IRC15:34
*** TimNZ has joined #ara15:35
dmsimardI think ara-server and ara-web are good candidates for putting in containers, there's going to be ansible roles for them too so hopefully it should be ok15:35
Jmainguynice15:35
Jmainguyyall are doing a great job, keep it up15:36
dmsimardtrying hard, thanks :D15:37
dmsimardapollo13: added comments on https://review.openstack.org/#/c/631287/ which otherwise looks good to me15:43
openstackgerritFlorian Apolloner proposed openstack/ara-server master: Added support for external auth via REMOTE_USER provided by webservers.  https://review.openstack.org/63128716:01
openstackgerritMerged openstack/ara-server master: Added very basic read/write authentication.  https://review.openstack.org/63128516:10
openstackgerritMerged openstack/ara-server master: First iteration of authentication and security docs  https://review.openstack.org/63218116:10
apollo13dmsimard: arg what did you just do :D16:10
dmsimardapollo13: ?16:10
apollo13https://review.openstack.org/#/c/632181/ zuul just said "change is closed" :D16:10
dmsimardapollo13: did you have comments for it ? I can address in a follow up16:11
apollo13I can no longer post comments there, so here is a screenshot: https://screenshots.firefox.com/1jBoogbKoDszHO3j/review.openstack.org16:11
apollo13btw, I think (didn't test it) that dynaconf allows us to use lower-cased names in settings files16:12
apollo13which might or might not be something to consider16:12
apollo13I personally find it hard to read uppercase settings, but ymmv (ie as in I do not care) :D16:12
dmsimardyeah I don't have a strong opinion either16:13
dmsimardapollo13: I'm not sure I follow your concern about "default"16:13
dmsimarddefault is the default env16:13
apollo13no it is not16:13
dmsimardis it not ?16:13
apollo13the default env is development16:13
dmsimardok but we're calling it default :p16:14
dmsimardI mean16:14
apollo13no we are not16:14
apollo13we just didn't notice :D16:14
apollo13see https://dynaconf.readthedocs.io/en/latest/guides/configuration.html16:14
apollo13ENV there says "development" is default16:14
dmsimardapollo13: yes, but we call the default env default16:14
dmsimardor wait16:14
dmsimardmaybe we don't16:15
apollo13did we somewhere tell dynaconf so?16:15
dmsimardhmmmm16:15
apollo13because otherwise, in https://dpaste.de/wqjp host=123 will win16:15
apollo13no matter what you put in default16:15
dmsimardapollo13: ah, this is where it comes from https://github.com/openstack/ara-server/blob/master/ara/server/settings.py#L21616:15
apollo13yes16:15
apollo13which is nice in the sense that [default] serves as fallback for __all__ env16:16
apollo13but technically the default env is development16:16
dmsimardok, I understand what you mean now16:16
dmsimardyes, I can clarify that16:16
dmsimardapollo13: the "provided by" part is an attempt at linking to other docs where relevant16:17
dmsimardi.e, I don't want to copy/paste django/drf/dynaconf docs in ara's docs16:17
dmsimardso it's kind of like a "read more" for users that might care16:17
apollo13yeah, I figured16:18
dmsimardapollo13: "privileges" might be a frenchism16:18
dmsimardI realize how it can be confusing, and now I'm not even sure it's a thing in english16:19
apollo13:D16:19
dmsimardyeah from a quick google it's not a thing16:19
dmsimardin french there is a word that looks like that and it means that is values something more than something else16:19
dmsimardsort of like a preference16:19
apollo13hurray, I could contribute something useful to docs^^16:20
apollo13https://github.com/openstack/ara-server/blob/master/.zuul.d/jobs.yaml#L15 anything stopping us from just getting rid of them?16:20
dmsimardsoon16:20
dmsimardin hindsight, it's a bit trickier than I had expected and I'm not positive how to properly do it16:20
dmsimard(yet)16:21
apollo13ok, what is the issue?16:21
dmsimardin ara-server, tox-linters actually shells out to a bash script which runs black, pep8, bandit, etc16:21
dmsimardit's a way to get feedback from all the linters in one shot instead of failing at the first occurrence and not running the ones after that16:22
apollo13yes16:22
dmsimardthe plan was to run this job against the other repos too16:22
dmsimardbut then it means we need that shell script in the other repos too ?16:22
apollo13ah16:22
dmsimardand then each project might have their own exceptions or w/e16:22
apollo13yikes16:23
apollo13I see, but we do not have a format env at all anymore in ara-server, so what does that offer?16:23
dmsimardin zuul, the configuration is shared across projects16:23
dmsimardara-clients and ara-plugins are still using that tox-format job16:23
apollo13ah okay, the comment could have said that ;)16:24
dmsimardIf zuul even lets us remove it (it might spit out an error because the job is still used elsewhere), it would break the other things16:24
dmsimardwell, I didn't expect that job to stick around that long16:24
apollo13yeah16:24
*** herald85 has quit IRC16:25
apollo13https://ara-server.readthedocs.io/en/latest/ look what I found :þ16:25
dmsimardyes it's actually not up to date16:25
dmsimardbecause the readthedocs webhook is broken16:25
dmsimardI need to trigger them manually for now16:25
apollo13ah16:26
dmsimardjust triggered it16:26
dmsimardso, what might happen with the linters job16:26
dmsimardis we could relocate it to ara-infra (where the other integration tests live) and have the linters job run the tests from ara-infra16:27
dmsimardpseudo-code: git clone https://github.com/openstack/ara-infra; ansible-playbook playbooks/linters.yaml16:27
Jmainguyis precedence the english version of the french "privleges"?16:27
Jmainguymeans one thing is more important than the other16:27
dmsimardJmainguy: precedence is a word in french too :)16:27
Jmainguywomp16:27
dmsimardJmainguy: dictionary proposes things like "focus on", "emphasize" or "prioritize" https://www.linguee.fr/francais-anglais/traduction/privil%C3%A9gier.html16:28
dmsimardthere's several words both in french and english that do not have a direct translation and it confuses my brain sometimes :p16:29
dmsimardapollo13: up to date (needed a ctrl+f5 for me tho): https://ara-server.readthedocs.io/en/latest/16:29
apollo13nice16:31
apollo13https://etherpad.openstack.org/p/ara-todo-1.0 looks rather nice16:34
dmsimardapollo13: the last big chunk is probably the CLI16:36
dmsimardthe equivalent to "ara playbook list" from 0.x, for example16:36
apollo13I've never used ara, so… :D16:36
dmsimardT_T16:36
* Jmainguy cries16:36
dmsimardapollo13: https://ara.readthedocs.io/en/stable/usage.html#querying-the-database-with-the-cli16:37
dmsimardapollo13: 0.x uses cliff as the CLI framework -- it has some nice features like different output formats (pretty tables, json, yaml, etc.), filtering of columns16:39
dmsimardI am not terribly attached to it16:40
apollo13I only know click, dunno, use whatever works for you :)16:41
apollo13generating nice tables seems like something nice though16:41
dmsimardI've tried click once (for the ara-slack bridge, infact)16:42
apollo13or just https://pypi.org/project/tabulate/#description16:42
dmsimardpretty tables aren't great for everything, though16:42
apollo13for the tables that is16:42
apollo13right16:42
dmsimardapollo13: attempting to display host facts through pretty tables is ugly: http://logs.openstack.org/22/631822/2/check/ara-integration-debian-py27-2.6.12/b5541b5/job-output.txt.gz#_2019-01-21_14_57_31_52348916:43
apollo13haha16:44
dmsimardapollo13: https://github.com/donnemartin/saws uses click and https://github.com/prompt-toolkit/python-prompt-toolkit16:51
apollo13oh, that looks nice16:52
dmsimardlol, aws has a lot of commands https://github.com/donnemartin/saws/blob/master/saws/data/SOURCES.txt16:54
apollo13well now you can probably imagine why security polcies in amazon are a nightmare17:12
dmsimardapollo13: do we care about MEDIA_ROOT and MEDIA_URL ?17:24
dmsimardI don't think we use them17:25
apollo13no, not as long as we allow uploads of files to the filesystem17:25
apollo13as we do not allow *17:25
dmsimardwould it break anything if we removed them from settings.py ?17:26
dmsimardI guess I could try17:26
apollo13nope, shouldn't change a thing17:26
apollo13we can probably even get rid of STATIC_ROOT/FILES if we use whitenoise to serve them17:27
dmsimardwhitenoise ?17:27
apollo13https://pypi.org/project/whitenoise/17:27
apollo13performance is obviously lower than letting apache/nginx do it, but I doubt we'd care17:27
dmsimardwell, the only thing we're serving is the DRF web interface, right17:27
apollo13admin interface too17:28
dmsimardand the admin panel, I guess17:28
dmsimardyeah17:28
dmsimardgood to know that it exists, we can consider it later17:28
openstackgerritDavid Moreau Simard proposed openstack/ara-server master: Misc updates to security and configuration documentation  https://review.openstack.org/63324817:30
openstackgerritDavid Moreau Simard proposed openstack/ara-server master: Add pip-wheel-metadata to gitignore  https://review.openstack.org/63324917:33
openstackgerritDavid Moreau Simard proposed openstack/ara-clients master: Fix pep8 issues  https://review.openstack.org/63325117:35
*** mgariepy has joined #ara19:43
*** etienne has joined #ara20:07
*** mgariepy has quit IRC20:25
*** med_ has quit IRC20:30
dmsimardapollo13, gvincent: once the last outstanding patches land, I'll tag alpha2 and upload to pypi21:08
openstackgerritMerged openstack/ara-clients master: Fix pep8 issues  https://review.openstack.org/63325121:16
openstackgerritMerged openstack/ara-server master: Added support for external auth via REMOTE_USER provided by webservers.  https://review.openstack.org/63128721:28
openstackgerritMerged openstack/ara-server master: Misc updates to security and configuration documentation  https://review.openstack.org/63324821:28
dmsimard1.0.0a2 tagged for ara-server, ara-plugins, ara-clients and ara-web -- everything but ara-web has been updated on pypi23:11
dmsimard- ara-server: https://github.com/openstack/ara-server/releases/tag/1.0.0a223:11
dmsimard- ara-clients: https://github.com/openstack/ara-clients/releases/tag/1.0.0a223:12
dmsimard- ara-plugins: https://github.com/openstack/ara-plugins/releases/tag/1.0.0a223:12
dmsimard- ara-web: https://github.com/openstack/ara-web/releases/tag/1.0.0a223:12
dmsimardLive demos have been updated as well, signing off for now, have a nice weekend everyone o/23:27

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!