| *** esikachev has joined #openstack-sahara | 00:31 | |
| *** esikachev has quit IRC | 00:35 | |
| *** jeremyfreudberg has joined #openstack-sahara | 00:57 | |
| *** shuyingya has joined #openstack-sahara | 01:26 | |
| *** shuyingya has quit IRC | 01:30 | |
| *** shuyingya has joined #openstack-sahara | 01:31 | |
| *** esikachev has joined #openstack-sahara | 01:31 | |
| *** jeremyfreudberg has quit IRC | 01:35 | |
| *** esikachev has quit IRC | 01:36 | |
| *** caowei has joined #openstack-sahara | 01:37 | |
| *** esikachev has joined #openstack-sahara | 02:32 | |
| *** esikachev has quit IRC | 02:37 | |
| *** esikachev has joined #openstack-sahara | 03:33 | |
| *** esikachev has quit IRC | 03:38 | |
| *** links has joined #openstack-sahara | 03:39 | |
| *** esikachev has joined #openstack-sahara | 04:34 | |
| *** esikachev has quit IRC | 04:38 | |
| *** esikachev has joined #openstack-sahara | 04:42 | |
| *** anshul has joined #openstack-sahara | 04:49 | |
| *** esikachev has quit IRC | 04:56 | |
| *** caowei has quit IRC | 04:57 | |
| *** esikachev has joined #openstack-sahara | 05:15 | |
| *** esikachev has quit IRC | 05:21 | |
| *** esikachev has joined #openstack-sahara | 05:23 | |
| *** esikachev has quit IRC | 05:34 | |
| *** esikachev has joined #openstack-sahara | 05:36 | |
| *** caowei has joined #openstack-sahara | 05:37 | |
| *** pgadiya has joined #openstack-sahara | 05:52 | |
| *** esikachev has quit IRC | 06:06 | |
| *** esikachev has joined #openstack-sahara | 06:09 | |
| *** links has quit IRC | 06:15 | |
| *** pcaruana has joined #openstack-sahara | 06:22 | |
| *** rcernin has joined #openstack-sahara | 06:40 | |
| *** abalutoiu has joined #openstack-sahara | 07:01 | |
| *** hoonetorg has quit IRC | 07:04 | |
| openstackgerrit | Merged openstack/puppet-sahara master: Use Python to compute release notes version https://review.openstack.org/499328 | 07:16 |
|---|---|---|
| *** hoonetorg has joined #openstack-sahara | 07:17 | |
| *** links has joined #openstack-sahara | 07:28 | |
| *** tesseract has joined #openstack-sahara | 07:32 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** tosky has joined #openstack-sahara | 08:44 | |
| *** esikachev has quit IRC | 08:49 | |
| *** shuyingya has quit IRC | 09:04 | |
| *** shuyingya has joined #openstack-sahara | 09:05 | |
| *** shuyingya has quit IRC | 09:10 | |
| *** shuyingya has joined #openstack-sahara | 09:10 | |
| *** tosky has quit IRC | 09:18 | |
| *** tosky has joined #openstack-sahara | 09:20 | |
| *** esikachev has joined #openstack-sahara | 09:23 | |
| *** hoonetorg has quit IRC | 09:36 | |
| *** hoonetorg has joined #openstack-sahara | 09:53 | |
| *** raissa has quit IRC | 10:51 | |
| *** raissa has joined #openstack-sahara | 10:52 | |
| *** dave-mccowan has joined #openstack-sahara | 11:18 | |
| *** pgadiya has quit IRC | 12:37 | |
| *** shuyingya has quit IRC | 12:45 | |
| *** raissa has quit IRC | 12:52 | |
| *** esikachev has quit IRC | 12:56 | |
| *** dave-mccowan has quit IRC | 12:57 | |
| *** raissa has joined #openstack-sahara | 12:58 | |
| *** dave-mcc_ has joined #openstack-sahara | 13:09 | |
| *** shuyingya has joined #openstack-sahara | 13:10 | |
| tellesnobrega | shuyingya, do you have a devstack master deployed at the moment? | 13:22 |
| *** esikachev has joined #openstack-sahara | 13:26 | |
| *** jeremyfreudberg has joined #openstack-sahara | 13:26 | |
| *** openstackgerrit has joined #openstack-sahara | 13:26 | |
| openstackgerrit | Iwona Kotlarska proposed openstack/sahara-dashboard master: Add import of cluster templates to GUI. https://review.openstack.org/499598 | 13:26 |
| iwonka | tellesnobrega ^ | 13:26 |
| iwonka | and about the docs, there's not so much i coul add there | 13:27 |
| iwonka | i mean, probably like one sentence | 13:27 |
| iwonka | because it's very general | 13:27 |
| tellesnobrega | iwonka, but it should be there. Maybe we can find a better place for more details | 13:28 |
| iwonka | i saw some docs with list of REST API functions | 13:29 |
| iwonka | so i should add something there | 13:29 |
| iwonka | and probably to thing like dosc with list of sahara CLI commands | 13:29 |
| iwonka | i don't know if gui is documented | 13:29 |
| tellesnobrega | gui is not necessary | 13:30 |
| shuyingya | tellesnobrega, I have | 13:30 |
| tellesnobrega | the rest should | 13:30 |
| iwonka | also, about gui, i found a bug in horizon | 13:30 |
| iwonka | or i think it's a bug | 13:30 |
| iwonka | workflows do not work when you want to submit a file | 13:31 |
| tellesnobrega | shuyingya, can you test creating a vanilla node group template? | 13:31 |
| tellesnobrega | iwonka, that is interesting | 13:31 |
| tosky | REST API function are almost self-generated, but you can check | 13:31 |
| shuyingya | let me try | 13:31 |
| tellesnobrega | thanks | 13:31 |
| iwonka | because they don't pass request.FILES to the constructor of the form | 13:31 |
| iwonka | what should i do about it? | 13:32 |
| iwonka | is it intentional? | 13:33 |
| tellesnobrega | iwonka, first is go to #openstack-horizon channel and ask them | 13:33 |
| shuyingya | tellesnobrega, I got some error when created ngt through gui. | 13:33 |
| tellesnobrega | if they agree with you, you can go create a bug for it on launchpad for horizon and if you want you can fix it | 13:33 |
| jeremyfreudberg | iwonka, what is the use of request.FILES? I mean, I know kind-of what it does, but I mean what exactly do we need it for in our case? | 13:34 |
| shuyingya | let me enabled the debug mode to trace what happened | 13:34 |
| tellesnobrega | shuyingya, same here | 13:35 |
| tellesnobrega | that is happening for vanilla and cdh I think | 13:35 |
| iwonka | jeremyfreudberg: without it, when file is submitted it's not passed to handle (or somethng ike this, i don't remember exactly) | 13:35 |
| iwonka | i don't need it for import to work, because i didn't use workflows | 13:36 |
| jeremyfreudberg | iwonka, oh ok so this only is affecting workflows, not forms? | 13:37 |
| iwonka | jeremyfreudberg: yes | 13:37 |
| *** ssmith has joined #openstack-sahara | 13:38 | |
| *** jeremyfreudberg has quit IRC | 13:40 | |
| *** jeremyfreudberg has joined #openstack-sahara | 13:40 | |
| jeremyfreudberg | iwonka, cool, i guess feel free to chat with horizon folks about it then...good to know it's not really blocking your work | 13:40 |
| shuyingya | tellesnobrega: the error comes from django I think | 13:41 |
| shuyingya | ERROR django.security.TooManyFieldsSent The number of GET/POST parameters exceeded settings.DATA_UPLOAD_MAX_NUMBER_FIELDS. | 13:41 |
| shuyingya | WARNING django.server "POST /project/data_processing/clusters/configure-nodegroup-template HTTP/1.1" 400 151092 | 13:41 |
| tellesnobrega | that is weird | 13:41 |
| tellesnobrega | it works for spark and storm and mapr I think | 13:42 |
| jeremyfreudberg | shuyingya, tellesnobrega i will try in my env in a minute | 13:42 |
| tellesnobrega | jeremyfreudberg, thanks | 13:42 |
| jeremyfreudberg | tellesnobrega, shuyingya what version of Django? | 13:48 |
| shuyingya | Name: Django | 13:49 |
| shuyingya | Version: 1.11.4 | 13:49 |
| shuyingya | jeremyfreudberg | 13:50 |
| tellesnobrega | same here | 13:50 |
| tellesnobrega | shuyingya, did you find this error on horizon logs? | 13:50 |
| shuyingya | yes. the error are above. I use python manage.py runserver get the error message | 13:51 |
| tellesnobrega | hmm | 13:52 |
| tellesnobrega | trying to look on journalctl for horizon logs | 13:52 |
| jeremyfreudberg | i can confirm spark storm ambari mapr works, vanilla and cdh does not | 13:55 |
| tellesnobrega | same here | 13:57 |
| shuyingya | and get same error in journalctl | 13:57 |
| tellesnobrega | jeremyfreudberg, can you verify this is really a django issue? | 13:57 |
| tellesnobrega | do you also see that on your logs? | 13:57 |
| shuyingya | I will downgrade the django to 1.11.3 and try again | 13:57 |
| tellesnobrega | cool shuyingya | 13:58 |
| tellesnobrega | thanks | 13:58 |
| *** esikachev has quit IRC | 14:00 | |
| shuyingya | uhm, same error on django 1.11.1 & 1.11.3 | 14:06 |
| tellesnobrega | maybe we need to increase that settings.DATA_UPLOAD_MAX_NUMBER_FIELDS | 14:09 |
| jeremyfreudberg | tellesnobrega, default of that should be 1000 anyway, i'm trying to see if that's overwritten somewhere | 14:09 |
| tellesnobrega | jeremyfreudberg, 1000 should be enough for sure | 14:09 |
| shuyingya | https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-DATA_UPLOAD_MAX_NUMBER_FIELDS | 14:09 |
| tellesnobrega | how can we check how many fields are being sent? | 14:10 |
| jeremyfreudberg | shuyingya, seems like there is even a problem going all the back to Django 1.10, when that parameter was introduced... | 14:11 |
| shuyingya | actually, 1000 is not enough if the config option are being sent | 14:11 |
| tellesnobrega | hmmm | 14:11 |
| tellesnobrega | true | 14:12 |
| jeremyfreudberg | i am surprised then that only vanilla and cdh fail.... don't ambari and mapr have lots of config options too? | 14:13 |
| tellesnobrega | they do | 14:13 |
| shuyingya | jeremyfreudberg, it works on 1.9.1 | 14:14 |
| tellesnobrega | but maybe some hidden stuff there on vanilla and cdh | 14:14 |
| tellesnobrega | ok so we can say for sure that this is the limit there | 14:14 |
| tellesnobrega | is this a fix for us or horizon team? | 14:14 |
| shuyingya | yep | 14:14 |
| tellesnobrega | yep for us or horizon? | 14:15 |
| jeremyfreudberg | tellesnobrega, i think we can make it part of sahara-dashboard install guide to disable this check | 14:15 |
| tellesnobrega | or at least increase it | 14:15 |
| jeremyfreudberg | yes | 14:15 |
| tellesnobrega | I'm not familiar with django | 14:16 |
| tellesnobrega | where would this be changed? | 14:16 |
| tellesnobrega | config file? or in code? | 14:16 |
| jeremyfreudberg | this is in local_settings.py stuff | 14:16 |
| shuyingya | let me ask my colleague, he is expertise in horizon | 14:17 |
| tellesnobrega | thanks shuyingya | 14:17 |
| tellesnobrega | setting DATA_UPLOAD_MAX_NUMBER_FIELDS to none on horizon/local/local_setting.py does the trick | 14:27 |
| *** esikachev has joined #openstack-sahara | 14:28 | |
| tellesnobrega | I will talk with horizon folks and find out what is the best way to solve this | 14:28 |
| tellesnobrega | and file a bug | 14:29 |
| shuyingya | nice tellesnobrega :) | 14:29 |
| iwonka | tellesnobrega, jeremyfreudberg: so, --filename should be removed completly or should it be optional? | 14:36 |
| openstackgerrit | Iwona Kotlarska proposed openstack/sahara-dashboard master: Add import of cluster templates to GUI. https://review.openstack.org/499598 | 14:37 |
| tellesnobrega | iwonka, optional I like it | 14:37 |
| tellesnobrega | you give the user the option to print it to file | 14:37 |
| tellesnobrega | not needing the >> filename.json | 14:37 |
| iwonka | so if not given, then just print? | 14:37 |
| tellesnobrega | yes | 14:39 |
| tellesnobrega | jeremyfreudberg, ^ | 14:39 |
| jeremyfreudberg | hmmm i think optional could be good.... still need to think for a little bit. shuyingya, if you're still around, thoughts on OSC command for template export having optional choice of stdout *or* direct to file? | 14:40 |
| jeremyfreudberg | we are kind of going against tradition with either way, usually osc commands use the osc formatters for output, we are not using those regardless... | 14:41 |
| tellesnobrega | jeremyfreudberg, can you show me an example of that (osc formatters) | 14:42 |
| jeremyfreudberg | http://paste.openstack.org/raw/620088/ | 14:43 |
| jeremyfreudberg | idk exactly how it's implemented, but i'm just talking about how usually osc commands have a standard way of manipulating their output | 14:44 |
| jeremyfreudberg | which is a tradition we will have to break from regardless of what we choose about stdout vs. file | 14:44 |
| jeremyfreudberg | in any case, i've thought about it | 14:46 |
| jeremyfreudberg | i'm on board with optional | 14:47 |
| jeremyfreudberg | so iwonka please work on that if you can | 14:47 |
| *** abalutoiu has quit IRC | 14:47 | |
| shuyingya | searched in osc project, they seldom need an option to specify where to store the output. The content of template is not so much, from point of my view, output to std is a better way. | 14:50 |
| iwonka | jeremyfreudberg: this feature can't have output formatters anyway i guess, it has to be json… | 14:52 |
| jeremyfreudberg | iwonka, that's what i mean | 14:53 |
| jeremyfreudberg | shuyingya, i do see your point | 14:53 |
| jeremyfreudberg | maybe we should talk to osc folks to settle this? | 14:53 |
| shuyingya | yes. they must have more experience here | 14:54 |
| tellesnobrega | +1 | 14:59 |
| *** anshul has quit IRC | 15:04 | |
| openstackgerrit | Iwona Kotlarska proposed openstack/sahara-dashboard master: Add import of cluster templates to GUI. https://review.openstack.org/499598 | 15:22 |
| iwonka | when will it be decided what to do about cli things? | 15:24 |
| jeremyfreudberg | iwonka, after someone (i guess it has to be me?) talks to openstackclient people | 15:28 |
| jeremyfreudberg | of course, i can't even figure out what their irc channel name is | 15:28 |
| *** tellesnobrega has quit IRC | 15:29 | |
| *** tellesnobrega has joined #openstack-sahara | 15:29 | |
| jeremyfreudberg | tosky, do you know where the osc people hang out? | 15:30 |
| *** pcaruana has quit IRC | 15:32 | |
| tosky | jeremyfreudberg: isn't it #openstack-sdk ? | 15:37 |
| tosky | jeremyfreudberg: sorry, #openstack-sdks | 15:38 |
| tosky | yep: https://docs.openstack.org/python-openstackclient/latest/#contributing | 15:39 |
| jeremyfreudberg | tosky, indeed, thanks | 15:43 |
| *** abalutoiu has joined #openstack-sahara | 15:45 | |
| iwonka | okay, i asked on #openstack-sdks | 15:48 |
| jeremyfreudberg | i just joined the channel as well | 15:49 |
| openstackgerrit | Merged openstack/python-saharaclient master: Add import of Cluster Templates https://review.openstack.org/498574 | 15:49 |
| openstackgerrit | Merged openstack/sahara master: Add export of cluster templates https://review.openstack.org/498484 | 15:49 |
| *** shuyingya has quit IRC | 15:57 | |
| *** hoonetorg has quit IRC | 16:18 | |
| *** jeremyfreudberg has quit IRC | 16:34 | |
| ssmith | GM, how close is Sahara or how can it be made to be like AWS Redshift? | 16:57 |
| *** dave-mcc_ is now known as dave-mccowan | 17:01 | |
| * tosky does not know Redshift, apart from the name | 17:04 | |
| tellesnobrega | ssmith, same position as tosky here, what features you feel we are missing in sahara to be more like aws redshift | 17:05 |
| ssmith | The use case is a clustered data analytics engine with any one of backend data stores to which any adhoc sql query can be thrown at. <--I believe that's the best description of Redshift | 17:07 |
| *** rcernin has quit IRC | 17:12 | |
| *** tesseract has quit IRC | 17:14 | |
| tellesnobrega | ssmith, I think sahara can be provide at this point a clustered data analytics engine with any (swift, hdfs, manila) data stores | 17:14 |
| tellesnobrega | I'm not sure about the adhoc sql querys thou | 17:15 |
| *** esikachev has quit IRC | 17:18 | |
| ssmith | Hmmm... could be very similar. Sahara has an analytics engine, data store sources and a "Jobs" interface to construct queries | 17:29 |
| tellesnobrega | true | 17:30 |
| *** jeremyfreudberg has joined #openstack-sahara | 17:33 | |
| jeremyfreudberg | jumping in later here (and possibly repeating) but usually sahara is thought of as something equivalent to amazon EMR. both are involved in the provisioning of a | 17:36 |
| jeremyfreudberg | nd interaction with the hadoop ecosystem | 17:36 |
| jeremyfreudberg | redshift is another thing entirely | 17:36 |
| tellesnobrega | also true | 17:37 |
| tellesnobrega | as I said I'm not familiar with Redshift but the definition that ssmith gave is in part similar to sahara | 17:37 |
| tellesnobrega | if you stretch a little bit | 17:38 |
| jeremyfreudberg | they are both big data analytics, but the means of accomplishing is different | 17:38 |
| jeremyfreudberg | of course i'm not super familiar with redshift either | 17:38 |
| jeremyfreudberg | one big question is, what is the open source world's equivalent of redshift, if it exists | 17:40 |
| tellesnobrega | jeremyfreudberg, interesting question for sure | 17:40 |
| jeremyfreudberg | one smaller question is, if that thing exists, is it the role of trove, or sahara, to offer it | 17:40 |
| tellesnobrega | I don't thing trove would offer it | 17:41 |
| tellesnobrega | due to the analytics engine part of it | 17:41 |
| jeremyfreudberg | right, but amazon's redshift does eventually boil down to postgres, which is why i brought up trove. but i agree with your assumption | 17:42 |
| tellesnobrega | hmm | 17:42 |
| tellesnobrega | maybe cooperation between sahara and trove | 17:42 |
| tellesnobrega | but I don't advocate on it | 17:42 |
| *** abalutoiu has quit IRC | 17:43 | |
| jeremyfreudberg | hmmm opens a bit of a can of worms. | 17:44 |
| jeremyfreudberg | (that being said, I have sometimes had the crazy thought about what if sahara and trove were re-architectured to share some common bits) | 17:44 |
| tellesnobrega | jeremyfreudberg, hard thing to do | 17:45 |
| tellesnobrega | very hard thing to do | 17:45 |
| jeremyfreudberg | a dangerous topic for sure | 17:45 |
| ssmith | Well, the "job" can be the adhoc query somehow? Plus Sahara will scale out as well | 17:46 |
| jeremyfreudberg | ssmith, could be. | 17:48 |
| tellesnobrega | not sure how to make adhoc querys but it could be done I guess | 17:48 |
| jeremyfreudberg | but before we get to that point we still have to define what are the prerequisite services/etc to be running something redshift-equivalent | 17:48 |
| jeremyfreudberg | google is unclear about this, ssmith do you have any thoughts on it? | 17:49 |
| *** links has quit IRC | 17:52 | |
| jeremyfreudberg | tellesnobrega, we still have a meeting today right? | 17:53 |
| tellesnobrega | yes | 17:53 |
| jeremyfreudberg | cool | 17:53 |
| ssmith | Well other than create cluster, load data, analyze very quickly I don't have any other use case info from the CIO other than is there anything in OS that's like Redshift. Once I find out more I'll construct the outline and let you know. | 17:55 |
| tellesnobrega | ssmith, sounds good ssmith | 17:56 |
| jeremyfreudberg | ssmith, cool | 17:58 |
| ssmith | The most comparisons that I'm reading are Hadoop/YARN/Hive/Elastic Mapreduce which hell yes Sahara can do. I think the component that's missing is getting the queries into the system possibly via API so we'll see. | 17:59 |
| jeremyfreudberg | look into apache phoneix | 17:59 |
| jeremyfreudberg | *phoenix | 17:59 |
| jeremyfreudberg | creates an sql interface on top of hbase | 18:00 |
| jeremyfreudberg | hbase being column oriented and scalable like redshift | 18:00 |
| tosky | technically Sahara is "just" the component-in-the-middle, so most of the things that Sahara can do are the things that the supported plugins can do | 18:00 |
| jeremyfreudberg | but still integrating with the hadoop world so it could potentially be added to an existing sahara plugin in the future | 18:00 |
| tellesnobrega | and hbase will be supported on sahara soon (shuyingya's work) iirc | 18:00 |
| tosky | MEETING TIME! | 18:01 |
| jeremyfreudberg | indeed | 18:01 |
| tosky | #openstack-meeting-alt | 18:01 |
| tellesnobrega | yes, thanks tosky | 18:01 |
| jeremyfreudberg | iwonka, feel free to join if around | 18:01 |
| tellesnobrega | ssmith, also | 18:02 |
| ssmith | Phoenix looks interesting. Can it be deployed/managed by Sahara? | 18:02 |
| tellesnobrega | not right now | 18:03 |
| tellesnobrega | but we welcome new plugins | 18:03 |
| jeremyfreudberg | ssmith, i agree phoenix looks interesting... of course i just found out about it 5 minutes ago. dependent on hbase on sahara (coming very soon) | 18:03 |
| jeremyfreudberg | on our radar, i guess | 18:04 |
| *** dave-mccowan has quit IRC | 18:07 | |
| *** iwonka has quit IRC | 18:08 | |
| *** dave-mccowan has joined #openstack-sahara | 18:14 | |
| *** iwonka has joined #openstack-sahara | 18:14 | |
| *** brad[]` has quit IRC | 18:20 | |
| *** iwonka has quit IRC | 18:25 | |
| *** iwonka has joined #openstack-sahara | 18:43 | |
| *** esikachev has joined #openstack-sahara | 18:43 | |
| *** rcernin has joined #openstack-sahara | 19:01 | |
| *** hoonetorg has joined #openstack-sahara | 19:14 | |
| *** brad[] has joined #openstack-sahara | 19:21 | |
| *** shuyingya has joined #openstack-sahara | 19:33 | |
| *** shuyingya has quit IRC | 19:38 | |
| *** brad[] has quit IRC | 19:38 | |
| *** esikachev has quit IRC | 19:42 | |
| *** brad[] has joined #openstack-sahara | 20:06 | |
| *** jeremyfreudberg has quit IRC | 20:32 | |
| *** tosky has quit IRC | 20:35 | |
| openstackgerrit | Iwona Kotlarska proposed openstack/python-saharaclient master: Add export of cluster templates https://review.openstack.org/498520 | 21:11 |
| openstackgerrit | Iwona Kotlarska proposed openstack/python-saharaclient master: Add export of node group templates to CLI https://review.openstack.org/486721 | 21:18 |
| iwonka | export in commandline ^ | 21:18 |
| *** dave-mccowan has quit IRC | 21:20 | |
| ssmith | Actually, it looks like Phoenix has already been added? https://review.openstack.org/gitweb?p=openstack%2Fsahara.git;a=commitdiff;h=4626ad7ca95bf9b216b8163558e85bbce4afca2f | 21:23 |
| openstackgerrit | Iwona Kotlarska proposed openstack/python-saharaclient master: Add import of node group templates https://review.openstack.org/487575 | 21:24 |
| ssmith | Also see it in the Ambari UI | 21:39 |
| *** abalutoiu has joined #openstack-sahara | 21:42 | |
| *** abalutoiu has quit IRC | 21:43 | |
| *** ssmith has quit IRC | 22:00 | |
| *** tosky has joined #openstack-sahara | 22:27 | |
| *** rcernin has quit IRC | 22:30 | |
| *** tosky has quit IRC | 22:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!