| *** ErikB has joined #openstack-sahara | 00:08 | |
| *** ErikB has quit IRC | 00:25 | |
| *** ErikB1 has joined #openstack-sahara | 00:32 | |
| *** ErikB1 has quit IRC | 01:24 | |
| *** elmiko_ has quit IRC | 01:43 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| *** witlessb has quit IRC | 02:00 | |
| *** zhiyan_ is now known as zhiyan | 02:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/91245 | 02:46 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/91247 | 02:47 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-dashboard: Updated from global requirements https://review.openstack.org/91248 | 02:47 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-extra: Updated from global requirements https://review.openstack.org/91249 | 02:47 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-image-elements: Updated from global requirements https://review.openstack.org/91250 | 02:47 |
| *** IlyaE has quit IRC | 02:58 | |
| *** IlyaE has joined #openstack-sahara | 03:09 | |
| *** RockKuo_Office has joined #openstack-sahara | 03:14 | |
| *** IlyaE has quit IRC | 03:36 | |
| *** shakamunyi has quit IRC | 04:05 | |
| *** Ch00k has joined #openstack-sahara | 04:49 | |
| *** akuznetsov has joined #openstack-sahara | 05:00 | |
| *** zhiyan is now known as zhiyan_ | 05:20 | |
| *** Ch00k has quit IRC | 06:33 | |
| *** IlyaE has joined #openstack-sahara | 06:50 | |
| *** martsen has joined #openstack-sahara | 06:52 | |
| *** martsen has quit IRC | 06:56 | |
| *** ekarlso has joined #openstack-sahara | 07:17 | |
| *** witlessb has joined #openstack-sahara | 07:54 | |
| *** IlyaE has quit IRC | 08:02 | |
| *** Ch00k has joined #openstack-sahara | 08:13 | |
| *** mojorison has quit IRC | 09:29 | |
| openstackgerrit | A change was merged to openstack/sahara-image-elements: Updated from global requirements https://review.openstack.org/91250 | 09:30 |
| openstackgerrit | A change was merged to openstack/sahara-dashboard: Updated from global requirements https://review.openstack.org/91248 | 09:31 |
| openstackgerrit | A change was merged to openstack/sahara-extra: Updated from global requirements https://review.openstack.org/91249 | 09:31 |
| *** mojorison has joined #openstack-sahara | 09:42 | |
| *** tosky has joined #openstack-sahara | 09:48 | |
| openstackgerrit | A change was merged to openstack/sahara: Fix eventlet monkey patch and threadlocal usage https://review.openstack.org/72655 | 10:31 |
| *** openstackgerrit has quit IRC | 10:51 | |
| *** RockKuo_Office has quit IRC | 11:03 | |
| *** witlessb has quit IRC | 11:44 | |
| *** _mattf is now known as mattf | 11:53 | |
| *** witlessb has joined #openstack-sahara | 11:56 | |
| tellesnobrega | SergeyLukjanov: the use_floating_ips=False should be sahara config or in the local.conf? | 12:12 |
| mattf | fyi - http://www.redhat.com/about/news/press-archive/2014/4/red-hat-to-acquire-inktank-provider-of-ceph | 12:13 |
| *** ErikB1 has joined #openstack-sahara | 12:17 | |
| tellesnobrega | SergeyLukjanov: found it | 12:17 |
| *** ErikB1 has quit IRC | 12:27 | |
| *** tmckay has joined #openstack-sahara | 12:55 | |
| *** witlessb has quit IRC | 12:57 | |
| *** witlessb has joined #openstack-sahara | 12:59 | |
| *** elmiko has joined #openstack-sahara | 13:01 | |
| *** Ch00k has quit IRC | 13:06 | |
| *** Ch00k has joined #openstack-sahara | 13:08 | |
| *** ErikB1 has joined #openstack-sahara | 13:10 | |
| *** tmckay has quit IRC | 13:22 | |
| _crobertsrh | NikitaKonovalov: I think I understand your desire to reorganize the patch sets, but I think the chain of deps is useful for testers since it's fairly useless to test some of the functionality without the other panels. | 13:23 |
| tosky | question: the documentation does not specify if it's "better" to create a new user or not: http://docs.openstack.org/developer/sahara/userdoc/configuration.guide.html | 13:31 |
| tosky | but what do you usually do? Do you use the "admin" user, or do you go with the generic admin one? | 13:31 |
| tosky | sorry | 13:31 |
| tosky | do you use the generic "admin" user, or do you create a "sahara" one? | 13:31 |
| NikitaKonovalov | _crobertsrh: ok, let's keep a chain. Indeed there is no use of clusters withot node groups | 13:32 |
| _crobertsrh | Right, that's where my slightly odd chain came from | 13:32 |
| NikitaKonovalov | \me has to remove a bunch of -1 | 13:33 |
| _crobertsrh | Hopefully, I can get someone outside of our group to do some reviewing sometime soon :) | 13:33 |
| *** skolekonov has quit IRC | 13:36 | |
| dmitryme | tosky: I think in the 'right' deployment a new user should be created for Sahara | 13:41 |
| tosky | dmitryme: thanks | 13:42 |
| dmitryme | the one which will have access to 'service' tenant only | 13:42 |
| tosky | yeah, I tought so, just to be sure | 13:42 |
| dmitryme | but I know at least one bug which makes it not convenient | 13:42 |
| dmitryme | https://bugs.launchpad.net/sahara/+bug/1314124 | 13:42 |
| tosky | right, I hit it too, I had to change the code to force the tenant, I think we talked about that bug here | 13:47 |
| tosky | so what for now in IceHouse? | 13:48 |
| dmitryme | tosky: what do you mean? | 13:52 |
| tosky | what is the suggested way? Because you are likely to hit that bug | 13:52 |
| dmitryme | em, IMHO it is better to suggest 'the right way' but leave a notice about the bug, so the user could decide | 13:54 |
| *** openstackgerrit has joined #openstack-sahara | 13:56 | |
| tosky | I see, thanks | 13:57 |
| *** tmckay has joined #openstack-sahara | 13:59 | |
| *** witlessb_ has joined #openstack-sahara | 13:59 | |
| *** tmckay has left #openstack-sahara | 13:59 | |
| *** tmckay has joined #openstack-sahara | 14:00 | |
| *** witlessb has quit IRC | 14:01 | |
| *** witlessb_ is now known as witlessb | 14:01 | |
| *** Ch00k has quit IRC | 14:14 | |
| *** ErikB1 has quit IRC | 14:51 | |
| * tmckay learns that hive load data moves my source file from hdfs to a hive warehouse location. doh. | 14:52 | |
| tmckay | magic: CREATE EXTERNAL TABLE IF NOT EXISTS test (a INT) location '${INPUT}'; | 15:00 |
| *** IlyaE has joined #openstack-sahara | 15:01 | |
| *** ErikB1 has joined #openstack-sahara | 15:01 | |
| *** tmckay has quit IRC | 15:09 | |
| *** tmckay has joined #openstack-sahara | 15:38 | |
| *** vrovachev has quit IRC | 15:45 | |
| *** tmckay has quit IRC | 15:48 | |
| *** tmckay has joined #openstack-sahara | 15:48 | |
| openstackgerrit | Talusani Mani Shanker proposed a change to openstack/sahara: Closes-Bug 1312908, replaced the word components with component(s) https://review.openstack.org/91403 | 16:07 |
| *** tellesnobrega has quit IRC | 16:11 | |
| *** tellesnobrega has joined #openstack-sahara | 16:14 | |
| *** Ch00k has joined #openstack-sahara | 16:22 | |
| *** ErikB1 has quit IRC | 16:22 | |
| *** ErikB1 has joined #openstack-sahara | 16:22 | |
| *** mattf is now known as _mattf | 16:43 | |
| *** Manishanker has joined #openstack-sahara | 16:54 | |
| *** IlyaE has quit IRC | 17:19 | |
| *** _mattf is now known as mattf | 17:24 | |
| *** IlyaE has joined #openstack-sahara | 17:25 | |
| *** IlyaE has quit IRC | 17:32 | |
| *** mattf is now known as _mattf | 17:49 | |
| *** IlyaE has joined #openstack-sahara | 17:50 | |
| openstackgerrit | Talusani Mani Shanker proposed a change to openstack/sahara: Replaced the word components with component(s) in sahara/plugins/general/exceptions.py and sahara/tests/unit/service/validation/edp/test_job_executor.py https://review.openstack.org/91403 | 17:50 |
| *** tmckay is now known as _tmckay | 17:59 | |
| *** _tmckay is now known as tmckay | 18:21 | |
| *** _mattf is now known as mattf | 18:27 | |
| *** Manishanker has quit IRC | 18:44 | |
| *** _crobertsrh has left #openstack-sahara | 18:53 | |
| *** crobertsrh has joined #openstack-sahara | 18:55 | |
| *** tosky has quit IRC | 18:58 | |
| *** ErikB1 has quit IRC | 19:01 | |
| *** ErikB1 has joined #openstack-sahara | 19:01 | |
| *** Manishanker has joined #openstack-sahara | 19:01 | |
| *** IlyaE has quit IRC | 19:21 | |
| *** IlyaE has joined #openstack-sahara | 19:28 | |
| *** Ch00k has quit IRC | 19:43 | |
| tmckay | oh crobertsrh, yoohoo | 20:05 |
| crobertsrh | ?? | 20:05 |
| tmckay | :) hi | 20:05 |
| tmckay | I have an interesting use case | 20:06 |
| crobertsrh | Ok, let's have it | 20:06 |
| tmckay | I've been playing with hive, which apparently can't do swift. (Wherever the swift configs need to get to, they don't. We might need new hive jars for swift, side note) | 20:06 |
| tmckay | so, hdfs works. But what also works for hive is local data -- that is, data in the local run dir | 20:07 |
| tmckay | if you put a data file in the "libs" component of a job, and launch it via oozie, it's bundled with the script and hive can read it | 20:07 |
| crobertsrh | Heh | 20:08 |
| tmckay | crobertsrh, except that, in sahara, we have no concept of a local data source | 20:08 |
| crobertsrh | I think you mentioned the potential for this awhile back | 20:08 |
| tmckay | the INPUT parameter gets generated from the input data source, and there is no way to specify a path that is just a path | 20:08 |
| tmckay | that is, no "hdfs://" | 20:08 |
| crobertsrh | doh! | 20:08 |
| crobertsrh | So, new data source type? "Local"? | 20:09 |
| crobertsrh | that seems incomplete | 20:09 |
| tmckay | in fact, from what I see, "hdfs://" is optional for many things, even things that expect hdfs. I can make a relative hdfs path that knows to look in the current user | 20:09 |
| crobertsrh | since it's still "libs" that are really being overloaded here | 20:09 |
| tmckay | maybe "raw"? | 20:09 |
| tmckay | crobertsrh, the trouble kind of comes in this case because we are forcing the generation of the INPUT param, I think | 20:10 |
| crobertsrh | Yes | 20:10 |
| tmckay | I don't know that we need to force an input data source. If someone bundles it directly, or passes a custom arg to a hive script through substitution, we should let them | 20:10 |
| tmckay | hmm, didn't try leaving the in put path blank. I bet Sahara will barf. | 20:11 |
| crobertsrh | So, we would allow <None> as a possible job input | 20:11 |
| tmckay | yeah, thinking out loud, I think we should allow it. Output too, for that matter. With args/params, a user can reasonably do something else. If they want to leave data sources blank, maybe we should let them. But maybe we need a "are you sure" or "allow blank data sources" checkbox or something, because it's maybe not the normal case | 20:12 |
| crobertsrh | Yeah, there could be a UI note when "None" is chosen for either | 20:13 |
| tmckay | or something that says in a nice way whether to autogenerate the INPUT and OUTPUT parameters | 20:13 |
| tmckay | still fuzzing on the details, and experimenting. Currently launching from the oozie cli to see what works | 20:13 |
| tmckay | so, to summarize, current thoughts are to allow None, or somehow allow the dropping of "hdfs://" at the beginning | 20:14 |
| tmckay | (new data source type "raw path" or some way to drop the scheme value) | 20:14 |
| tmckay | crobertsrh, http://www.fpaste.org/98265/89135139/ | 20:19 |
| tmckay | crobertsrh, in this one I left the INPUT in the workflow and added a TABLE param and changed the hive script to use that instead | 20:19 |
| crobertsrh | Ah, very clever | 20:20 |
| tmckay | the hive script designates that the TABLE path is local. Note that the OUTPUT is hdfs, but there is no actual need for "hdfs://host:port" on the path | 20:20 |
| tmckay | I think we may need to massage path handling a little bit | 20:21 |
| *** IlyaE has quit IRC | 20:22 | |
| tmckay | crobertsrh, yeah, sahara is cracking the whip on no input or no output id | 20:24 |
| crobertsrh | Yeah, that was intentional | 20:24 |
| crobertsrh | We thought we knew best | 20:24 |
| tmckay | heh | 20:24 |
| tmckay | we used to :) | 20:24 |
| tmckay | now we know best better | 20:24 |
| *** IlyaE has joined #openstack-sahara | 20:25 | |
| crobertsrh | betterer, I say | 20:25 |
| tmckay | crobertsrh, alright, I'll think about this some more and maybe create a launchpad bug, maybe we can hit on the meeting tomorrow | 20:26 |
| tmckay | all I was trying to do was add hive to integration tests, lol | 20:26 |
| crobertsrh | For sure. If not, it would be a fine summit mini-topic | 20:26 |
| tmckay | test failed | 20:26 |
| tmckay | if you go to the trouble of uploading your input to hdfs on a long-running cluster it works fine | 20:27 |
| tmckay | I think I can still spin a test out of this, even given limitations | 20:29 |
| crobertsrh | Yeah, but clearly a target rich environment for improvement | 20:29 |
| *** Manishanker has quit IRC | 20:31 | |
| *** ErikB2 has joined #openstack-sahara | 20:39 | |
| *** ErikB1 has quit IRC | 20:41 | |
| *** crobertsrh is now known as _crobertsrh | 20:48 | |
| *** IlyaE has quit IRC | 21:08 | |
| *** elmiko is now known as elmiko_ | 21:20 | |
| *** IlyaE has joined #openstack-sahara | 21:25 | |
| *** ErikB2 has quit IRC | 21:39 | |
| *** ErikB2 has joined #openstack-sahara | 21:48 | |
| *** ErikB2 has quit IRC | 21:58 | |
| *** ErikB2 has joined #openstack-sahara | 22:03 | |
| *** elmiko_ has quit IRC | 22:22 | |
| *** witlessb has quit IRC | 22:31 | |
| *** ErikB2 has quit IRC | 22:42 | |
| *** IlyaE has quit IRC | 22:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!