| *** alazarev has quit IRC | 00:06 | |
| *** bandarji has quit IRC | 00:17 | |
| *** alazarev has joined #openstack-sahara | 00:36 | |
| *** ViswaV_ has joined #openstack-sahara | 01:39 | |
| *** ViswaV has quit IRC | 01:41 | |
| *** alazarev has quit IRC | 01:42 | |
| *** bradd1 has joined #openstack-sahara | 02:13 | |
| *** ViswaV has joined #openstack-sahara | 02:24 | |
| *** ViswaV_ has quit IRC | 02:26 | |
| *** bradd1 has quit IRC | 03:13 | |
| *** akuznetsov has joined #openstack-sahara | 03:33 | |
| *** akuznetsov has quit IRC | 04:08 | |
| *** akuznetsov has joined #openstack-sahara | 04:24 | |
| *** akuznetsov has quit IRC | 04:31 | |
| *** alazarev has joined #openstack-sahara | 04:49 | |
| *** akuznetsov has joined #openstack-sahara | 04:57 | |
| *** alazarev has quit IRC | 05:01 | |
| *** ViswaV has quit IRC | 05:15 | |
| *** yanndegat has quit IRC | 05:15 | |
| *** _elmiko has quit IRC | 05:15 | |
| *** _crobertsrh has quit IRC | 05:15 | |
| *** ekarlso has quit IRC | 05:15 | |
| *** aignatov has quit IRC | 05:15 | |
| *** SergeyLukjanov has quit IRC | 05:15 | |
| *** NikitaKonovalov has quit IRC | 05:15 | |
| *** dmitryme has quit IRC | 05:15 | |
| *** boris-42 has quit IRC | 05:15 | |
| *** ghenriks has quit IRC | 05:15 | |
| *** bogdando has quit IRC | 05:15 | |
| *** _mattf has quit IRC | 05:15 | |
| *** venza has quit IRC | 05:15 | |
| *** ruhe has quit IRC | 05:15 | |
| *** sreshetnyak has quit IRC | 05:15 | |
| *** akuznetsov has quit IRC | 05:15 | |
| *** openstackgerrit has quit IRC | 05:15 | |
| *** akuznetsov has joined #openstack-sahara | 05:27 | |
| *** ViswaV has joined #openstack-sahara | 05:27 | |
| *** openstackgerrit has joined #openstack-sahara | 05:27 | |
| *** ghenriks has joined #openstack-sahara | 05:27 | |
| *** _elmiko has joined #openstack-sahara | 05:27 | |
| *** yanndegat has joined #openstack-sahara | 05:27 | |
| *** ruhe has joined #openstack-sahara | 05:27 | |
| *** sreshetnyak has joined #openstack-sahara | 05:27 | |
| *** bogdando has joined #openstack-sahara | 05:27 | |
| *** _crobertsrh has joined #openstack-sahara | 05:27 | |
| *** ekarlso has joined #openstack-sahara | 05:27 | |
| *** boris-42 has joined #openstack-sahara | 05:27 | |
| *** _mattf has joined #openstack-sahara | 05:27 | |
| *** venza has joined #openstack-sahara | 05:27 | |
| *** aignatov has joined #openstack-sahara | 05:27 | |
| *** dmitryme has joined #openstack-sahara | 05:27 | |
| *** NikitaKonovalov has joined #openstack-sahara | 05:27 | |
| *** SergeyLukjanov has joined #openstack-sahara | 05:27 | |
| *** akuznetsov has quit IRC | 05:28 | |
| *** ViswaV has quit IRC | 05:28 | |
| *** yanndegat has quit IRC | 05:28 | |
| *** _elmiko has quit IRC | 05:28 | |
| *** _crobertsrh has quit IRC | 05:28 | |
| *** ekarlso has quit IRC | 05:28 | |
| *** aignatov has quit IRC | 05:28 | |
| *** SergeyLukjanov has quit IRC | 05:28 | |
| *** NikitaKonovalov has quit IRC | 05:28 | |
| *** dmitryme has quit IRC | 05:28 | |
| *** boris-42 has quit IRC | 05:28 | |
| *** ghenriks has quit IRC | 05:28 | |
| *** bogdando has quit IRC | 05:28 | |
| *** _mattf has quit IRC | 05:28 | |
| *** venza has quit IRC | 05:28 | |
| *** ruhe has quit IRC | 05:28 | |
| *** sreshetnyak has quit IRC | 05:28 | |
| *** openstackgerrit has quit IRC | 05:28 | |
| *** ViswaV has joined #openstack-sahara | 05:33 | |
| *** openstackgerrit has joined #openstack-sahara | 05:33 | |
| *** ghenriks has joined #openstack-sahara | 05:33 | |
| *** _elmiko has joined #openstack-sahara | 05:33 | |
| *** yanndegat has joined #openstack-sahara | 05:33 | |
| *** ruhe has joined #openstack-sahara | 05:33 | |
| *** sreshetnyak has joined #openstack-sahara | 05:33 | |
| *** bogdando has joined #openstack-sahara | 05:33 | |
| *** _crobertsrh has joined #openstack-sahara | 05:33 | |
| *** ekarlso has joined #openstack-sahara | 05:33 | |
| *** boris-42 has joined #openstack-sahara | 05:33 | |
| *** _mattf has joined #openstack-sahara | 05:33 | |
| *** venza has joined #openstack-sahara | 05:33 | |
| *** aignatov has joined #openstack-sahara | 05:33 | |
| *** dmitryme has joined #openstack-sahara | 05:33 | |
| *** NikitaKonovalov has joined #openstack-sahara | 05:33 | |
| *** SergeyLukjanov has joined #openstack-sahara | 05:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/97006 | 06:01 |
|---|---|---|
| *** jpich has joined #openstack-sahara | 06:45 | |
| *** witlessb has joined #openstack-sahara | 07:12 | |
| *** julienvey has joined #openstack-sahara | 08:18 | |
| *** akuznetsov has joined #openstack-sahara | 08:18 | |
| *** akuznetsov has quit IRC | 08:28 | |
| *** vrovachev has joined #openstack-sahara | 08:36 | |
| *** akuznetsov has joined #openstack-sahara | 08:38 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 09:57 | |
| *** skolekonov has joined #openstack-sahara | 10:41 | |
| *** skolekonov has quit IRC | 10:52 | |
| *** Timotey has joined #openstack-sahara | 11:07 | |
| *** akuznetsov has joined #openstack-sahara | 11:34 | |
| *** akuznetsov has quit IRC | 11:36 | |
| *** akuznetsov has joined #openstack-sahara | 11:37 | |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Add Spark 1.0.0 to the version list https://review.openstack.org/98380 | 11:38 |
| *** ViswaV_ has joined #openstack-sahara | 12:23 | |
| *** ViswaV has quit IRC | 12:26 | |
| *** _crobertsrh is now known as crobertsrh | 12:57 | |
| *** miqui has joined #openstack-sahara | 12:58 | |
| *** _elmiko is now known as elmiko | 13:02 | |
| *** witlessb_ has joined #openstack-sahara | 13:40 | |
| *** witlessb has quit IRC | 13:42 | |
| *** witlessb_ is now known as witlessb | 13:42 | |
| *** vrovachev has quit IRC | 13:48 | |
| *** ViswaV has joined #openstack-sahara | 14:48 | |
| *** Timotey has quit IRC | 14:52 | |
| *** ViswaV_ has quit IRC | 14:52 | |
| *** tmckay has quit IRC | 14:59 | |
| *** tmckay has joined #openstack-sahara | 15:01 | |
| *** yanndegat has left #openstack-sahara | 15:14 | |
| *** akuznetsov has quit IRC | 15:18 | |
| *** akuznetsov has joined #openstack-sahara | 15:19 | |
| *** openstackgerrit has quit IRC | 15:22 | |
| *** bradd1 has joined #openstack-sahara | 15:26 | |
| *** jpich has quit IRC | 15:28 | |
| *** openstackgerrit has joined #openstack-sahara | 15:28 | |
| *** openstackgerrit has quit IRC | 15:53 | |
| *** IvanBerezovskiy has left #openstack-sahara | 16:04 | |
| *** alazarev has joined #openstack-sahara | 16:13 | |
| alazarev | tmckay: hi, want to talk about spurk and pluggable EDP, do you have time? | 16:18 |
| tmckay | yes, sure | 16:19 |
| *** elmiko is now known as _elmiko | 16:21 | |
| tmckay | alazarev, I started looking at the job manager yesterday, I think it's not too hard to refactor and not have direct dependence on Oozie. | 16:21 |
| tmckay | I'm hoping to make a draft CR for comments | 16:21 |
| alazarev | tmckay: yes, and this is what I wanted to say | 16:21 |
| alazarev | now factory is on “workflow” level | 16:22 |
| alazarev | we can move it up and do it on “manager” level | 16:22 |
| tmckay | yes, I agree -- I think I was coming to the same place :) | 16:23 |
| alazarev | in this case managers for oozie jobs will know about oozie and specific workflow | 16:23 |
| alazarev | managers for other jobs will know what they need | 16:23 |
| tmckay | yes, agreed | 16:23 |
| alazarev | what was the reason for taking oozie as default engine? we don’t get much benefits from it | 16:24 |
| tmckay | alazarev, first pass may not be perfect but it will be good enough for lots of discussion on the CR | 16:24 |
| tmckay | on oozie .... | 16:24 |
| alazarev | support of complex workflows in the future? | 16:24 |
| tmckay | alazarev, when we started EDP, we were trying to do it very fast. We were still drawing on the white board, and we wanted something to work for Havanna | 16:24 |
| tmckay | so, someone (aignatov? I'd have to dig up the pad) suggested Oozie because it gave us a general job manager for all the initial job types without us doing a lot of work | 16:25 |
| alazarev | may be it is good idea to move out of oozie for some of other jobs, need to think about this more | 16:26 |
| tmckay | So, I think it was strictly a pragmatic decision. Having the job engine let us focus on making the edp objects, UI, general workflow, etc, without worrying about job management details | 16:26 |
| tmckay | alazarev, I agree, now that we have basic functionality, we should generalize. There is too much specific Oozie knowledge in the job manager right now. | 16:26 |
| tmckay | In fact, tiny detail -- | 16:26 |
| tmckay | get_oozie_server() is used as a way to get an instance for uploading files to hdfs! | 16:27 |
| tmckay | It shouldn't be. It should be looking for a namenode, or a datanode. Anything will do for hdfs | 16:27 |
| tmckay | And, it will be common, because Spark runs from hdfs as well | 16:28 |
| alazarev | yes, agree | 16:28 |
| tmckay | some small things like that are obvious improvements. | 16:28 |
| tmckay | alazarev, I look forward to your comments, I really hope to have a simple refactor up in the next couple of days, something to use as a working model | 16:29 |
| tmckay | I think it will help us think | 16:29 |
| alazarev | it is also used to make proxy http call to oozie API | 16:29 |
| tmckay | yes, but the hdfs parts can be changed | 16:29 |
| alazarev | ^ oozie_server | 16:29 |
| alazarev | agree | 16:31 |
| tmckay | we may have done something better up front, but myself and crobertsrh were just getting up to speed and aignatov and nprivalova were working very quickly with Havanna right around the corner | 16:31 |
| tmckay | EDP work == slightly crazy | 16:31 |
| alazarev | make it somehow and refactor next is a good approach :) | 16:32 |
| alazarev | and one more question, why don’t we support input/output sources for Java jobs? | 16:32 |
| alazarev | we can pass them as arguments | 16:33 |
| alazarev | for other jobs we specify input/output params _in addition_ to other params | 16:34 |
| alazarev | but for java we require to set them manually | 16:34 |
| tmckay | alazarev, yes, I talked to crobertsrh about ways to make that more consistent. The reason is that the other job types (pig, hive, mapreduce, mapreduce.streaming) all require an input source and an output source. They don't make sense without one. So, Sahara requires it, and autogenerates the arguments | 16:34 |
| tmckay | For Java actions, there may not be any input or output source. Could just be a calculation and screen dump, like the Pi example. | 16:35 |
| alazarev | so, as always, to make thnks as generic as possible | 16:35 |
| alazarev | things | 16:35 |
| *** openstackgerrit has joined #openstack-sahara | 16:35 | |
| tmckay | So, maybe we should change it, so that you can at least indicate that a data source should be used as an argument, and then Sahara gets the path out and passes it as an argument for the user | 16:35 |
| tmckay | I'm thinking of it like "select" a data source object as a Java job argument, and then Sahara takes care of the argument generation | 16:36 |
| alazarev | yes, that’s what I suggesting | 16:37 |
| tmckay | alazarev, and even for the other job types, we need a way to override the default Sahara argument generation | 16:37 |
| tmckay | for instance, the native bigpetstore pig job expects "$input" and "$output" in the pig script, but we generate "$INPUT" and "$OUTPUT". It breaks. | 16:38 |
| alazarev | overriding will be needed for comlpex workflows, but I don’t see use case for it now | 16:38 |
| tmckay | there is a workaround, though -- you have to pass extra arguments manually. Or change the pig script | 16:38 |
| tmckay | so, edp still definitely has some rough edges | 16:39 |
| alazarev | is any specification about $inpit/$INPUT? | 16:39 |
| tmckay | Not from the pig side. It's simply how the script is written, you can use any symbols you like. But, it is apparently case sensitive. We should mention the parameter generation in the Sahara docs, so users know what Sahara is going to put out. | 16:41 |
| tmckay | in bigpetstore, jayunit100 wrote the pig script without Sahara in mind and I discovered the problem | 16:41 |
| alazarev | in this case we should allow to specify input/output names, some kind of advanced options | 16:43 |
| tmckay | yes, exactly | 16:43 |
| *** bandarji has joined #openstack-sahara | 16:43 | |
| tmckay | crobertsrh, ^^ we had this discussion at Summit I think | 16:43 |
| crobertsrh | Ah, yeah, we did talk about that a bit :) | 16:44 |
| tmckay | hmm, maybe another blueprint | 16:44 |
| tmckay | If only I didn't have to sleep, there is so much I could do | 16:45 |
| tmckay | I think that's where the coffee industry started | 16:50 |
| *** akuznetsov has quit IRC | 17:11 | |
| openstackgerrit | A change was merged to openstack/sahara-image-elements: Use Spark 1.0.0 https://review.openstack.org/98140 | 17:12 |
| *** bradd1 has quit IRC | 17:24 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Migrated integration tests to testtools https://review.openstack.org/96981 | 17:30 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Migrated unit tests to testtools https://review.openstack.org/96979 | 17:30 |
| *** palar has joined #openstack-sahara | 17:48 | |
| *** akuznetsov has joined #openstack-sahara | 17:48 | |
| *** witlessb has quit IRC | 17:54 | |
| *** witlessb has joined #openstack-sahara | 17:56 | |
| *** witlessb has quit IRC | 17:57 | |
| *** alazarev has quit IRC | 18:18 | |
| *** julienvey has quit IRC | 18:21 | |
| *** alazarev has joined #openstack-sahara | 18:31 | |
| *** _elmiko is now known as elmiko | 18:39 | |
| *** akuznetsov has quit IRC | 18:45 | |
| *** witlessb has joined #openstack-sahara | 19:02 | |
| *** julienvey has joined #openstack-sahara | 19:05 | |
| *** julienve_ has joined #openstack-sahara | 19:06 | |
| *** julienvey has quit IRC | 19:09 | |
| *** julienve_ has quit IRC | 19:12 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Changing job excecution status to 'FAILED' in case of exception https://review.openstack.org/97637 | 19:19 |
| *** alazarev has quit IRC | 19:33 | |
| *** palar has quit IRC | 19:41 | |
| *** julienvey has joined #openstack-sahara | 19:58 | |
| *** juice has quit IRC | 20:01 | |
| *** juice has joined #openstack-sahara | 20:01 | |
| *** bradd1 has joined #openstack-sahara | 20:01 | |
| *** crobertsrh is now known as _crobertsrh | 20:18 | |
| *** witlessb has quit IRC | 20:21 | |
| *** bandarji has quit IRC | 20:42 | |
| *** alazarev has joined #openstack-sahara | 20:52 | |
| *** bradd1 has quit IRC | 21:10 | |
| *** palar has joined #openstack-sahara | 21:48 | |
| *** bandarji has joined #openstack-sahara | 21:52 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added secondary name node heap size param to vanilla plugin https://review.openstack.org/98540 | 21:59 |
| *** bradd1 has joined #openstack-sahara | 22:01 | |
| *** elmiko is now known as _elmiko | 22:04 | |
| *** julienvey has quit IRC | 22:09 | |
| *** bandarji has quit IRC | 22:29 | |
| *** palar has quit IRC | 22:52 | |
| *** palar has joined #openstack-sahara | 22:53 | |
| *** palar has quit IRC | 22:57 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added several checks on deleted cluster to prevent error logs https://review.openstack.org/98556 | 23:24 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added several checks on deleted cluster to prevent error logs https://review.openstack.org/98556 | 23:28 |
| *** bandarji has joined #openstack-sahara | 23:37 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara-dashboard: Added ability to import keypair from launch cluster dialog https://review.openstack.org/98557 | 23:44 |
| *** bandarji has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!