Wednesday, 2014-01-22

« Tuesday, 2014-01-21 Index Thursday, 2014-01-23 »
*** kbroughton has quit IRC00:14
*** Jayunit100 has joined #savanna00:40
*** IlyaE has joined #savanna00:53
*** nosnos has joined #savanna01:35
*** kbroughton has joined #savanna01:38
*** kbroughton has quit IRC01:42
*** tmckay has quit IRC01:45
*** IlyaE has quit IRC02:01
*** rnirmal has joined #savanna03:26
openstackgerritA change was merged to openstack/savanna: Read Swift credentials from input_data OR output_data  https://review.openstack.org/6750903:32
*** akuznetsov has joined #savanna04:25
*** rnirmal has quit IRC04:26
*** crobertsrh has joined #savanna04:50
*** DinaBelova_ is now known as DinaBelova04:58
openstackgerritA change was merged to openstack/savanna: Add code to configure cluster for external hdfs  https://review.openstack.org/4782805:33
*** nope_ has joined #savanna05:39
*** Ch00k has joined #savanna05:49
*** kbroughton has joined #savanna05:50
*** nope_ has quit IRC05:51
*** IlyaE has joined #savanna06:02
*** DinaBelova is now known as DinaBelova_06:04
*** IlyaE has quit IRC06:10
*** aignatov_ is now known as aignatov06:17
*** IlyaE has joined #savanna06:26
*** IlyaE has quit IRC06:32
*** NikitaKonovalov_ is now known as NikitaKonovalov06:41
*** nosnos_ has joined #savanna06:43
*** nosnos has quit IRC06:43
*** NikitaKonovalov is now known as NikitaKonovalov_06:50
*** xuhanp has joined #savanna06:56
*** mobiuslava has joined #savanna06:58
*** nadya__ has joined #savanna07:05
*** Ch00k has quit IRC07:05
*** Ch00k has joined #savanna07:06
*** nope_ has joined #savanna07:18
*** NikitaKonovalov_ is now known as NikitaKonovalov07:18
*** aignatov is now known as aignatov_07:20
*** aignatov_ is now known as aignatov07:54
*** kbroughton has quit IRC07:56
openstackgerritJenkins proposed a change to openstack/savanna: Imported Translations from Transifex  https://review.openstack.org/6833608:00
*** aignatov is now known as aignatov_08:06
*** DinaBelova_ is now known as DinaBelova08:15
*** Ch00k has quit IRC08:17
*** SergeyLukjanov_ is now known as SergeyLukjanov08:23
*** dmitryme has joined #savanna08:53
openstackgerritA change was merged to openstack/python-savannaclient: Add cluster-template-create to CLI  https://review.openstack.org/6821109:02
openstackgerritA change was merged to openstack/python-savannaclient: Add cluster-create to CLI  https://review.openstack.org/6821209:02
openstackgerritA change was merged to openstack/python-savannaclient: Fix default service-type name in help  https://review.openstack.org/6821709:03
*** Ch00k has joined #savanna09:07
*** aignatov_ is now known as aignatov09:10
*** Ch00k has quit IRC09:18
*** Ch00k has joined #savanna09:20
*** SergeyLukjanov is now known as SergeyLukjanov_a09:27
*** SergeyLukjanov_a is now known as SergeyLukjanov_09:28
*** aignatov is now known as aignatov_09:34
*** aignatov_ is now known as aignatov09:37
*** SergeyLukjanov_ is now known as SergeyLukjanov09:40
*** SergeyLukjanov is now known as SergeyLukjanov_09:47
*** NikitaKonovalov is now known as NikitaKonovalov_09:47
*** aignatov is now known as aignatov_09:49
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598109:49
*** xuhanp has quit IRC09:52
*** aignatov_ is now known as aignatov09:53
*** DinaBelova is now known as DinaBelova_09:58
*** akuznetsov has quit IRC10:12
*** akuznetsov has joined #savanna10:13
*** qwerty_nor has joined #savanna10:14
*** Ch00k has quit IRC10:22
*** Ch00k has joined #savanna10:23
*** xuhanp has joined #savanna10:25
*** xuhanp has quit IRC10:26
*** ylobankov has quit IRC10:38
*** ylobankov has joined #savanna10:52
*** aignatov is now known as aignatov_11:26
openstackgerritDmitry Mescheryakov proposed a change to openstack/savanna: Make error logging more safe  https://review.openstack.org/6413211:31
*** nadya__ has quit IRC11:33
*** aignatov_ is now known as aignatov11:50
*** kbroughton has joined #savanna12:00
*** IvanBerezovskiy has joined #savanna12:02
*** Ch00k has quit IRC12:11
*** Ch00k has joined #savanna12:13
*** aignatov is now known as aignatov_12:26
*** Ch00k has quit IRC12:35
*** Ch00k has joined #savanna12:36
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Update development install guide  https://review.openstack.org/6836412:38
*** Ch00k has quit IRC12:40
*** Ch00k has joined #savanna12:41
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: WIP Add integration tests to Intel plugin WIP  https://review.openstack.org/6681212:55
*** tmckay has joined #savanna13:00
*** Ch00k has quit IRC13:10
*** aignatov_ is now known as aignatov13:11
*** Ch00k has joined #savanna13:12
*** tmckay has quit IRC13:30
*** tmckay has joined #savanna13:45
*** tmckay has quit IRC13:47
*** kbroughton has quit IRC13:49
*** akuznetsov has quit IRC13:49
*** aignatov is now known as aignatov_13:57
*** nadya__ has joined #savanna14:00
*** kbroughton has joined #savanna14:03
*** akuznetsov has joined #savanna14:04
*** aignatov_ is now known as aignatov14:09
*** matsuhashi has joined #savanna14:14
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598114:30
*** matsuhashi has quit IRC14:45
*** tmckay has joined #savanna14:53
*** nosnos_ has quit IRC14:57
*** dmitryme has quit IRC15:00
tmckaycrobertsrh, ping15:09
crobertsrhyessir?15:09
tmckayfyi, I just moved https://bugs.launchpad.net/savanna/+bug/1269972 to "In Progress" and made you the assignee.  The api side is committed with "Partial-bug"15:09
tmckayActually, the importance can probably be bumped down.15:10
crobertsrhOk, thanks.  I hope to get around to that today, but it might not be until tomorrow.15:10
tmckayHigh => medium, I don't think it's a huge issues, especially with the api side fixed15:11
tmckaythanks15:11
*** rnirmal has joined #savanna15:11
crobertsrhI agree with the importance bumpdown15:11
tmckaycrobertsrh, same treatment for https://bugs.launchpad.net/savanna/+bug/126996815:13
crobertsrhThat one might be a bigger problem since the UI is still currently sending {} instead of []15:13
openstackgerritChad Roberts proposed a change to openstack/savanna-dashboard: Adding HDFS as a possible data source type  https://review.openstack.org/6756315:25
*** DinaBelova_ is now known as DinaBelova15:25
*** SergeyLukjanov_ is now known as SergeyLukjanov15:33
*** nadya__ has quit IRC15:47
*** akuznetsov has quit IRC15:47
*** aignatov is now known as aignatov_15:49
*** akuznetsov has joined #savanna15:55
*** sballe has joined #savanna16:02
*** IvanBerezovskiy has left #savanna16:05
*** SergeyLukjanov is now known as SergeyLukjanov_a16:07
*** nope_ has quit IRC16:07
*** SergeyLukjanov_a is now known as SergeyLukjanov_16:08
tmckaycrobertsrh, (for the public record), as we chatted about the api will accept {} for now and throw them away.  I did that so the change could be merged without a sync (https://bugs.launchpad.net/savanna/+bug/1270882)16:15
*** aignatov_ is now known as aignatov16:19
*** _ruhe is now known as ruhe16:20
*** nadya__ has joined #savanna16:26
*** SergeyLukjanov_ is now known as SergeyLukjanov16:28
*** nadya__ has quit IRC16:32
*** mattf has quit IRC16:33
*** nadya__ has joined #savanna16:35
*** tmckay has quit IRC16:37
*** aignatov is now known as aignatov_16:42
*** tmckay has joined #savanna16:45
*** mattf has joined #savanna16:45
*** mattf has quit IRC16:46
*** mattf has joined #savanna16:46
*** mattf has quit IRC16:54
*** tmckay has quit IRC16:54
*** nope_ has joined #savanna17:00
*** rnirmal has quit IRC17:04
*** kbroughton_ has joined #savanna17:05
*** kbroughton has quit IRC17:06
*** kbroughton_ is now known as kbroughton17:06
*** mattf has joined #savanna17:06
*** mattf has quit IRC17:06
*** mattf has joined #savanna17:06
*** tmckay has joined #savanna17:06
*** SergeyLukjanov is now known as SergeyLukjanov_17:10
*** mattf has quit IRC17:15
*** rnirmal has joined #savanna17:17
*** SergeyLukjanov_ is now known as SergeyLukjanov17:18
*** tmckay has quit IRC17:19
*** mattf has joined #savanna17:25
*** SergeyLukjanov is now known as SergeyLukjanov_17:28
*** mattf is now known as _mattf17:33
*** Ch00k has quit IRC17:35
*** Ch00k has joined #savanna17:37
*** nope_ has quit IRC17:38
*** _mattf is now known as mattf17:40
*** mattf has quit IRC17:41
*** mattf has joined #savanna17:41
*** Ch00k has quit IRC17:41
*** jdev789 has joined #savanna17:54
*** jdev789 has quit IRC18:00
*** jdev789 has joined #savanna18:16
*** Ch00k has joined #savanna18:17
*** Ch00k has quit IRC18:17
*** Ch00k has joined #savanna18:18
*** SergeyLukjanov_ is now known as SergeyLukjanov18:21
*** Ch00k has quit IRC18:22
*** Ch00k has joined #savanna18:22
* mattf writes "del source._info['credentials']" and thinks we're doing something wrong18:23
*** nadya__ has quit IRC18:26
*** tmckay has joined #savanna18:27
*** nadya__ has joined #savanna18:34
*** qwerty_nor has quit IRC18:35
*** nadya__ has quit IRC18:35
*** ruhe is now known as _ruhe18:39
*** qwerty_nor has joined #savanna18:41
*** DinaBelova is now known as DinaBelova_18:57
*** _ruhe is now known as ruhe19:01
*** kbroughton has left #savanna19:13
*** nadya__ has joined #savanna19:16
*** nadya__ has quit IRC19:30
openstackgerritJonathan Maron proposed a change to openstack/savanna: Update Ambari Repo location and services refactoring  https://review.openstack.org/6757719:31
openstackgerritChad Roberts proposed a change to openstack/savanna-dashboard: Changing the args type for Pig jobs to a list  https://review.openstack.org/6846719:34
*** Ch00k_ has joined #savanna19:40
*** Ch00k has quit IRC19:41
mattfcrobertsrh, btw, "Job executions" doesn't have a "Create new job execution blah blah" button like Clusters has, on your radar?19:57
crobertsrhmattf:  Yeah, it's been thought of, just not done yet.19:57
*** NikitaKonovalov_ is now known as NikitaKonovalov19:57
mattfthanks19:58
Jayunit100Does savanna aim to package oozie templates as well?20:01
openstackgerritJonathan Maron proposed a change to openstack/savanna: Update Ambari Repo location and services refactoring  https://review.openstack.org/6757720:01
*** nope_ has joined #savanna20:04
*** DinaBelova_ is now known as DinaBelova20:04
*** nope_ has quit IRC20:06
crobertsrhI'm having an issue with the latest from master.  I pulled and went to db-sync, but I realized that's no longer an option, so I did some reading and it seems like "upgrade head" was a good option.20:08
crobertsrhBut that appeared to fail.  Running the api with my current database state just erroring out.  Any thoughts on how I can hard-reset my database?20:09
ruhecrobertsrh: remove old DB file (if you're using sqlite) and run savanna-db-manage upgrade head20:11
crobertsrhwhere would I find the old db?20:12
SergeyLukjanovNOTE: https://git.openstack.org/cgit/openstack/savanna/log/?h=milestone-proposed20:13
ruhecrobertsrh: default connection string is #connection=sqlite:////savanna/openstack/common/db/$sqlite_db20:13
crobertsrhAh yes, just found it :)20:13
crobertsrhthanks20:13
crobertsrhGreat, working now.20:14
ruheSergey (not SergeyLukjanov, but other Sergey), who added alembic migrations will update Savanna docs to reflect DB migration changes20:15
ruheJayunit100: doesn't it already have templates for oozie? at least Vanilla and HDP plugins can deploy Oozie20:18
openstackgerritJonathan Maron proposed a change to openstack/savanna: Update Ambari Repo location and services refactoring  https://review.openstack.org/6757720:19
Jayunit100i mean wrapping up oozie scheduling, something like what hue provides i guess20:20
Jayunit100so i could tell savanna "run this job 5 times a day" or "run this job once the size of the file is > 200GB"20:20
ruheJayunit100: ah, that's a nice feature. so, i guess savanna aims for that20:21
ruhebut20:22
ruhein OpenStack you gotta play nice with other projects20:22
ruhethere are projects for scheduling20:22
ruhe* there is20:22
ruheonce it ready, savanna will use it to schedule things20:22
ruhescheduling things in cloud isn't an easy thing20:25
*** Ch00k_ has quit IRC20:27
*** SergeyLukjanov is now known as SergeyLukjanov_20:28
*** akuznetsov has quit IRC20:31
*** tmckay has quit IRC20:32
Jayunit100ruhe: oh really theres another openstack scheduler?20:37
Jayunit100it schedules at what level of the stack?20:37
*** DinaBelova is now known as DinaBelova_20:37
Jayunit100ooze DOES the scheduling already.20:37
ruheJayunit100: oozie will work only on existing cluster and only with a single cluster20:39
*** qwerty_nor has quit IRC20:39
ruheand doesn't allow to utilise computing resources efficiently20:39
*** tmckay has joined #savanna20:39
ruhewhat we can do - is schedule MapReduce job to run each N hours or "run this job once the size of the file is > 200GB"20:40
ruheand spin up a cluster only when you need to run the job20:40
ruhekill the cluster once the job is finished20:41
*** NikitaKonovalov is now known as NikitaKonovalov_20:42
*** qwerty_nor has joined #savanna20:43
ruheand when nova scheduler gets smart enough, we'll be able to launch cluster on the least loaded compute nodes and closest to the data stored somewhere in the cloud object storage (swift, gluster, ceph)20:43
*** NikitaKonovalov_ is now known as NikitaKonovalov20:47
Jayunit100ah okay20:49
Jayunit100so,your saying "you shouldn't use oozie daemons on an ephemeral cluster"20:49
*** SergeyLukjanov_ is now known as SergeyLukjanov20:51
*** NikitaKonovalov is now known as NikitaKonovalov_20:57
*** SergeyLukjanov is now known as SergeyLukjanov_21:02
*** tmckay has quit IRC21:19
*** ruhe is now known as _ruhe21:23
mattfJayunit100, oozie is the job manager w/i in the cluster, should always be there if the cluster itself is ephemeral or not21:34
mattfthe OS scheduler helps w/ provisioning the cluster, after that it's ignorant of what happens w/i the cluster21:34
Jayunit100"ooze = job mgr w/i cluster" <-- that implies that if cluster is gone so is ooze.21:43
mattfDUCK!21:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-list to CLI  https://review.openstack.org/6849121:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-show to CLI  https://review.openstack.org/6849221:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-delete to CLI  https://review.openstack.org/6849321:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-data-list to CLI  https://review.openstack.org/6849421:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-data-delete to CLI  https://review.openstack.org/6849521:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-list to CLI  https://review.openstack.org/6849621:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-show to CLI  https://review.openstack.org/6849721:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-delete to CLI  https://review.openstack.org/6849821:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-template-list to CLI  https://review.openstack.org/6849921:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-template-show to CLI  https://review.openstack.org/6850021:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-template-delete to CLI  https://review.openstack.org/6850121:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-list to CLI  https://review.openstack.org/6850221:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-show to CLI  https://review.openstack.org/6850321:48
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-delete to CLI  https://review.openstack.org/6850421:48
mattfJayunit100, yup, there's an instance of oozie in every cluster21:49
mattfsave to unduck21:50
mattfsafe too21:50
* Jayunit100 unducking21:51
openstackgerritChad Roberts proposed a change to openstack/savanna-dashboard: Changing the args type for Pig jobs to a list  https://review.openstack.org/6846721:51
Jayunit100+1 for more cmd options21:51
mattfonly thing missing is -create options for edp features21:54
mattfsome of them are rather wonky21:54
mattfpost to /jobs to make a new /job_executions/{id}21:54
mattfpost to /jobs/{id}/execute to make a new /job_executions/{id}21:54
Jayunit100how can ooze live IN a cluster, yet persist when cluster DIES  ?21:55
mattfit doesn't, it dies21:56
Jayunit100"hould always be there if the cluster itself is ephemeral or not"21:57
mattfephemeral means it lives for the duration of a job21:57
mattfso when it comes up so does an instance of oozie within it21:57
Jayunit100gotcha21:57
Jayunit100so "s/always/at_least"21:58
mattfsure. edp won't function w/o an oozie endpoint in the cluster to talk to22:01
openstackgerritChad Roberts proposed a change to openstack/savanna-dashboard: Changing the args type for Pig jobs to a list  https://review.openstack.org/6846722:07
*** crobertsrh is now known as _crobertsrh22:13
*** mattf is now known as _mattf22:31
*** nadya__ has joined #savanna22:31
*** nadya__ has quit IRC22:35
*** _mattf is now known as mattf22:37
*** qwerty_nor has quit IRC23:01
*** sballe has quit IRC23:22
*** sballe has joined #savanna23:22
*** sballe_ has joined #savanna23:34
*** sballe has quit IRC23:35
« Tuesday, 2014-01-21 Index Thursday, 2014-01-23 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!