Wednesday, 2014-07-09

*** IlyaE has quit IRC00:01
openstackgerritTrevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines  https://review.openstack.org/10062200:06
*** IlyaE has joined #openstack-sahara00:17
*** _RobLevas has joined #openstack-sahara00:23
*** _RobLevas has quit IRC00:32
*** Networkn3rd has joined #openstack-sahara01:17
*** Networkn3rd has quit IRC01:22
*** _RobLevas has joined #openstack-sahara01:30
*** _RobLevas has quit IRC01:34
*** zhiyan_ is now known as zhiyan01:47
*** IlyaE has quit IRC01:50
*** zhiyan is now known as zhiyan_01:56
*** zhiyan_ is now known as zhiyan01:56
*** bradd1 has quit IRC01:58
*** bradd1 has joined #openstack-sahara02:30
*** IlyaE has joined #openstack-sahara02:37
*** IlyaE has quit IRC02:51
*** IlyaE has joined #openstack-sahara03:06
*** ViswaV has joined #openstack-sahara03:07
*** ViswaV__ has joined #openstack-sahara03:07
*** ViswaV_ has quit IRC03:09
*** ViswaV has quit IRC03:11
*** IlyaE has quit IRC03:12
*** _RobLevas has joined #openstack-sahara03:32
*** _RobLevas has quit IRC03:36
*** IlyaE has joined #openstack-sahara03:40
*** ViswaV__ has quit IRC04:13
*** ViswaV has joined #openstack-sahara04:14
*** IlyaE has quit IRC04:21
*** _RobLevas has joined #openstack-sahara04:34
*** _RobLevas has quit IRC04:39
*** akuznetsov has quit IRC04:46
*** akuznetsov has joined #openstack-sahara04:53
*** bradd1 has quit IRC05:01
*** IlyaE has joined #openstack-sahara05:03
*** akuznetsov has quit IRC05:12
*** IlyaE has quit IRC05:25
*** akuznetsov has joined #openstack-sahara05:26
*** IlyaE has joined #openstack-sahara05:59
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex  https://review.openstack.org/10535806:01
*** xianghui has joined #openstack-sahara06:07
*** Longgeek has joined #openstack-sahara06:36
*** IlyaE has quit IRC06:56
*** xianghui has quit IRC07:04
*** IlyaE has joined #openstack-sahara07:07
*** skolekonov has joined #openstack-sahara07:08
*** IlyaE has quit IRC07:15
openstackgerritA change was merged to stackforge/sahara-ci-config: Resolve problem with java downloading for spark images  https://review.openstack.org/10538807:50
*** jpich has joined #openstack-sahara07:58
*** xianghui has joined #openstack-sahara08:06
*** julienvey has joined #openstack-sahara08:15
openstackgerritA change was merged to openstack/python-saharaclient: Remove docutils pin  https://review.openstack.org/10562708:18
*** yiconglu has joined #openstack-sahara08:26
*** witlessb has joined #openstack-sahara08:53
*** tosky has joined #openstack-sahara08:59
openstackgerritA change was merged to openstack/sahara: Imported Translations from Transifex  https://review.openstack.org/10535809:03
*** Timotey has joined #openstack-sahara09:14
*** yiconglu has quit IRC09:19
*** _RobLevas has joined #openstack-sahara09:19
*** Longgeek has quit IRC09:27
openstackgerritA change was merged to openstack/sahara: Update docs to reflect the changes in security group section in horizon  https://review.openstack.org/10468709:38
openstackgerritA change was merged to openstack/sahara: Updated from global requirements  https://review.openstack.org/10517409:42
*** vrovachev has joined #openstack-sahara09:46
*** Longgeek has joined #openstack-sahara09:58
*** Longgeek has quit IRC10:05
*** xianghui has quit IRC10:49
*** xianghui has joined #openstack-sahara11:03
*** _RobLevas has quit IRC11:32
*** Longgeek has joined #openstack-sahara11:38
*** _RobLevas has joined #openstack-sahara11:42
openstackgerritMatthew Farrellee proposed a change to openstack/sahara: Update image registry docs to use cli  https://review.openstack.org/10571512:02
*** _crobertsrh is now known as crobertsrh12:12
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Add vanilla plugin with Hadoop 2.4.0  https://review.openstack.org/10509912:16
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Add vanilla plugin with Hadoop 2.4.0  https://review.openstack.org/10509912:20
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Add oslo.i18n lib to requirements  https://review.openstack.org/10572312:31
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Use oslo.i18n  https://review.openstack.org/10572412:31
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Remove oslo-incubator's gettextutils  https://review.openstack.org/10572512:31
*** akuznetsov has quit IRC12:34
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Use oslo.i18n  https://review.openstack.org/10572412:36
*** sballe has joined #openstack-sahara13:01
tellesnobregahey, storm uses only one config file and it is a YAML file. I need to parse it so it can be loaded, is it ok to use pyyaml lib to parse it? this means including in the dependencies for sahara13:04
tellesnobregaactually i'm a little confused how this is going to work. This files contains storm running configurations, like master ip, zookeeper ip and so on. This can be written without the user, by using the information of node groups we can figure out this ips and write the configuration file. Other configurations could be managed by the user. For a first prototype, should i have it all or just the basics to have it running, and leave t13:09
elmikotellesnobrega: do you have a spec or blueprint talking about the storm change?13:20
tellesnobreganot yet13:20
elmikoi would imagine you could probably get pyyaml brought in, but first we'll probably have to discuss the spec and understand the impact13:21
tellesnobregasure13:21
tellesnobregai will work on this spec now13:21
elmikoas for the config stuff, it does sound like you will need to create the yaml config dynamically13:22
tellesnobregait needs to be because of the zookeeper and master node information13:22
elmikoyou might follow the design pattern used for the workflow.xml files and create a yaml template that can be filled in at runtime. then you wouldn't necessarily need pyyaml.13:22
elmikojust throwin ideas out, i don't know much about storm13:23
tellesnobregaelmiko: sounds ok. it may be easier to use pyyaml this way so i could have a dictionary and it translates to yaml (i'm not sure that it does, but i think so)13:24
*** Longgeek has quit IRC13:25
tellesnobregathis file needs to be sent to every node in the topology. I'm not sure what configurations sahara needs13:25
tellesnobregapyyaml writes the file alright13:26
tmckaySergeyLukjanov, aignatov, hi, I've got gate-sahara-pep8 failures on jenkins for https://review.openstack.org/#/c/100622/, but in my local branch tox -e pep8 passes (even after wiping out tox and rebuilding).  Is jenkins using a new pep8 definition that we don't have in the sahara sources?  It would be much simpler if they matched.13:26
elmikotellesnobrega: ack13:26
*** Longgeek has joined #openstack-sahara13:35
*** venza_ has joined #openstack-sahara13:43
tellesnobregaelmiko: created the blueprint, im gonna start writing the spec13:46
elmikosounds good, then we can at least get some ideas on the table about the changes needed13:46
*** venza has quit IRC13:48
*** anteaya has quit IRC13:48
*** anteaya has joined #openstack-sahara13:55
tmckaySergeyLukjanov, elmiko, croberts, okay, apparently this is related to changes from aignatov in https://review.openstack.org/#/c/100090/2.  the sahara/tox.ini has "ignores" for some errors, but the gate tests must not have those ignores -- looks like maybe gate tests start with icehouse as a base?  Not too familiar with those.  However, if I edit tox.ini and do not ignore H305 and H307 I can get the same errors for https://review.14:00
elmikotmckay: i thought those were being pulled out with the pep8 fix change?14:01
aignatovtmckay: probably you’ll need to rebase on the latest commit in sahara master14:02
tmckaywell, apparently, the gate tests are lagging.  So if you want a clean bill of health from jenkins, you might have to unignore something14:02
aignatovYou’ll see all these errors14:02
tmckayaignatov, I just rebased yesterday :)  I'll try again14:02
dmitrymetmckay: I think gate rebases your change on master before testing it14:02
SergeyLukjanovtmckay, ack - gate always rebase your changes to master before running jobs14:03
tmckayokay, I'll get an up to date master and try again14:03
SergeyLukjanovtellesnobrega, ack - let's discuss on spec14:03
SergeyLukjanovbut IMO pyyaml will be ok14:04
openstackgerritTrevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines  https://review.openstack.org/10062214:04
tellesnobregaSergeyLukjanov: sure, i'm writing it now.14:04
aignatovtmckay: Also https://review.openstack.org/#/c/101845/ could affects you :)14:04
SergeyLukjanovtellesnobrega, cool, thx!14:04
dmitrymetmckay: I agree with aignatov: most probably the CR he pointed out affected you14:05
tmckayaignatov, dmitryme, SergeyLukjanov, thanks!  I guessed I missed the rebase by a few hours, lol.14:06
tmckayI have the same errors now14:07
tmckayI was confused :)14:07
* tmckay writes down "When in doubt, always rebase first"14:07
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: Fixed configuring instances for Vanilla 2.0  https://review.openstack.org/10552514:07
*** ViswaV_ has joined #openstack-sahara14:13
*** ViswaV__ has joined #openstack-sahara14:14
*** ViswaV has quit IRC14:16
*** ViswaV_ has quit IRC14:18
openstackgerritTrevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines  https://review.openstack.org/10062214:18
*** Networkn3rd has joined #openstack-sahara14:27
openstackgerritA change was merged to openstack/python-saharaclient: Jar files for pig and mapreduce EDP tests  https://review.openstack.org/10560314:42
tellesnobregaSergeyLukjanov: i'm a little lost here what is the impact in the api. Haven't thought about it yet. Do you know of api impact when adding new plugins?14:54
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: Fix scaling cluster Vanilla for Hadoop 2.3  https://review.openstack.org/10575614:54
*** _RobLevas is now known as RobLevas14:57
*** IlyaE has joined #openstack-sahara15:02
SergeyLukjanovtellesnobrega, no impact15:04
tellesnobregagreat :)15:04
tellesnobregathats what i thougt, but wasnt sure15:04
SergeyLukjanovtellesnobrega, by API impact we mean new endpoints addition or updating existing ones15:04
*** Networkn3rd has quit IRC15:09
*** zhiyan is now known as zhiyan_15:10
*** Hao has joined #openstack-sahara15:11
*** sballe has quit IRC15:21
*** Longgeek has quit IRC15:36
*** Longgeek has joined #openstack-sahara15:37
*** Longgeek has quit IRC15:40
*** Longgeek has joined #openstack-sahara15:40
*** RobLevas has quit IRC16:10
openstackgerritSofiia Kostiuchenko proposed a change to openstack/sahara: Fixed EDP job execution failure  https://review.openstack.org/10577816:11
*** vrovachev has quit IRC16:11
*** ViswaV has joined #openstack-sahara16:12
*** ViswaV__ has quit IRC16:13
*** Hao has quit IRC16:14
tmckaySergeyLukjanov, aignatov, maybe dmitryme :), I am wondering about using paramiko Channel objects to run ssh commands on the remote host asynchronously.  The trouble is, that depending on how a park job is launched with spark-submit from the command line, the driver program will run synchronously in the shell, or the driver program will be deployed into the cluster.  Driver deployment does not work for all cases (python spark prog16:15
tmckayI am thinking about launching spark synchronously in ssh but asynchronously to Sahara run_job, because of course it could take a really, really long time16:16
tmckayAny problem with paramiko Channels that you know of?  Everything else we do is synchronous16:16
tmckayMy other thought is to use synchronous ssh inside a python subprocess in Sahara, and then track them for job status16:17
tmckaylaunch a subprocess per spark job run16:17
* tmckay has to look more closely at subprocesses/threads in Sahara16:18
tmckaycrobertsrh, elmiko, too, ^^ in case you have experience here16:18
* tmckay needs some food16:19
openstackgerritTelles Mota Vidal Nóbrega proposed a change to openstack/sahara-specs: Storm Integration  https://review.openstack.org/10578416:20
dmitrymetmckay: keeping ssh connection for a long time does not sound good16:22
dmitrymecan we use ssh to run command in the background16:22
dmitrymewe’ll have to run another ssh commands later to track job status16:24
*** jpich has quit IRC16:25
*** miqui has quit IRC16:27
*** miqui has joined #openstack-sahara16:28
crobertsrhsorry, just got back from lunch....looking back16:32
*** shakamunyi has joined #openstack-sahara16:33
*** shakamunyi has quit IRC16:33
*** RobLevas has joined #openstack-sahara16:33
tmckaydmitryme, yes, ssh background is one way, I have seen examples using "screen" and "nohup" but from the comment it seems difficult to always get it right :)  That's why I wondered about Paramiko channels.16:33
tmckayThey let you do things asynchronously already.16:33
*** shakamunyi has joined #openstack-sahara16:34
*** bandarji has joined #openstack-sahara16:34
dmitrymetmckay: they do. Our calls are synchronous because we explicitly block - https://github.com/openstack/sahara/blob/master/sahara/utils/ssh_remote.py#L12916:36
dmitrymebut again, I don’t like idea of long waiting ssh16:37
tmckayokay, I see.  So, it would be a possibility with some new flag to execute_command, but I'll see if I can get ssh to return and leave the command running16:37
tmckaydmitryme, thanks16:38
dmitrymeon the other side, I think people are dramatizing when they are speaking about running in background16:38
dmitrymeat least I did it once with nohup and it worked16:38
openstackgerritTelles Mota Vidal Nóbrega proposed a change to openstack/sahara-specs: Storm Integration  https://review.openstack.org/10578416:39
tmckayokay thanks :) many people have asked this question.  That is encouraging.16:39
*** Networkn3rd has joined #openstack-sahara16:39
*** Hao has joined #openstack-sahara16:42
*** miqui_ has joined #openstack-sahara16:54
*** miqui has quit IRC16:55
elmiko+1 about not wanting long running ssh17:05
*** Networkn3rd has quit IRC17:05
*** julienvey has quit IRC17:15
*** IlyaE has quit IRC17:39
*** julienvey has joined #openstack-sahara17:46
*** julienvey has quit IRC17:48
*** julienve_ has joined #openstack-sahara17:48
*** tosky has quit IRC17:49
*** tosky__ has joined #openstack-sahara17:49
*** ViswaV has quit IRC17:52
*** julienve_ has quit IRC17:52
*** ViswaV has joined #openstack-sahara17:52
crobertsrhI'm trying to organize a list of dashboard changes that need to be done for juno (with some thought as to whether or not there will be time).  Please feel free to add any thoughts you might have.   https://etherpad.openstack.org/p/sahara-juno-post-merge-changes   I'm hoping to get started on these very soon (ideally the merge will be done very soon--even if it isn't, I'll be adding these changes as deps on top of it).17:53
*** tosky__ is now known as tosky17:54
*** IlyaE has joined #openstack-sahara17:57
elmikocrobertsrh: well, you got my concern :)17:59
*** Hao has quit IRC17:59
crobertsrhI'm assuming that I have missed many things with head down in merge land.17:59
*** Longgeek has quit IRC18:01
elmikoit's nice that you're finally starting to see daylight there18:01
crobertsrhFor sufficiently low values of daylight18:06
elmikolol18:06
*** xianghui has quit IRC18:11
tmckay ssh -i ~/.ssh/id_rsa.pub ubuntu@10.3.8.240 '/opt/spark/bin/spark-submit /opt/spark/lib/spark-examples-1.0.0-hadoop2.0.0-mr1-cdh4.2.0.jar --class org.apache.spark.examples.SparkPi --master spark://spark-test-one-spark-master-1-001:7077 > /tmp/app 2>&1 & echo $!'18:27
tmckay250318:27
tmckaydmitryme, ^^ :-D18:27
tmckaythat last number is the pid18:27
tmckayand the spark log trace in this case is in /tmp/app.  We could maybe save that under /mnt/spark/appid.log or something18:28
tmckayand it returns immediately, of course18:28
tmckaySo, we can know when the pid is done, and we can save the log18:29
tmckaythe trick to the return is apparently just to redirect or close stderr and stdout18:29
*** IlyaE has quit IRC18:30
*** IlyaE has joined #openstack-sahara18:31
*** julienvey has joined #openstack-sahara18:49
openstackgerritTrevor McKay proposed a change to openstack/sahara: Allow plugins to choose the EDP implementation  https://review.openstack.org/10101618:51
*** julienvey has quit IRC18:53
dmitrymetmckay: yep, looks good to me18:54
*** Hao has joined #openstack-sahara19:02
*** Hao has quit IRC19:11
*** IlyaE has quit IRC19:12
*** ViswaV has quit IRC19:20
*** shakamunyi has quit IRC19:20
*** ViswaV has joined #openstack-sahara19:23
*** Hao has joined #openstack-sahara19:23
*** Hao has quit IRC19:30
*** IlyaE has joined #openstack-sahara19:34
*** miqui_ is now known as miqui19:37
*** shakamunyi has joined #openstack-sahara19:49
*** shakamunyi has quit IRC19:49
*** julienvey has joined #openstack-sahara19:49
*** julienvey has quit IRC19:54
*** IlyaE has quit IRC20:03
*** bandarji has quit IRC20:04
*** bandarji has joined #openstack-sahara20:04
*** bandarji has quit IRC20:05
*** IlyaE has joined #openstack-sahara20:07
*** shakamunyi has joined #openstack-sahara20:15
*** shakamunyi has quit IRC20:15
*** bandarji has joined #openstack-sahara20:16
*** IlyaE has quit IRC20:20
*** shakamunyi has joined #openstack-sahara20:24
*** Hao has joined #openstack-sahara20:27
*** bandarji has quit IRC20:28
*** IlyaE has joined #openstack-sahara20:30
*** Hao has quit IRC20:32
*** julienvey has joined #openstack-sahara20:50
*** julienvey has quit IRC20:55
*** IlyaE has quit IRC20:58
*** bandarji has joined #openstack-sahara20:59
*** IlyaE has joined #openstack-sahara21:03
*** shakamunyi has quit IRC21:08
*** shakamunyi has joined #openstack-sahara21:09
*** shakamunyi has quit IRC21:09
*** shakamunyi has joined #openstack-sahara21:10
*** IlyaE has quit IRC21:20
*** Hao has joined #openstack-sahara21:29
*** crobertsrh is now known as _crobertsrh21:29
*** Hao has quit IRC21:34
*** sballe has joined #openstack-sahara21:42
*** julienvey has joined #openstack-sahara21:51
*** IlyaE has joined #openstack-sahara21:52
*** shakamunyi has quit IRC21:53
*** shakamunyi has joined #openstack-sahara21:54
*** shakamunyi has quit IRC21:54
*** julienvey has quit IRC21:56
*** ViswaV has quit IRC21:59
*** IlyaE has quit IRC22:05
*** IlyaE has joined #openstack-sahara22:12
*** miqui_ has joined #openstack-sahara22:13
*** miqui has quit IRC22:13
*** IlyaE has quit IRC22:23
*** bandarji has quit IRC22:23
*** RobLevas has quit IRC22:25
*** Hao has joined #openstack-sahara22:30
*** RobLevas has joined #openstack-sahara22:31
*** RobLevas is now known as _RobLevas22:31
*** Hao has quit IRC22:36
*** IlyaE has joined #openstack-sahara22:37
*** boris-42 has quit IRC22:37
*** boris-42 has joined #openstack-sahara22:39
*** ViswaV has joined #openstack-sahara22:43
*** elmiko is now known as _elmiko22:45
*** julienvey has joined #openstack-sahara22:52
*** julienvey has quit IRC22:56
*** ViswaV has quit IRC23:04
*** witlessb has quit IRC23:25
*** _RobLevas has quit IRC23:32
*** Hao has joined #openstack-sahara23:32
*** Hao has quit IRC23:37
*** julienvey has joined #openstack-sahara23:52
*** IlyaE has quit IRC23:53
*** julienvey has quit IRC23:57
sballeSergeyLukjanov, hi. I am trying to get sahara working using the vanilla plug-in. Have you seen an error like this before: export HEAT_INCLUDE_PASSWORD=1?23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!