| *** witlessb has quit IRC | 00:38 | |
| *** hdd has quit IRC | 01:11 | |
| *** hdd has joined #openstack-sahara | 01:15 | |
| *** zigo_ has joined #openstack-sahara | 01:21 | |
| *** zigo has quit IRC | 01:21 | |
| *** macjack has joined #openstack-sahara | 01:23 | |
| *** zigo has joined #openstack-sahara | 01:27 | |
| *** zigo_ has quit IRC | 01:27 | |
| *** zigo has quit IRC | 01:46 | |
| *** zigo has joined #openstack-sahara | 01:47 | |
| *** zigo has quit IRC | 02:00 | |
| *** zigo has joined #openstack-sahara | 02:00 | |
| *** tellesnobrega has quit IRC | 02:06 | |
| *** Longgeek has joined #openstack-sahara | 02:55 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Separate the codes of CDH5 and CDH5.3.0 https://review.openstack.org/147933 | 03:56 |
|---|---|---|
| *** coolsvap has joined #openstack-sahara | 04:12 | |
| *** Networkn3rd has joined #openstack-sahara | 04:30 | |
| *** hdd has quit IRC | 04:45 | |
| *** Networkn3rd has quit IRC | 05:05 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Separate the codes of CDH5 and CDH5.3.0 https://review.openstack.org/147933 | 05:08 |
| *** Networkn3rd has joined #openstack-sahara | 05:09 | |
| *** hdd has joined #openstack-sahara | 05:26 | |
| *** Poornima has joined #openstack-sahara | 05:34 | |
| *** Networkn3rd has quit IRC | 05:35 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Imported Translations from Transifex https://review.openstack.org/148803 | 06:15 |
| *** tellesnobrega_ has quit IRC | 06:22 | |
| *** teju has joined #openstack-sahara | 06:47 | |
| *** sgotliv has quit IRC | 07:11 | |
| *** hdd has quit IRC | 07:35 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 07:41 |
| *** chlong has quit IRC | 07:46 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Fixing log messages to avoid information duplication https://review.openstack.org/147504 | 07:56 |
| *** witlessb has joined #openstack-sahara | 08:00 | |
| *** sgotliv has joined #openstack-sahara | 08:19 | |
| *** sgotliv_ has joined #openstack-sahara | 08:26 | |
| *** sgotliv has quit IRC | 08:26 | |
| openstackgerrit | lu huichun proposed openstack/sahara: Add HBase service test in cdh plugin integration test https://review.openstack.org/148847 | 08:57 |
| *** sgotliv_ has quit IRC | 09:10 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-specs: New integration tests for Sahara https://review.openstack.org/147219 | 09:21 |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-specs: New integration tests for Sahara https://review.openstack.org/147219 | 09:22 |
| *** sgotliv_ has joined #openstack-sahara | 09:26 | |
| openstackgerrit | Merged openstack/sahara-image-elements: Add the swift_hadoop element to Spark images https://review.openstack.org/147985 | 09:55 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Migrate to oslo.log https://review.openstack.org/148861 | 10:00 |
| *** Poornima has quit IRC | 10:00 | |
| *** Poornima has joined #openstack-sahara | 10:01 | |
| *** Poornima has quit IRC | 10:09 | |
| openstackgerrit | Merged openstack/python-saharaclient: Remove cyclic dependency https://review.openstack.org/147771 | 10:11 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Migrate to oslo.log https://review.openstack.org/148861 | 10:13 |
| *** tellesnobrega_ has joined #openstack-sahara | 10:15 | |
| openstackgerrit | Merged openstack/python-saharaclient: Use requests-mock for mocking https://review.openstack.org/147772 | 10:20 |
| *** teju has left #openstack-sahara | 10:22 | |
| *** Poornima has joined #openstack-sahara | 10:23 | |
| *** macjack has quit IRC | 10:25 | |
| *** Poornima has quit IRC | 10:45 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Drop uuidutils https://review.openstack.org/148876 | 10:52 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Drop uuidutils https://review.openstack.org/148876 | 10:53 |
| *** tosky has joined #openstack-sahara | 11:10 | |
| *** tellesnobrega_ has quit IRC | 11:13 | |
| *** Poornima has joined #openstack-sahara | 11:20 | |
| *** Poornima is now known as Poornima_K | 11:38 | |
| *** Poornima_K is now known as Poornima | 11:38 | |
| *** tellesnobrega has joined #openstack-sahara | 11:47 | |
| *** skolekonov has joined #openstack-sahara | 11:54 | |
| *** Poornima has quit IRC | 11:58 | |
| *** chlong has joined #openstack-sahara | 12:02 | |
| *** Longgeek has quit IRC | 12:58 | |
| *** Longgeek has joined #openstack-sahara | 13:08 | |
| *** balle has joined #openstack-sahara | 13:19 | |
| balle | hi all | 13:20 |
| balle | just tried to execute the example spark job and found the following failure in my log file | 13:20 |
| balle | Can't run job execution No such file or directory: /usr/lib/python2.7/site-packages/sahara/service/edp/resources/launch_command.py | 13:20 |
| balle | any hint what i may doing wrong? the script is not part of the rpm package in use (rhel7) | 13:21 |
| balle | hm i guess it's a bug in the rpm? so nevermind | 13:37 |
| *** elmiko has quit IRC | 13:41 | |
| sreshetnyak | hi balle | 13:45 |
| sreshetnyak | looks like a bug in Sahara | 13:46 |
| *** elmiko has joined #openstack-sahara | 13:47 | |
| openstackgerrit | Sebastien Badia proposed stackforge/puppet-sahara: Migrate postgresql backend to use openstacklib::db::postgresql https://review.openstack.org/145511 | 13:48 |
| *** mikehn has quit IRC | 13:54 | |
| *** tellesnobrega has quit IRC | 13:54 | |
| *** mikehn has joined #openstack-sahara | 13:57 | |
| elmiko | balle: looks like it might be a bug with the installer | 13:57 |
| *** crobertsrh has joined #openstack-sahara | 13:57 | |
| *** tellesnobrega has joined #openstack-sahara | 14:06 | |
| balle | is spark only cluster with data from hdfs stable in current version of sahara? | 14:07 |
| *** Longgeek has quit IRC | 14:08 | |
| elmiko | balle: not sure i understand. i think all the plugins can use hdfs. | 14:09 |
| *** Longgeek has joined #openstack-sahara | 14:09 | |
| openstackgerrit | Ruslan Kamaldinov proposed openstack/sahara: Removed sad line https://review.openstack.org/148932 | 14:12 |
| *** tmckay has joined #openstack-sahara | 14:14 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: New integration tests - part 2 https://review.openstack.org/148935 | 14:16 |
| openstackgerrit | Sebastien Badia proposed stackforge/puppet-sahara: fixtures: Unpin puppetlabs-mysql module https://review.openstack.org/145510 | 14:21 |
| *** coolsvap is now known as coolsvap|afk | 14:25 | |
| *** egafford has joined #openstack-sahara | 14:35 | |
| *** coolsvap|afk is now known as coolsvap | 14:36 | |
| balle | elmiko: i am currently trying to get spark connect with hdfs. both separate clusters in savana but it fails due to port 8020 being not open on the hdfs cluster | 14:45 |
| balle | hdfs with hadoop 2.4.1 | 14:45 |
| elmiko | balle: could it be an issue with the security groups in openstack? | 14:45 |
| balle | no the port is not open on the namenode | 14:46 |
| balle | i am using the ubuntu image btw | 14:46 |
| elmiko | hmm, that's odd | 14:46 |
| elmiko | yea, i think ubuntu is the only image that supports spark currently | 14:46 |
| elmiko | balle: i think you might want to open a bug, sounds like the image distributed is not configured properly | 14:47 |
| elmiko | tmckay: ^^ spark + hdfs stuff | 14:47 |
| * tmckay reading back | 14:47 | |
| tmckay | I don't know that 8020 being open outside of the cluster is a bug | 14:49 |
| tmckay | If I've got separate clusters, I may have to open that up myself | 14:49 |
| balle | the problem is that there is nothing listening on 8020 | 14:49 |
| tmckay | For example, yesterday I launched a cluster with a default security group, but I hadn't added rules to the default | 14:49 |
| tmckay | So Sahara could not even ssh into the instances I created until I added ssh rules to the default group | 14:50 |
| elmiko | but this sounds like the namenode is not listening | 14:51 |
| balle | i can mkdir and put / rm etc stuff to hdfs from local namenode | 14:51 |
| elmiko | although in a spark deploy isn't there just one type of node? | 14:51 |
| balle | but i cannot access it via spark | 14:51 |
| tmckay | hmmm. I'd have to try this. | 14:52 |
| tmckay | I haven't actually ever tried to connect spark to an external hdfs. Only swift, which is up for review (WIP) | 14:53 |
| tmckay | balle, can you open a bug with details to track it? Any config information, or recipe for reproducibility would be helpful | 14:54 |
| tmckay | balle, cluster detains in sahara for the hdfs cluster should list the port for the namenode | 14:54 |
| tmckay | balle, you might be able to make this work by using a fully elaborated file path in the spark job submission (hdfs://host.port etc) | 14:56 |
| tmckay | "cluster details" | 14:56 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Migrate to oslo.log https://review.openstack.org/148861 | 15:07 |
| *** egafford has quit IRC | 15:13 | |
| *** egafford has joined #openstack-sahara | 15:13 | |
| balle | tmckay: ok filed a bug report https://bugs.launchpad.net/sahara/+bug/1413256 | 15:15 |
| elmiko | balle: thanks! | 15:16 |
| tmckay | balle, thanks. I am trying to fix up an OS instance right now where I can run multiple clusters -- neutron is not set up correctly :) But, I can switch back to nova and try to reproduce | 15:20 |
| tmckay | elmiko, I've got neutron going with default address ranges in devstack local.conf, but it fails assigning floating ips because it says the subnet created for the public network cannot reach the public network. This stuff was made automatically. | 15:26 |
| tmckay | strange. I think I'll switch back | 15:28 |
| elmiko | weird.. | 15:28 |
| elmiko | tmckay: just be aware that there are many features of sahara which won't work on nova-networking | 15:29 |
| *** Networkn3rd has joined #openstack-sahara | 15:30 | |
| tmckay | elmiko, like? | 15:30 |
| elmiko | floatingips! | 15:30 |
| elmiko | i think some of the indirect stuff may not work either, it will all be corner case stuff for what you are doing | 15:31 |
| elmiko | it seems like i keep running into bugs where not having neutron is an issue | 15:31 |
| tmckay | heh. I've had floating ips working on nova, I believe. | 15:31 |
| elmiko | i don't think that would work anymore, i just tried it yesterday and it failed | 15:31 |
| tmckay | maybe the shared resource can be a model for how to set up the neutron net... not sure what it's using | 15:31 |
| tmckay | I just don't get why an auto created net and subnet wouldn't just work | 15:33 |
| openstackgerrit | Ruslan Kamaldinov proposed openstack/sahara: Removed unused variable from tests https://review.openstack.org/148960 | 15:33 |
| elmiko | tmckay: yea, highly frustrating | 15:34 |
| tmckay | elmiko, I found a page where someone had a similar problem (same messages) and he posted how he wiped out the nets by hand and used the cli to recreate, success | 15:47 |
| tmckay | I may give that a try sometime soon | 15:47 |
| elmiko | tmckay: cool | 15:48 |
| elmiko | yea, for me the debug route would be to start checking all the virtual routers and what not, and even then you might be left scratching your head lol | 15:49 |
| elmiko | tmckay: you were correct about the floating ips on nova, i just double checked | 15:58 |
| elmiko | my test was for a corner case where they weren't working | 15:58 |
| tmckay | elmiko, ah, good to know. I think I'm going to have to go with nova for a bit before circling back to neutron | 16:00 |
| elmiko | probably best for all of us to be testing slightly different configs | 16:01 |
| *** skolekonov has quit IRC | 16:02 | |
| * tosky is for neutron | 16:08 | |
| elmiko | tosky: yea, i've been using neutron without issue for long time now | 16:11 |
| elmiko | long time = >6months | 16:11 |
| *** IvanBerezovskiy has left #openstack-sahara | 16:17 | |
| *** tosky_ has joined #openstack-sahara | 17:03 | |
| *** tosky has quit IRC | 17:03 | |
| *** Networkn3rd has quit IRC | 17:09 | |
| *** Networkn3rd has joined #openstack-sahara | 17:11 | |
| *** hdd has joined #openstack-sahara | 17:19 | |
| openstackgerrit | Merged openstack/sahara: Imported Translations from Transifex https://review.openstack.org/148803 | 17:31 |
| *** tosky_ has quit IRC | 17:36 | |
| *** tosky_ has joined #openstack-sahara | 17:37 | |
| *** Networkn3rd has quit IRC | 17:45 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Enable Spark jobs to access Swift URLs https://review.openstack.org/147949 | 17:46 |
| tmckay | elmiko, another belated "invalid" from triage day https://bugs.launchpad.net/sahara/+bug/1400477 | 18:01 |
| tmckay | I couldn't find any issues (I was hoping to try neutron too) | 18:02 |
| jodah | alazarev Not sure if you worked on the anti-affinity feature, but I'm curious to know when we'd want to place replicas on different racks versus placing them in different AZs. Any thoughts on this? | 18:04 |
| *** tosky_ has quit IRC | 18:26 | |
| *** Networkn3rd has joined #openstack-sahara | 18:31 | |
| *** coolsvap is now known as coolsvap|afk | 18:33 | |
| *** amcrn has joined #openstack-sahara | 18:45 | |
| *** sgotliv_ has quit IRC | 18:51 | |
| *** Longgeek has quit IRC | 19:13 | |
| tellesnobrega | tmckay, did you write a spark-edp spec? | 19:20 |
| tmckay | tellesnobrega, spark EDP might have happened before we started using specs | 19:21 |
| tellesnobrega | tmckay, i imagined that, couldn't find | 19:22 |
| tellesnobrega | its ok, i'm writing the blueprint for storm-edp so i can start the spec | 19:22 |
| tmckay | elmiko, what do you think about putting a SparkWrapper and a MainWrapper for Java actions in the same JAR? I'm wondering if they should be separated. Very similar, but they are different and for different plugins | 19:23 |
| tmckay | also, splitting this up makes the CRs easier to handle since there are no dependencies | 19:29 |
| *** sgotliv has joined #openstack-sahara | 19:30 | |
| openstackgerrit | Merged openstack/sahara: Removed sad line https://review.openstack.org/148932 | 19:39 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-extra: Add a SparkWrapper class https://review.openstack.org/147955 | 19:44 |
| *** jodah has quit IRC | 19:54 | |
| *** jodah has joined #openstack-sahara | 19:55 | |
| tellesnobrega | tmckay, does storm edp brings REST api impacts? | 20:16 |
| tmckay | tellesnobrega, I don't think so. | 20:23 |
| tmckay | anything that needs to be specified should be doable through configuration values | 20:24 |
| tellesnobrega | sounds about right | 20:25 |
| tmckay | other than that, basic job endpoints we've got now should work | 20:28 |
| *** akuznetsov has joined #openstack-sahara | 20:34 | |
| *** Networkn3rd has quit IRC | 20:37 | |
| elmiko | tmckay: i'm not sure what makes the most sense in java(re: splitting the jars), but i think making more digestable pieces is probably for the best | 20:38 |
| tmckay | elmiko, yeah, going down that path now. No sense having dependencies on CRs and/or jars when they're not needed | 20:39 |
| elmiko | tmckay: agreed | 20:39 |
| tmckay | elmiko, and technially, from a QE perspective, if you change that jar you can't know if the other class changed | 20:39 |
| tmckay | elmiko, thanks for the 2 cents | 20:40 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara-specs: Storm EDP specification https://review.openstack.org/149047 | 20:56 |
| *** akuznetsov has quit IRC | 21:05 | |
| *** witlessb has quit IRC | 21:06 | |
| *** witlessb has joined #openstack-sahara | 21:07 | |
| *** amcrn has quit IRC | 21:28 | |
| *** Networkn3rd has joined #openstack-sahara | 21:38 | |
| *** chlong has quit IRC | 22:05 | |
| *** crobertsrh is now known as _crobertsrh | 22:05 | |
| *** Networkn3rd has quit IRC | 22:06 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara: Add Swift integration with Spark https://review.openstack.org/146659 | 22:18 |
| *** hdd has quit IRC | 22:30 | |
| *** tellesnobrega_ has joined #openstack-sahara | 22:32 | |
| *** Longgeek has joined #openstack-sahara | 22:36 | |
| *** Longgeek has quit IRC | 22:40 | |
| *** tellesnobrega_ has quit IRC | 22:41 | |
| *** tellesnobrega_ has joined #openstack-sahara | 22:49 | |
| *** tellesnobrega_ has quit IRC | 23:00 | |
| *** chlong has joined #openstack-sahara | 23:33 | |
| *** witlessb has quit IRC | 23:37 | |
| *** hdd has joined #openstack-sahara | 23:37 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!