| *** Networkn3rd has joined #openstack-sahara | 01:56 | |
| *** stanchan has joined #openstack-sahara | 02:55 | |
| *** Networkn3rd has quit IRC | 03:16 | |
| *** coolsvap|afk is now known as coolsvap | 03:27 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add CDH5.4 support in sahara https://review.openstack.org/177660 | 03:33 |
|---|---|---|
| openstackgerrit | Merged openstack/sahara: [CDH] swift lib support https://review.openstack.org/173205 | 03:56 |
| openstackgerrit | Merged openstack/sahara: Adding config hints for CDH plugin https://review.openstack.org/169466 | 03:56 |
| *** nkrinner has joined #openstack-sahara | 04:57 | |
| *** Nikolay_St has quit IRC | 05:09 | |
| *** hdd has joined #openstack-sahara | 05:24 | |
| *** sgotliv has quit IRC | 06:01 | |
| *** Longgeek has joined #openstack-sahara | 06:03 | |
| *** Longgeek has quit IRC | 06:08 | |
| *** pcaruana has joined #openstack-sahara | 06:09 | |
| *** tnovacik has joined #openstack-sahara | 06:28 | |
| *** Longgeek has joined #openstack-sahara | 06:56 | |
| *** Longgeek has quit IRC | 07:02 | |
| *** Longgeek has joined #openstack-sahara | 07:06 | |
| *** sgotliv has joined #openstack-sahara | 07:34 | |
| *** Nikolay_St has joined #openstack-sahara | 07:49 | |
| *** sgotliv has quit IRC | 07:57 | |
| *** Longgeek has quit IRC | 08:05 | |
| *** witlessb has joined #openstack-sahara | 08:28 | |
| *** witlessb has quit IRC | 08:29 | |
| *** witlessb has joined #openstack-sahara | 08:31 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Added unit tests for service/api module https://review.openstack.org/178205 | 08:33 |
| *** pino|work has joined #openstack-sahara | 08:33 | |
| *** esikachev has joined #openstack-sahara | 08:48 | |
| *** sgotliv has joined #openstack-sahara | 08:53 | |
| *** Longgeek has joined #openstack-sahara | 09:06 | |
| *** Longgeek has quit IRC | 09:12 | |
| *** Longgeek has joined #openstack-sahara | 09:13 | |
| openstackgerrit | Pino Toscano proposed openstack/sahara-image-elements: Improvements to README.rst of elements https://review.openstack.org/177135 | 09:19 |
| *** tnovacik has quit IRC | 09:31 | |
| *** sgotliv has quit IRC | 09:55 | |
| *** tnovacik has joined #openstack-sahara | 10:10 | |
| *** sgotliv has joined #openstack-sahara | 10:13 | |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Fix management IPs usage https://review.openstack.org/178277 | 10:16 |
| *** _crobertsrh has quit IRC | 10:41 | |
| openstackgerrit | Nikita Konovalov proposed openstack/sahara: Improved unit test coverage of periodic module https://review.openstack.org/178197 | 11:20 |
| *** coolsvap is now known as coolsvap|afk | 11:33 | |
| *** crobertsrh has joined #openstack-sahara | 11:41 | |
| *** esikachev has quit IRC | 11:45 | |
| *** witlessb has quit IRC | 11:45 | |
| *** witlessb has joined #openstack-sahara | 11:46 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Improve unit tests of general utils https://review.openstack.org/178627 | 11:54 |
| *** openstackgerrit has quit IRC | 12:07 | |
| *** openstackgerrit has joined #openstack-sahara | 12:07 | |
| *** esikachev has joined #openstack-sahara | 12:08 | |
| *** raildo has joined #openstack-sahara | 12:42 | |
| *** chlong has joined #openstack-sahara | 12:44 | |
| tellesnobrega | elmiko, regarding that rebalance call that storm needs when scaling up, I'm thinking in implementing it at edp.service.storm.engine and calling it when scaling does that sound ok? | 13:12 |
| tellesnobrega | or should it be at plugins.storm.plugin.py | 13:12 |
| elmiko | tellesnobrega: i think it might make sense to have it in the engine class | 13:22 |
| elmiko | and then call it once the scaling nodes have been added | 13:22 |
| elmiko | that makes sense to me | 13:22 |
| tellesnobrega | elmiko, ok... i need to have the storm edp approved to get this as well... i'll add as dependency | 13:29 |
| tellesnobrega | can you take a look? https://review.openstack.org/#/c/168302/ | 13:29 |
| elmiko | tellesnobrega: ack, i'll take a look | 13:30 |
| tellesnobrega | thanks | 13:30 |
| *** flaviosr has joined #openstack-sahara | 13:47 | |
| tellesnobrega | anyone can give me and flaviosr a hand on installing sahara production mode | 13:48 |
| tellesnobrega | ? | 13:48 |
| elmiko | what issues are you having? | 13:48 |
| tellesnobrega | flaviosr, way in here | 13:49 |
| tellesnobrega | he is actually the one trying to install it | 13:49 |
| flaviosr | elmiko, I am installing sahara on a juno production cloud, sahara guides says you need pip to install it, but pip updates some dependencies not supported by other openstack modules | 13:51 |
| flaviosr | there is another way to install it, or installing in a virtualenv is the way to go? | 13:52 |
| elmiko | flaviosr: is it possible to isolate sahara into a virtual machine of it's own? | 13:52 |
| elmiko | otherwise, virtualenv might be your best bet | 13:52 |
| flaviosr | the best scenario is to install it on the same machine | 13:53 |
| elmiko | then probably best to use a virtualenv | 13:53 |
| flaviosr | ok, thanks :) | 13:53 |
| elmiko | np | 13:54 |
| *** hdd has quit IRC | 13:54 | |
| tellesnobrega | thanks elmiko | 14:06 |
| -openstackstatus- NOTICE: gerrit has been restarted to clear a stuck events queue. any change events between 13:29-14:05 utc should be rechecked or have their approval votes reapplied to trigger jobs | 14:06 | |
| tellesnobrega | after some good experience we are putting sahara for production here, maybe change the way experiments are carried out in the lab | 14:06 |
| elmiko | tellesnobrega: awesome! | 14:07 |
| tellesnobrega | i need to study a little more on the job types and stuff like that to present to the lab how to use it and why it is way better than making clusters by hand | 14:08 |
| elmiko | i'd be curious to hear your results, i think most of the team would be | 14:08 |
| SergeyLukjanov | elmiko, tmckay, please, take a look on release notes draft https://etherpad.openstack.org/p/sahara-kilo-release-notes | 14:08 |
| SergeyLukjanov | crobertsrh, ^^ | 14:08 |
| elmiko | SergeyLukjanov: ack | 14:09 |
| crobertsrh | ack | 14:09 |
| SergeyLukjanov | we probably need to add more info to it | 14:09 |
| SergeyLukjanov | sample for other projects - https://wiki.openstack.org/wiki/ReleaseNotes/Kilo | 14:09 |
| tmckay | SergeyLukjanov, okay, will do | 14:09 |
| tellesnobrega | glad to see that storm and zookeeper are part of kilo :) | 14:17 |
| *** flaviosr has quit IRC | 14:19 | |
| *** egafford has joined #openstack-sahara | 14:21 | |
| *** nkrinner has quit IRC | 14:24 | |
| *** esikachev has quit IRC | 14:29 | |
| *** esikachev has joined #openstack-sahara | 14:42 | |
| openstackgerrit | Nikita Konovalov proposed openstack/sahara: Remove unused code from sqlalchemy api module https://review.openstack.org/178708 | 14:57 |
| *** Networkn3rd has joined #openstack-sahara | 15:03 | |
| *** Nikolay_St has quit IRC | 15:23 | |
| elmiko | tellesnobrega: i posted a couple questions on the storm edp review | 15:24 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add unit tests for exceptions module https://review.openstack.org/178728 | 15:28 |
| *** esikachev has quit IRC | 15:29 | |
| *** hdd has joined #openstack-sahara | 15:31 | |
| *** esikachev has joined #openstack-sahara | 15:32 | |
| openstackgerrit | Nikita Konovalov proposed openstack/sahara: Fix InvalidRequestError being skipped https://review.openstack.org/178745 | 15:44 |
| *** esikachev has quit IRC | 15:48 | |
| tellesnobrega | elmiko, looking into it right now | 16:07 |
| tellesnobrega | elmiko, new patch on the way | 16:10 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Storm EDP implementation https://review.openstack.org/168302 | 16:13 |
| *** chlong has quit IRC | 16:19 | |
| *** pino|work has quit IRC | 16:20 | |
| *** pcaruana has quit IRC | 16:22 | |
| *** pino|work has joined #openstack-sahara | 16:27 | |
| openstackgerrit | Pino Toscano proposed openstack/sahara-image-elements: Improvements to README.rst of elements https://review.openstack.org/177135 | 16:41 |
| pino|work | elmiko: all yours ;) | 16:42 |
| elmiko | pino|work: thanks! | 17:01 |
| elmiko | tellesnobrega: thanks | 17:01 |
| *** sgotliv has quit IRC | 17:13 | |
| *** tellesnobrega_ has joined #openstack-sahara | 17:16 | |
| tellesnobrega | elmiko, thanks | 17:21 |
| *** tellesnobrega_ has quit IRC | 17:21 | |
| elmiko | np, i just wasn't sure about those few lines | 17:21 |
| tellesnobrega | sure, it was a good catch | 17:21 |
| tellesnobrega | SergeyLukjanov, tmckay if you guys can take a look https://review.openstack.org/168302 it would be great | 17:25 |
| *** stanchan has quit IRC | 17:34 | |
| *** stanchan has joined #openstack-sahara | 17:49 | |
| *** hdd has quit IRC | 17:57 | |
| *** hdd has joined #openstack-sahara | 17:58 | |
| *** stanchan has quit IRC | 18:00 | |
| *** stanchan has joined #openstack-sahara | 18:04 | |
| *** dhellmann has quit IRC | 18:05 | |
| *** stanchan has quit IRC | 18:22 | |
| tmckay | tellesnobrega, so there is no possibility of passing arguments to a storm topology? | 18:22 |
| tmckay | everything is already set in the jar, I guess? | 18:23 |
| tellesnobrega | huum, actually there is... | 18:23 |
| tellesnobrega | if the user creates a jar that expects for args | 18:23 |
| tmckay | looks like the args stuff was removed in the last patch | 18:23 |
| tmckay | where elmiko commented | 18:24 |
| tellesnobrega | yes, I forgot about this point, changing it back again | 18:24 |
| tellesnobrega | normally args arent very normal, but its possible, so i will fix that | 18:25 |
| tmckay | :) okay. don't listen to elmiko too closely :) (just kidding) | 18:25 |
| elmiko | hehe, that's what i was curious about | 18:25 |
| tellesnobrega | lol | 18:25 |
| elmiko | the args were set but never passed to the storm call | 18:25 |
| elmiko | tmckay: thanks.... ;) | 18:25 |
| tellesnobrega | my bad, being crazy swamped, didnt think it through | 18:25 |
| tmckay | elmiko, gotcha. He removed them altogether -- one way to fix it | 18:25 |
| tmckay | "No args for you!" | 18:25 |
| elmiko | tmckay: right, i didn't know which was correct | 18:26 |
| elmiko | lol | 18:26 |
| tmckay | I will refrain from obvious -1 stat increase | 18:26 |
| *** Nikolay_St has joined #openstack-sahara | 18:31 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Storm EDP implementation https://review.openstack.org/168302 | 18:36 |
| elmiko | tellesnobrega: ok, looks much better | 18:37 |
| tellesnobrega | elmiko, thanks | 18:37 |
| * tmckay looking now | 18:38 | |
| tmckay | oops, tellesnobrega, args192 | 18:39 |
| tmckay | cut and paste from the review page? | 18:39 |
| tellesnobrega | exactly | 18:39 |
| tellesnobrega | lol | 18:39 |
| tmckay | heh | 18:39 |
| elmiko | i need to have my eyes examined... | 18:39 |
| tmckay | tellesnobrega, here's a hint | 18:40 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Storm EDP implementation https://review.openstack.org/168302 | 18:40 |
| tmckay | if you run %(topology_name)s %(args)s together with no space, and make sure that the space is on the front of args, you can avoid an extra space in the case that args is empty | 18:41 |
| tmckay | tellesnobrega, I think I did this in the spark stuff, not sure ^^ | 18:41 |
| tellesnobrega | tmckay, cool | 18:41 |
| tmckay | that extra space caused me problems in test code so I found a way to get rid of it. | 18:41 |
| tellesnobrega | cool | 18:41 |
| * tmckay looks at the spark stuff, I know i did this somewhere | 18:41 | |
| tmckay | tellesnobrega, yeah, around line 252 in sahara/service/edp/spark/engine.py | 18:44 |
| tmckay | it tags on the " " after assigning args, and then leaves it out in the format. Up to you. | 18:44 |
| tellesnobrega | i think i did that as well | 18:45 |
| tmckay | tellesnobrega, not for args, not as it currently is | 18:46 |
| tellesnobrega | looking at it now | 18:46 |
| tmckay | only makes sense for stuff that might not be there | 18:46 |
| openstackgerrit | Nikolay Starodubtsev proposed openstack/sahara: Add unittests for EDP API https://review.openstack.org/178799 | 18:47 |
| tmckay | tellesnobrega, if the test code doesn't look for the command output, it's probably not necessary | 18:50 |
| tellesnobrega | sure, i'm running tests again to check if i need that or not | 18:50 |
| tmckay | could add another unit test for the case when "args" is specified | 18:52 |
| tellesnobrega | yeah | 18:54 |
| tellesnobrega | thinking about that | 18:54 |
| tmckay | egafford, ping, we have a similar case with storm topologies as spark jobs -- uploaded and run out of /tmp/storm-edp on master | 19:01 |
| tmckay | egafford, do you think https://review.openstack.org/#/c/150117/2/sahara/plugins/spark/plugin.py is generalizable, or do we need a parallel impl for storm? | 19:01 |
| tmckay | tellesnobrega, ^^ This thing loops around once in a while in a cron job on the cluster and cleans up the job dir | 19:02 |
| tmckay | maybe something to add for Storm jobs, since it follows the pattern of the spark engine | 19:02 |
| tmckay | hmm, each time we do something like this, we'll need the same functionality | 19:03 |
| tellesnobrega | seems ok to me | 19:03 |
| *** Nikolay_St has quit IRC | 19:03 | |
| tellesnobrega | i can do it | 19:04 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Storm EDP implementation https://review.openstack.org/168302 | 19:05 |
| tellesnobrega | tmckay, elmiko, egafford ^ | 19:05 |
| tmckay | tellesnobrega, not a burning issue -- anytime in Liberty would be fine. If we can refactor and generalize between spark and storm (not sure if it's viable, with the cluster provisioning involved) then that probably gets us the same functionality in another engine, too, should we have one | 19:05 |
| egafford | tmckay: Looking now. | 19:05 |
| tellesnobrega | cool | 19:05 |
| elmiko | tellesnobrega: ack | 19:05 |
| *** dhellmann has joined #openstack-sahara | 19:06 | |
| tmckay | egafford, not a top priority, but maybe generalization (supporting classes at any rate) makes some sense | 19:06 |
| tmckay | egafford, if we do it twice, we'll probably do it N times :) | 19:06 |
| egafford | tmckay, tellesnobrega: There's nothing Spark-specific in that CR really, I agree. | 19:06 |
| tmckay | but there is the provisioning side | 19:06 |
| tmckay | which is in the spark plugin | 19:06 |
| tmckay | sharing that stuff across the storm plugin might take some small amount of refactoring, I would guess | 19:07 |
| egafford | So long as other engines use the same job storage structure, the solution is reusable; it's just looking at the size and age of directories. | 19:07 |
| tellesnobrega | how many plugins can we think of that would follow the same pattern? | 19:07 |
| egafford | tellesnobrega: Anything that doesn't use Oozie (or another engine with a pre-ordained job storage mechanism) can sensibly follow pattern, I'd think. | 19:08 |
| tmckay | egafford, would you recommend copy and paste, or refactor and share? | 19:08 |
| tmckay | I'm on the fence ... | 19:09 |
| elmiko | i wonder if we should consider creating an alternative to the EdpOozieEngine that could encapsulate some of these patterns? | 19:11 |
| egafford | tmckay: Yeah, we don't have a perfect place to abstract to just now, but... | 19:11 |
| elmiko | we could call it EdpNotOozieEngine ;) | 19:11 |
| tmckay | elmiko, could be. we could make an "EdpSshEngine" base class | 19:12 |
| tmckay | heh | 19:12 |
| elmiko | yea, exactly. your name is much more reasonable | 19:12 |
| tmckay | egafford, tellesnobrega, why don't we copy and paste, see how it turns out, then look for commonality in another CR? | 19:13 |
| tmckay | tellesnobrega, if you're swamped, just add a bug to track it and come back to it in Liberty sometime | 19:13 |
| elmiko | +1 for making it another CR | 19:14 |
| tmckay | you've got 6 months before it's released :) | 19:14 |
| tellesnobrega | i'm ok with that | 19:14 |
| egafford | tmckay: It's certainly an option, and not a bad one. | 19:14 |
| tellesnobrega | tmckay, lol, sure | 19:14 |
| tellesnobrega | it's already on my todo | 19:14 |
| tmckay | egafford, ringing endorsement "it's an option, and not a bad one" | 19:14 |
| tmckay | sign me up! | 19:15 |
| egafford | tmckay: Hedging wildly is the better part of discretion, which is the better part of valor, so >25% of valor is hedging wildly. Maths. | 19:18 |
| tmckay | lol | 19:18 |
| *** stanchan has joined #openstack-sahara | 19:18 | |
| elmiko | nice | 19:18 |
| *** nkrinner has joined #openstack-sahara | 19:29 | |
| *** Nikolay_St has joined #openstack-sahara | 19:31 | |
| *** Nikolay_St has quit IRC | 19:36 | |
| openstackgerrit | Emilien Macchi proposed stackforge/puppet-sahara: Beaker tests https://review.openstack.org/178821 | 19:47 |
| *** Nikolay_St has joined #openstack-sahara | 19:54 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Storm EDP implementation https://review.openstack.org/168302 | 20:15 |
| tellesnobrega | missed indentation tmckay, elmiko, egafford ^ | 20:15 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Implementation of Storm scaling https://review.openstack.org/178837 | 20:42 |
| *** raildo has quit IRC | 20:58 | |
| *** Networkn3rd has quit IRC | 21:02 | |
| *** nkrinner has quit IRC | 21:25 | |
| *** Longgeek has quit IRC | 21:29 | |
| *** crobertsrh is now known as _crobertsrh | 21:45 | |
| *** hdd has quit IRC | 22:03 | |
| *** witlessb has quit IRC | 22:25 | |
| *** egafford has quit IRC | 22:38 | |
| *** stanchan has quit IRC | 23:43 | |
| *** tnovacik has quit IRC | 23:44 | |
| *** chlong has joined #openstack-sahara | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!