| *** Longgeek has joined #openstack-sahara | 00:38 | |
| *** Longgeek has quit IRC | 00:44 | |
| *** htruta has quit IRC | 00:59 | |
| *** david-lyle has quit IRC | 01:09 | |
| *** tmckay has quit IRC | 01:14 | |
| *** chen12 has joined #openstack-sahara | 01:20 | |
| *** saneax has quit IRC | 01:48 | |
| *** ViswaV has quit IRC | 01:54 | |
| *** Poornima has joined #openstack-sahara | 02:25 | |
| *** Longgeek has joined #openstack-sahara | 02:40 | |
| *** Longgeek has quit IRC | 02:52 | |
| *** Poornima has quit IRC | 03:14 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Enable Java Keystore KMS service in CDH5.4 https://review.openstack.org/189373 | 03:43 |
|---|---|---|
| *** Longgeek has joined #openstack-sahara | 03:44 | |
| *** Longgeek has quit IRC | 03:46 | |
| *** Longgeek has joined #openstack-sahara | 03:47 | |
| *** Poornima has joined #openstack-sahara | 03:53 | |
| *** Poornima has quit IRC | 03:54 | |
| *** Poornima has joined #openstack-sahara | 03:54 | |
| *** saneax has joined #openstack-sahara | 04:17 | |
| *** ghenriks has joined #openstack-sahara | 04:24 | |
| *** vgridnev has joined #openstack-sahara | 05:05 | |
| *** Nikolay_St has quit IRC | 05:09 | |
| *** Nikolay_St has joined #openstack-sahara | 05:26 | |
| *** Nikolay_St has quit IRC | 05:34 | |
| *** vgridnev has quit IRC | 05:36 | |
| *** links has joined #openstack-sahara | 05:55 | |
| *** vgridnev has joined #openstack-sahara | 05:56 | |
| *** vgridnev has quit IRC | 06:02 | |
| *** vgridnev has joined #openstack-sahara | 06:33 | |
| *** coolsvap|away is now known as coolsvap | 06:44 | |
| *** coolsvap is now known as coolsvap|away | 06:56 | |
| *** esikachev has joined #openstack-sahara | 07:01 | |
| *** chen12 has quit IRC | 07:07 | |
| *** chen12 has joined #openstack-sahara | 07:07 | |
| *** Nikolay_St has joined #openstack-sahara | 07:30 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add CM API support for enable hdfs HA https://review.openstack.org/195901 | 07:33 |
| *** hdd has joined #openstack-sahara | 07:33 | |
| *** hdd has quit IRC | 07:34 | |
| *** vgridnev has quit IRC | 07:34 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Enable Java Keystore KMS service in CDH5.4 https://review.openstack.org/189373 | 07:35 |
| *** pino|work has joined #openstack-sahara | 07:53 | |
| *** Poornima has quit IRC | 08:25 | |
| *** Poornima has joined #openstack-sahara | 08:58 | |
| *** pino|work has quit IRC | 09:14 | |
| *** pino|work has joined #openstack-sahara | 09:15 | |
| openstackgerrit | Xingchao Yu proposed openstack/puppet-sahara: Update README.md for project namespace change https://review.openstack.org/195948 | 09:29 |
| *** tosky has joined #openstack-sahara | 09:32 | |
| openstackgerrit | Xingchao Yu proposed openstack/puppet-sahara: Update README.md for project namespace change https://review.openstack.org/195948 | 09:50 |
| *** links has quit IRC | 10:36 | |
| *** links has joined #openstack-sahara | 11:17 | |
| *** sgotliv has quit IRC | 11:18 | |
| *** sgotliv has joined #openstack-sahara | 11:19 | |
| *** links has quit IRC | 11:19 | |
| *** Longgeek_ has joined #openstack-sahara | 12:17 | |
| *** Longgeek has quit IRC | 12:21 | |
| *** Poornima has quit IRC | 12:21 | |
| *** raildo has joined #openstack-sahara | 12:37 | |
| *** AndreyPavlov has quit IRC | 12:38 | |
| *** Longgeek has joined #openstack-sahara | 12:41 | |
| *** htruta has joined #openstack-sahara | 12:44 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Improvement check scale in scenario tests https://review.openstack.org/195513 | 12:44 |
| *** tellesnobrega has joined #openstack-sahara | 12:45 | |
| *** Longgeek_ has quit IRC | 12:45 | |
| *** htruta has quit IRC | 12:53 | |
| *** tellesnobrega has quit IRC | 13:04 | |
| *** tellesnobrega has joined #openstack-sahara | 13:08 | |
| *** tosky_ has joined #openstack-sahara | 13:27 | |
| *** tosky is now known as Guest9357 | 13:28 | |
| *** tosky_ is now known as tosky | 13:28 | |
| *** tosky_ has joined #openstack-sahara | 13:31 | |
| *** tosky is now known as Guest59661 | 13:31 | |
| *** tosky_ is now known as tosky | 13:31 | |
| *** Guest9357 has quit IRC | 13:31 | |
| *** Guest59661 has quit IRC | 13:32 | |
| *** hdd has joined #openstack-sahara | 13:37 | |
| *** raildo has quit IRC | 13:45 | |
| *** raildo has joined #openstack-sahara | 13:47 | |
| *** htruta has joined #openstack-sahara | 13:49 | |
| *** Tiby has joined #openstack-sahara | 13:52 | |
| *** egafford has joined #openstack-sahara | 13:59 | |
| egafford | alazarev: Ping | 14:01 |
| elmiko | egafford: he's west coast, might not be around yet | 14:02 |
| egafford | elmiko: Ah yes. I'll ping in a bit. | 14:03 |
| elmiko | he might, i dunno. just an fyo | 14:03 |
| elmiko | er fyi ;) | 14:03 |
| *** saneax has quit IRC | 14:12 | |
| pino|work | elmiko: fyo = for your organization, right? ;) | 14:16 |
| tellesnobrega | elmiko, when we don't have enough quotas for multiple clusters... what should be the expected output | 14:16 |
| tellesnobrega | ? | 14:16 |
| tellesnobrega | i can't remember what we decided | 14:16 |
| tellesnobrega | or if we did | 14:16 |
| elmiko | tellesnobrega: i thought the default behavior is return the cluster ids and let the user inspect them | 14:16 |
| elmiko | i suppose if you can't create *any* clusters then maybe return an error. | 14:17 |
| tellesnobrega | hum, ok | 14:17 |
| elmiko | like if you don't have quota for anything, or if you don't have permission to use the multiple cluster, then i could see returning an error | 14:17 |
| elmiko | but in general, i would prefer to see it as a thin wrapper around cluster creation | 14:18 |
| elmiko | this goes back into the idea of treating the mulitple clusters as a resource | 14:18 |
| elmiko | since we are not, then it should just be a thin wrapper | 14:18 |
| elmiko | tellesnobrega: if you would like, we can bring it up with the api-wg to get more ideas? | 14:19 |
| elmiko | pino|work: yea, that works ;) | 14:19 |
| tellesnobrega | api-wg? | 14:20 |
| elmiko | the api working group | 14:21 |
| tellesnobrega | elmiko, i like the idea (as much as possible) to return error on extreme cases and let the user figure out the state of each cluster if we can create at least one | 14:21 |
| elmiko | tellesnobrega: yea, i agree | 14:21 |
| tellesnobrega | i just need to figure out how to do it | 14:21 |
| elmiko | maybe in v2 api we can revisit this and make it into a proper task with more options for querying the status of the entire operation. (but that's a ways away) | 14:22 |
| tellesnobrega | for sure | 14:22 |
| elmiko | i would make it simple to start, don't try to encode too much logic into the /multiple endpoint | 14:23 |
| elmiko | if the user doesn't have quota to make any clusters, just let the clusters each go into error state. (or w/e happens when you don't have quota) | 14:23 |
| elmiko | otherwise we are starting to put too much logic into this "helper" endpoint | 14:24 |
| tellesnobrega | yes, right now, the response is if all get created, we get a list of ids | 14:24 |
| tellesnobrega | if quotas exceed, at any time, we get quotas exceeded exceptions | 14:25 |
| tellesnobrega | just need to change this | 14:25 |
| elmiko | ok | 14:25 |
| *** hdd has quit IRC | 14:33 | |
| *** sgotliv has quit IRC | 14:34 | |
| tellesnobrega | elmiko, figured out one way to do it, but i got a problem | 14:37 |
| tellesnobrega | just added a check before the exception (quota exceeded is launched) to check if there is no cluster id in the list | 14:38 |
| tellesnobrega | this worked perfectly | 14:38 |
| elmiko | ok, that could work | 14:38 |
| tellesnobrega | the problem is, since i'm not throwing the exception, it keeps trying to create clusters | 14:39 |
| elmiko | i'm just concerned about queries where some work and some don't | 14:39 |
| elmiko | yea | 14:39 |
| tellesnobrega | when it should stop | 14:39 |
| tellesnobrega | this work for queries where you go out of quota mid way through the call | 14:40 |
| elmiko | yea | 14:40 |
| tellesnobrega | the return is a list of ids | 14:40 |
| tellesnobrega | but it tries to create all clusters, doesn't stop once the quota problem is found | 14:40 |
| elmiko | hmm | 14:41 |
| elmiko | i would only return an error if *none* of the clusters are created | 14:41 |
| tellesnobrega | it is working this way | 14:46 |
| tellesnobrega | i get quotas exception when none is created | 14:46 |
| *** esikachev has quit IRC | 14:46 | |
| tellesnobrega | and a list of clusters even if some are created and some are not | 14:46 |
| elmiko | that sounds good to me | 14:47 |
| tellesnobrega | cool, i'm going to write a test for no quotas case and put under review | 14:47 |
| elmiko | cool | 14:47 |
| tosky | SergeyLukjanov, degorenko, sreshetnyak: did you ever try the Namenode HA implemented in Kilo? I have some troubles building a cluster following the instruction, do you have a working profile? | 14:51 |
| *** htruta has quit IRC | 14:58 | |
| *** Nikolay_St has quit IRC | 15:02 | |
| tellesnobrega | anyone knows of any problems with the tests?? just pulled from masters and ran the tests and I got 151 errors, most failing to load spark, cdh and vanilla plugin | 15:11 |
| tellesnobrega | and some with ImportError: No module named v2_3_0.versionhandler | 15:12 |
| elmiko | haven't seen that, i'll give it a run | 15:12 |
| elmiko | tellesnobrega: did you try running with -r? | 15:12 |
| tellesnobrega | if running with -f now | 15:13 |
| tellesnobrega | to reinstall venv | 15:13 |
| tellesnobrega | see what happens | 15:13 |
| elmiko | unit tests all passed for me | 15:14 |
| elmiko | tellesnobrega: yea, everything comes up green on mine | 15:16 |
| tellesnobrega | weird | 15:18 |
| tellesnobrega | still failing here | 15:18 |
| tellesnobrega | maybe its a OS X problem | 15:18 |
| tellesnobrega | changing to linux | 15:18 |
| elmiko | i'm using fedora 21, fwiw | 15:19 |
| *** hdd has joined #openstack-sahara | 15:22 | |
| *** Tiby has quit IRC | 15:27 | |
| *** tellesnobrega has quit IRC | 15:34 | |
| *** tellesnobrega has joined #openstack-sahara | 15:38 | |
| openstackgerrit | Sergey Vilgelm proposed openstack/sahara: Use right oslo.service entry points https://review.openstack.org/196111 | 15:39 |
| openstackgerrit | Sergey Vilgelm proposed openstack/sahara: Use right oslo.service entry points https://review.openstack.org/196111 | 15:41 |
| tellesnobrega | elmiko, it is a OS X problem i guess | 15:46 |
| tellesnobrega | worked on ubuntu | 15:47 |
| elmiko | ok, cool | 15:48 |
| *** ViswaV has joined #openstack-sahara | 15:51 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara: Allow multiple clusters creation https://review.openstack.org/193586 | 15:51 |
| *** ViswaV_ has joined #openstack-sahara | 15:55 | |
| *** htruta has joined #openstack-sahara | 15:56 | |
| *** ViswaV has quit IRC | 15:56 | |
| tellesnobrega | elmiko, done ^ | 15:57 |
| elmiko | tellesnobrega: ack | 15:57 |
| *** hdd has quit IRC | 16:09 | |
| *** tellesnobrega has quit IRC | 16:28 | |
| *** tellesnobrega has joined #openstack-sahara | 16:35 | |
| *** pino|work has quit IRC | 16:43 | |
| *** pino|work has joined #openstack-sahara | 16:44 | |
| *** pino|work has quit IRC | 16:51 | |
| *** Longgeek has quit IRC | 17:02 | |
| *** tellesnobrega has quit IRC | 17:03 | |
| *** ViswaV_ has quit IRC | 17:09 | |
| *** ViswaV has joined #openstack-sahara | 17:22 | |
| *** ViswaV_ has joined #openstack-sahara | 17:24 | |
| *** ViswaV has quit IRC | 17:26 | |
| egafford | alazarev: ping | 17:44 |
| alazarev | egafford, pong | 17:44 |
| egafford | Hi alazarev. tosky found this bug: https://bugs.launchpad.net/sahara/+bug/1468722. I investigated for a bit, and found that it looks as though the cluster deletion periodic job is counting on the existence of a trust_id, but that the trust_id is only created for transient clusters. | 17:45 |
| openstack | Launchpad bug 1468722 in Sahara "Periodic cleanup of non-final clusters moves the cluster into Error instead of removing it" [High,New] - Assigned to Ethan Gafford (egafford) | 17:45 |
| egafford | I was wondering if I was missing something critical about the periodic cluster cleanup feature, or whether we need to do one of (have the feature only apply to transient clusters, add trusts to permanent clusters, or change the way we build contexts in this case to not rely on trusts.) | 17:46 |
| egafford | Thoughts? | 17:47 |
| alazarev | egafford, yes, I saw a bug, looks like real issue | 17:47 |
| egafford | alazarev: Okay, cool. It would certainly be simple enough to add trusts to standing clusters, and gate the cleanup job on both CONF.use_identity_api_v3 and the existence of a trust_id (for backward compatibility.) | 17:48 |
| egafford | Are there downsides to that path that you can see? | 17:48 |
| alazarev | egafford, I thought about the same thing | 17:48 |
| alazarev | egafford, it will also help with 'long operations' | 17:49 |
| alazarev | egafford, there is a bp for that | 17:49 |
| *** hdd has joined #openstack-sahara | 17:49 | |
| egafford | Yup, I see it. | 17:49 |
| egafford | And see that you're targeting it (and assigned) for L2. | 17:50 |
| *** ViswaV_ has quit IRC | 17:50 | |
| *** hdd has quit IRC | 17:50 | |
| *** hdd has joined #openstack-sahara | 17:52 | |
| egafford | alazarev: Okay, so should we assume that this feature will be bugged for Kilo until a backport from L2? I'd be happy to implement the change (we're in a late-test product cycle here, and it'd be ideal to have the cluster cleanup feature for permanent clusters,) but if you're on track and have a plan, that's fine as well. | 17:53 |
| egafford | tosky: ^ This discussion is relevant to your interests. | 17:54 |
| tosky | yep | 17:54 |
| alazarev | egafford, go ahead, I'm on parental leave now :) | 17:54 |
| egafford | alazarev: Congrats! | 17:55 |
| egafford | Will do; thanks alazarev. | 17:55 |
| alazarev | egafford, thanks! | 17:55 |
| egafford | Will handle that; enjoy! | 17:55 |
| elmiko | alazarev: congrats! | 18:00 |
| *** Longgeek has joined #openstack-sahara | 18:02 | |
| *** ViswaV has joined #openstack-sahara | 18:03 | |
| *** hogepodge has quit IRC | 18:06 | |
| *** Longgeek has quit IRC | 18:08 | |
| *** hdd has quit IRC | 18:11 | |
| *** tosky has quit IRC | 18:14 | |
| *** hogepodge has joined #openstack-sahara | 18:41 | |
| *** Nikolay_St has joined #openstack-sahara | 18:51 | |
| elmiko | i think we need to talk about spark at the next meeting, because the sahara codebase references plugins for 0.9.1 and 1.0.0 of spark. | 18:52 |
| elmiko | and we are now introducing a patch that will create spark 1.3 images | 18:52 |
| elmiko | https://review.openstack.org/#/c/195054 | 18:52 |
| elmiko | but we have no way to generate older spark images, and i'm not sure we even want to | 18:52 |
| elmiko | regardless though, we should update the plugin version in sahara | 18:53 |
| elmiko | SergeyLukjanov: ^^ | 18:53 |
| elmiko | i'm going to add this to the agenda for next time | 18:53 |
| elmiko | egafford, crobertsrh, ^^ curious about your thoughts as well | 18:56 |
| *** Nikolay_St has quit IRC | 18:56 | |
| egafford | elmiko: I think it makes sense to stay current and deprecate older plugin versions fairly aggressively while we're in an adoption phase, certainly. | 18:58 |
| elmiko | cool, i guess we just need to be agreesive about this | 18:59 |
| elmiko | although for spark things are easy so far since everything just uses spark-submit | 18:59 |
| egafford | elmiko: It'll only get harder; might as well go for it. Well, I don't know that we need to very much, but I think it helps us a lot more than it hurts our users right now, and that'll change, so the time is now. Yup; thanks Spark! | 19:00 |
| elmiko | egafford: i'm just wondering if i should make a bug to deprecate 0.9.1, and then followup with 1.0.0 once this patch goes through? | 19:01 |
| elmiko | or will folks want a spec to talk about how we handle spark versions? | 19:01 |
| egafford | (Like with the vanilla 1 case. Good to deprecate now before enough people use us that at least one of them is crazy enough to be on Hadoop 1 still.) 1.0.0 is a judgment call; I'm not deep enough in Spark to make it. | 19:01 |
| egafford | 0.9.1 definitely, I think. | 19:02 |
| elmiko | from what i could tell from spark summit, 1.0.0 is ancient... | 19:02 |
| elmiko | like, anyone who is serious about spark is going to want the latest version | 19:02 |
| egafford | Sure; I wonder if that's a function of the audience or representative, but point taken. | 19:02 |
| elmiko | yea | 19:02 |
| elmiko | that's why i added it to the agenda for next week | 19:02 |
| egafford | Right, but this kinda gets to the "it's not big data unless it's 10,000 cores" argument. | 19:02 |
| elmiko | well... | 19:03 |
| egafford | I mean, sure, that's really big data, but folks use the tools successfully at much smaller scale and further from the bleeding edge. | 19:03 |
| elmiko | the more recent versions add some really nice optimizations and end user features | 19:03 |
| elmiko | totally agree about size argument | 19:03 |
| egafford | Sure; and I'm totally for deprecating aggressively on general principle at this stage. I just don't have the context to judge on 1.0.0, but I also am a little bit skeptical of assuming version usage patterns match Spark Summit. | 19:05 |
| elmiko | good point | 19:05 |
| egafford | You're almost certainly right; you have more context. Just makin' arguments. | 19:05 |
| elmiko | no, it's a good argument | 19:05 |
| elmiko | it looked to me like the version differences in spark were pretty different from how things are in hadoop world | 19:06 |
| elmiko | best to talk with willb and rnowling next week about it too | 19:06 |
| egafford | elmiko: Totally. | 19:06 |
| elmiko | from my understanding, a 1.0.0 app will still run on 1.4.0 | 19:06 |
| elmiko | you just get all sort of nice under-the-hood optos | 19:07 |
| elmiko | then again, maybe i just have spark madness </ren hoek voice> | 19:07 |
| egafford | You have had this Spark cluster since you were a child! | 19:08 |
| elmiko | it's soo steeenky | 19:08 |
| egafford | And... scene. | 19:08 |
| elmiko | sorry, i loved that episode lol | 19:08 |
| egafford | Oh, it was completely fabulous. | 19:08 |
| elmiko | http://i.imgur.com/mRISGFa.jpg | 19:09 |
| elmiko | should be our new logo for sahara | 19:09 |
| egafford | A few thoughts: 1) copyright infringement. 2) relevance. 3) kinda sends the wrong message about user experience. | 19:10 |
| egafford | Beyond that, I'm strongly for. | 19:10 |
| elmiko | LOL | 19:10 |
| elmiko | man, why do you always have to bring reality into these discussions... ;) | 19:10 |
| egafford | I was always a big fan of Mr. Horse. | 19:11 |
| elmiko | oh yea | 19:11 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-sahara: acceptance: install RDO repo the right way https://review.openstack.org/196217 | 19:52 |
| *** hdd has joined #openstack-sahara | 19:59 | |
| *** Longgeek has joined #openstack-sahara | 20:04 | |
| *** hdd has quit IRC | 20:05 | |
| *** Longgeek has quit IRC | 20:09 | |
| *** hdd has joined #openstack-sahara | 20:13 | |
| *** hdd has quit IRC | 20:18 | |
| *** sgotliv has joined #openstack-sahara | 20:52 | |
| *** raildo has quit IRC | 21:06 | |
| *** htruta has quit IRC | 21:13 | |
| *** hdd has joined #openstack-sahara | 21:15 | |
| *** Longgeek has joined #openstack-sahara | 21:22 | |
| *** openstack has joined #openstack-sahara | 21:24 | |
| *** Longgeek has quit IRC | 21:26 | |
| openstackgerrit | Merged openstack/puppet-sahara: acceptance: install RDO repo the right way https://review.openstack.org/196217 | 21:48 |
| *** htruta has joined #openstack-sahara | 21:50 | |
| *** hdd has quit IRC | 21:54 | |
| *** Nikolay_St has joined #openstack-sahara | 22:18 | |
| *** htruta has quit IRC | 22:19 | |
| openstackgerrit | Merged openstack/sahara: Updated from global requirements https://review.openstack.org/195122 | 22:34 |
| *** Nikolay_St has quit IRC | 22:37 | |
| *** saneax has joined #openstack-sahara | 22:47 | |
| *** egafford has quit IRC | 22:58 | |
| *** ekarlso has quit IRC | 23:18 | |
| *** ekarlso has joined #openstack-sahara | 23:18 | |
| *** Longgeek has joined #openstack-sahara | 23:22 | |
| *** Longgeek has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
