| *** Longgeek has quit IRC | 00:03 | |
| *** ViswaV has quit IRC | 00:55 | |
| *** ViswaV has joined #openstack-sahara | 00:56 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| *** witlessb has quit IRC | 02:01 | |
| *** chlong_ has joined #openstack-sahara | 02:08 | |
| *** chlong__ has joined #openstack-sahara | 02:09 | |
| *** chlong_ has quit IRC | 02:13 | |
| openstackgerrit | lu huichun proposed openstack/sahara: [EDP] Add scheduling EDP jobs in sahara(oozie engine implementation) https://review.openstack.org/182310 | 02:16 |
|---|---|---|
| *** chlong_ has joined #openstack-sahara | 02:28 | |
| *** chlong__ has quit IRC | 02:31 | |
| *** chlong has joined #openstack-sahara | 02:31 | |
| *** chlong_ has quit IRC | 02:34 | |
| openstackgerrit | Ken Chen proposed openstack/sahara-specs: Add HDFS HA support in Cloudera plugin https://review.openstack.org/196929 | 02:37 |
| *** Longgeek has joined #openstack-sahara | 03:00 | |
| *** sgotliv has joined #openstack-sahara | 03:04 | |
| *** Longgeek has quit IRC | 03:05 | |
| *** Longgeek has joined #openstack-sahara | 03:37 | |
| *** ViswaV has quit IRC | 03:44 | |
| *** ViswaV has joined #openstack-sahara | 03:45 | |
| *** saneax has quit IRC | 03:49 | |
| *** Longgeek has quit IRC | 04:05 | |
| *** Poornima has joined #openstack-sahara | 04:06 | |
| *** Longgeek has joined #openstack-sahara | 04:06 | |
| *** coolsvap|away is now known as coolsvap | 04:08 | |
| *** Poornima has quit IRC | 04:11 | |
| *** hogepodge has quit IRC | 04:32 | |
| *** hogepodge has joined #openstack-sahara | 04:32 | |
| *** Poornima has joined #openstack-sahara | 04:38 | |
| *** saneax has joined #openstack-sahara | 04:44 | |
| *** hogepodge has quit IRC | 05:42 | |
| *** hogepodge has joined #openstack-sahara | 05:42 | |
| openstackgerrit | Ken Chen proposed openstack/sahara-specs: Add HDFS HA support in Cloudera plugin https://review.openstack.org/196929 | 05:43 |
| *** nkrinner has joined #openstack-sahara | 05:44 | |
| openstackgerrit | Ken Chen proposed openstack/sahara-specs: Add HDFS HA support in Cloudera plugin https://review.openstack.org/196929 | 05:53 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Minor - move definition to avoid AttributeError https://review.openstack.org/192223 | 06:48 |
| *** sgotliv has quit IRC | 06:51 | |
| *** hdd has quit IRC | 07:11 | |
| *** pino|work has joined #openstack-sahara | 07:12 | |
| *** saneax has quit IRC | 07:22 | |
| *** witlessb has joined #openstack-sahara | 07:29 | |
| *** saneax has joined #openstack-sahara | 07:33 | |
| *** chlong has quit IRC | 07:35 | |
| *** Poornima has quit IRC | 08:02 | |
| *** saneax has quit IRC | 08:28 | |
| *** saneax has joined #openstack-sahara | 08:29 | |
| *** venza has joined #openstack-sahara | 08:43 | |
| openstackgerrit | lu huichun proposed openstack/sahara: [EDP] edp job will not be deleted if raise exception after run job when running edp jobs, if raise exceptions after the job run or add job request has been sent to the specific job engine. the edp job still run in the specific job engine like oozie. https://review.openstack.org/196981 | 08:45 |
| *** Poornima has joined #openstack-sahara | 08:45 | |
| venza | hello | 08:47 |
| venza | how do I go about adding CI test for Spark 1.3? | 08:47 |
| venza | see here: https://review.openstack.org/#/c/196686/ | 08:47 |
| -openstackstatus- NOTICE: OpenStack CI is down due to hard drive failures | 08:49 | |
| *** ChanServ changes topic to "OpenStack CI is down due to hard drive failures" | 08:49 | |
| *** sgotliv has joined #openstack-sahara | 08:54 | |
| SergeyLukjanov | venza, ping degorenko | 09:04 |
| SergeyLukjanov | venza, hi | 09:04 |
| SergeyLukjanov | elmiko, ++ to upgrade spark plugin to 1.3 | 09:05 |
| SergeyLukjanov | elmiko, in addition to it it'll be great to have all its functinality in both HDP and CDH plugin | 09:06 |
| *** sgotliv_ has joined #openstack-sahara | 09:06 | |
| *** sgotliv has quit IRC | 09:09 | |
| degorenko | venza, you should add scenario file to sahara repo and add job and case for spark 1.3 here https://github.com/stackforge/sahara-ci-config/ | 09:10 |
| venza | ok thanks | 09:11 |
| degorenko | if you have any questions you can ask me | 09:12 |
| venza | degorenko: should I also change sahara/tests/integration/configs/config.py and sahara/tests/unit/plugins/spark/test_plugin.py ? | 09:18 |
| venza | there are some references to spark 1.0.0 in there | 09:18 |
| degorenko | in sahara repo you should change all tests files, which need for run tests, and add new scenario to directory etc/scenario/sahara-ci/spark-1.3.0 for example. After that you can upload change to sahara-ci-config, add me to reviewers :) that's it | 09:21 |
| degorenko | venza, ^^ | 09:26 |
| venza | thanks! will do | 09:29 |
| *** tosky has joined #openstack-sahara | 09:32 | |
| *** sgotliv_ has quit IRC | 09:39 | |
| *** sgotliv_ has joined #openstack-sahara | 09:54 | |
| *** saneax has quit IRC | 10:06 | |
| *** saneax has joined #openstack-sahara | 10:06 | |
| *** coolsvap is now known as coolsvap|away | 10:14 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [CDH] Provide ability to configure gateway configs https://review.openstack.org/191830 | 10:40 |
| *** Poornima has quit IRC | 10:45 | |
| *** chlong has joined #openstack-sahara | 10:49 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] New version of HDP plugin https://review.openstack.org/184292 | 10:54 |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] New version of HDP plugin https://review.openstack.org/184292 | 11:12 |
| openstackgerrit | Timur Nurlygayanov proposed openstack/puppet-sahara: Fixed issue with rabbit_ha_queues parameter https://review.openstack.org/197023 | 11:17 |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] New version of HDP plugin https://review.openstack.org/184292 | 11:17 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara: [WIP] Draft support for SIGHUP https://review.openstack.org/197033 | 11:58 |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: [WIP] Add cluster validation https://review.openstack.org/196713 | 12:13 |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding is_public and is_protected fields support https://review.openstack.org/195065 | 12:21 |
| openstackgerrit | lu huichun proposed openstack/sahara: [EDP] Delete edp job if raise exception https://review.openstack.org/196981 | 12:33 |
| tellesnobrega | SergeyLukjanov, can you send me the link for the saharaclient specs etherpad? | 12:55 |
| SergeyLukjanov | tellesnobrega, https://etherpad.openstack.org/p/sahara-liberty-client | 12:55 |
| tellesnobrega | SergeyLukjanov, thanks | 12:55 |
| *** saneax has quit IRC | 13:08 | |
| elmiko | SergeyLukjanov: yea, i think we should talk about it at the meeting this week. i added it to the agenda | 13:14 |
| *** htruta has joined #openstack-sahara | 13:35 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] Add base services support for HDP 2.2 https://review.openstack.org/185100 | 13:39 |
| openstackgerrit | Merged openstack/sahara: Enable Java Keystore KMS service in CDH5.4 https://review.openstack.org/189373 | 13:44 |
| openstackgerrit | Merged openstack/sahara: Minor - move definition to avoid AttributeError https://review.openstack.org/192223 | 13:47 |
| openstackgerrit | Daniele Venzano proposed openstack/sahara: Support Spark 1.3.x https://review.openstack.org/196686 | 13:55 |
| pino|work | venza: are you daniele venzano? | 14:16 |
| venza | pino|work: yes it's me | 14:16 |
| pino|work | venza: hi! just note (regarding your recheck) that the CI is broken (see also /topic) | 14:17 |
| venza | ah ok, didn't notice | 14:17 |
| * elmiko waves at venza | 14:19 | |
| venza | elmiko: hi! | 14:19 |
| elmiko | seems like SergeyLukjanov is good with talking about updating the spark version in sahara | 14:20 |
| venza | there is version 0.9.1 that is old and no one should be using it, so support should be dropped from sahara | 14:22 |
| venza | I don't know the policy about that | 14:22 |
| elmiko | agreed, drop 0.9.1, add 1.3.1 | 14:23 |
| elmiko | it probably won't be a big deal, i agree with your assessment that the plugin will probably work with any version of spark at this point | 14:24 |
| tosky | didn't we discuss about one cycle for deprecation, then removal? | 14:24 |
| tosky | at least in the past | 14:24 |
| *** egafford has joined #openstack-sahara | 14:24 | |
| tosky | even if deprecated became "you can still run your cluster and keep your node group/cluster templates, but not create new ones" at least for vanilla :) | 14:25 |
| elmiko | good point tosky, this is why i added it to the agenda for the meeting =) | 14:26 |
| elmiko | although i can't imagine there will be much complaint in dropping 0.9.1 | 14:26 |
| tosky | probably not, but still it was a plugin enabled by default, so better discuss it first | 14:27 |
| elmiko | yup | 14:27 |
| elmiko | and if we want to take a cycle to deprecate, i don't have an objection, but spark is moving very fast currently | 14:27 |
| venza | yes, they are at version 1.4, but we didn't have time to test it yet | 14:29 |
| elmiko | i've been playing with a local 1.4 image, but it's just based on a fedora impl, no cdh or hdp | 14:30 |
| venza | I will try to make it to the 14h meeting | 14:31 |
| elmiko | cool | 14:31 |
| *** hdd has joined #openstack-sahara | 14:36 | |
| openstackgerrit | Daniele Venzano proposed stackforge/sahara-ci-config: Rename Spark tests to use 1.3.x version https://review.openstack.org/197114 | 14:36 |
| *** Networkn3rd has joined #openstack-sahara | 14:50 | |
| *** sgotliv has joined #openstack-sahara | 14:53 | |
| *** ChanServ changes topic to "launchpad openid is down. login to openstack services will fail until launchpad openid is happy again" | 14:55 | |
| -openstackstatus- NOTICE: The log volume was repaired and brought back online at 14:00 UTC. Log links today from before that time may be missing, and changes should be rechecked if fresh job logs are desired for them. | 14:55 | |
| *** sgotliv_ has quit IRC | 14:55 | |
| *** nkrinner has quit IRC | 15:00 | |
| *** saneax has joined #openstack-sahara | 15:04 | |
| *** ViswaV_ has joined #openstack-sahara | 15:40 | |
| *** ViswaV has quit IRC | 15:41 | |
| *** tosky has quit IRC | 15:58 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] Add base services support for HDP 2.2 https://review.openstack.org/185100 | 16:00 |
| *** tosky has joined #openstack-sahara | 16:01 | |
| *** sgotliv has quit IRC | 16:07 | |
| *** ViswaV_ has quit IRC | 16:08 | |
| *** ViswaV has joined #openstack-sahara | 16:09 | |
| *** ViswaV has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack/sahara: Fix compatible issues in unit tests for python 3 https://review.openstack.org/188323 | 16:24 |
| *** dontalton has joined #openstack-sahara | 16:29 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: [WIP] Add EDP services to new HDP plugin https://review.openstack.org/194580 | 16:32 |
| *** openstackgerrit has quit IRC | 16:50 | |
| *** openstackgerrit has joined #openstack-sahara | 16:51 | |
| *** ViswaV has joined #openstack-sahara | 17:07 | |
| *** htruta_ has joined #openstack-sahara | 17:25 | |
| *** chlong has quit IRC | 17:56 | |
| *** ViswaV_ has joined #openstack-sahara | 18:01 | |
| *** ViswaV has quit IRC | 18:01 | |
| *** ViswaV_ has quit IRC | 18:05 | |
| *** ViswaV has joined #openstack-sahara | 18:06 | |
| *** Networkn3rd has quit IRC | 18:09 | |
| *** tosky has quit IRC | 18:14 | |
| *** tellesnobrega_ has joined #openstack-sahara | 18:15 | |
| tellesnobrega_ | how often does sahara checks if a hadoop job is finished? or it gets an signal that it ended? | 18:16 |
| tellesnobrega_ | SergeyLukjanov, ^ | 18:17 |
| *** tellesnobrega_ has quit IRC | 18:20 | |
| *** Networkn3rd has joined #openstack-sahara | 18:25 | |
| *** htruta_ has quit IRC | 18:26 | |
| *** sgotliv has joined #openstack-sahara | 18:30 | |
| *** Longgeek has quit IRC | 18:57 | |
| *** Longgeek has joined #openstack-sahara | 18:58 | |
| openstackgerrit | Chad Roberts proposed openstack/sahara: Allowing job binary objects to be updated https://review.openstack.org/190250 | 19:18 |
| SergeyLukjanov | tellesnobrega, it's a periodic job | 19:24 |
| SergeyLukjanov | tellesnobrega, https://github.com/openstack/sahara/blob/master/sahara/service/periodic.py#L38 | 19:25 |
| *** tosky has joined #openstack-sahara | 19:29 | |
| tellesnobrega | SergeyLukjanov, thanks | 19:42 |
| tellesnobrega | htruta, ^ | 19:42 |
| htruta | SergeyLukjanov, tellesnobrega cool | 19:43 |
| htruta | so, this 'periodic_interval_max' is general for every kind of job? | 19:44 |
| htruta | isn't there a specific 'periodic_interval_max_hadoop_vanilla_xpto' | 19:44 |
| htruta | ? | 19:44 |
| elmiko | htruta: no, just the one | 19:47 |
| htruta | elmiko: ok. so, the sahara job_finish property says when sahara sees that the job has finished | 19:48 |
| htruta | and not when the job was actually finished | 19:48 |
| elmiko | htruta: yea, that could be. and based on the interval max it should be no longer than 1 minute after the job finished. but, | 19:49 |
| elmiko | i'm not sure if we pull job info out of the information returned from the cluster. | 19:49 |
| elmiko | doesn't oozie provide timing info about jobs? | 19:49 |
| tellesnobrega | elmiko, probably does.. htruta is looking for a more automated way to get the actual end time (experiments and stuff) | 19:51 |
| elmiko | yea, i figured as much. | 19:52 |
| tellesnobrega | egafford, do you mind taking a look at this: https://review.openstack.org/#/c/193586/2 | 19:52 |
| htruta | elmiko, it does... does it have an api? | 19:52 |
| elmiko | htruta: you might want to look at https://github.com/openstack/sahara/blob/master/sahara/service/edp/job_manager.py | 19:52 |
| elmiko | look at L59 and :132 | 19:53 |
| elmiko | er L132 | 19:53 |
| elmiko | looks like we just stamp the end_time when sahara notices that the job has ended | 19:53 |
| elmiko | htruta: also, https://github.com/openstack/sahara/blob/master/sahara/service/engine.py#L201 | 19:54 |
| elmiko | i think those are the only times when sahara updates the job execution model for the end time | 19:55 |
| htruta | elmiko: I see | 20:00 |
| htruta | so, it's made at edp, not plugin level... | 20:00 |
| elmiko | looks like it | 20:00 |
| htruta | I'll see what I can do with oozie directly | 20:00 |
| elmiko | probably not very exacting, but within a minute or two i would imagine | 20:00 |
| htruta | anyway, the maximum of 60 seconds wont' hurt me | 20:00 |
| elmiko | cool | 20:01 |
| htruta | elmiko: thanks | 20:01 |
| elmiko | htruta: np, gl! | 20:01 |
| egafford | tellesnobrega: Looking now. :) | 20:27 |
| tellesnobrega | egafford, thanks | 20:27 |
| *** pino|work has quit IRC | 20:48 | |
| openstackgerrit | Sergey Gotliv proposed openstack/sahara: Modify launch_command to support global variables https://review.openstack.org/197315 | 20:57 |
| egafford | tellesnobrega: Reviewed. | 20:57 |
| tellesnobrega | egafford, thanks | 20:57 |
| tellesnobrega | egafford, im gonna work on the suggestions tomorrow | 21:01 |
| egafford | tellesnobrega: Sensible! | 21:01 |
| tellesnobrega | its 18h and I'm going to play some soccer right now lol | 21:02 |
| elmiko | nice | 21:03 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-sahara: Use beaker-puppet_install_helper to install puppet https://review.openstack.org/197327 | 21:03 |
| openstackgerrit | Sergey Gotliv proposed openstack/sahara: Adding support for the Spark Shell job https://review.openstack.org/197333 | 21:12 |
| tosky | oh, sgotliv | 21:12 |
| *** egafford has quit IRC | 21:42 | |
| *** witlessb has quit IRC | 21:48 | |
| *** sreshetnyak has quit IRC | 21:49 | |
| *** AndreyPavlov has quit IRC | 21:49 | |
| *** degorenko has quit IRC | 21:49 | |
| *** ylobankov has quit IRC | 21:49 | |
| *** hdd has quit IRC | 22:06 | |
| *** dontalton has quit IRC | 22:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/197362 | 22:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements https://review.openstack.org/197284 | 22:46 |
| *** Networkn3rd has quit IRC | 23:44 | |
| *** tosky has quit IRC | 23:45 | |
| *** htruta_ has joined #openstack-sahara | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!