| *** alazarev has quit IRC | 00:00 | |
| *** Networkn3rd has quit IRC | 00:04 | |
| *** alazarev has joined #openstack-sahara | 00:12 | |
| *** ViswaV has joined #openstack-sahara | 00:27 | |
| *** Networkn3rd has joined #openstack-sahara | 00:36 | |
| *** tnovacik has quit IRC | 00:41 | |
| *** ViswaV has quit IRC | 00:52 | |
| *** alazarev has quit IRC | 00:59 | |
| *** alazarev has joined #openstack-sahara | 01:29 | |
| *** hochi has joined #openstack-sahara | 01:31 | |
| *** bradd1 has quit IRC | 01:55 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| *** bradd1 has joined #openstack-sahara | 02:00 | |
| *** witlessb has quit IRC | 02:01 | |
| *** openstackgerrit has quit IRC | 02:02 | |
| *** openstackgerrit has joined #openstack-sahara | 02:02 | |
| *** alazarev has quit IRC | 02:07 | |
| *** Networkn3rd has quit IRC | 02:28 | |
| *** Networkn3rd has joined #openstack-sahara | 02:29 | |
| *** alazarev has joined #openstack-sahara | 02:36 | |
| *** tmckay_bbl has quit IRC | 03:17 | |
| *** _mattf is now known as mattf | 03:30 | |
| *** Longgeek has joined #openstack-sahara | 03:31 | |
| *** alazarev has quit IRC | 03:36 | |
| *** alazarev has joined #openstack-sahara | 04:09 | |
| openstackgerrit | A change was merged to openstack/sahara: Fix some of tests that rely on hash ordering https://review.openstack.org/115132 | 04:14 |
|---|---|---|
| openstackgerrit | A change was merged to openstack/sahara: Make Vanilla 2.3.0 plugin deprecated https://review.openstack.org/115563 | 04:18 |
| openstackgerrit | A change was merged to openstack/sahara: Correction of words decoMMiSSion-decoMMiSSioning https://review.openstack.org/115232 | 04:18 |
| openstackgerrit | A change was merged to openstack/sahara: Removed one round trip to server for HDFS put https://review.openstack.org/115120 | 04:18 |
| *** bradd1 has quit IRC | 04:26 | |
| *** Longgeek_ has joined #openstack-sahara | 04:52 | |
| *** Longgeek has quit IRC | 04:56 | |
| *** ViswaV has joined #openstack-sahara | 04:57 | |
| *** ViswaV_ has joined #openstack-sahara | 04:58 | |
| *** ViswaV has quit IRC | 05:01 | |
| *** alazarev has quit IRC | 05:02 | |
| *** ghenriks has quit IRC | 05:22 | |
| *** ghenriks has joined #openstack-sahara | 05:24 | |
| *** mikehn_ has quit IRC | 05:32 | |
| *** ViswaV_ has quit IRC | 05:37 | |
| *** mikehn has joined #openstack-sahara | 05:39 | |
| *** alazarev has joined #openstack-sahara | 05:55 | |
| *** alazarev has quit IRC | 05:55 | |
| *** alazarev has joined #openstack-sahara | 05:56 | |
| *** k4n0 has joined #openstack-sahara | 06:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/117429 | 06:10 |
| *** akuznetsov has quit IRC | 06:12 | |
| *** tnovacik has joined #openstack-sahara | 06:14 | |
| *** alazarev has quit IRC | 06:24 | |
| *** akuznetsov has joined #openstack-sahara | 07:10 | |
| *** witlessb has joined #openstack-sahara | 07:20 | |
| *** witlessb has quit IRC | 07:25 | |
| *** tnovacik has quit IRC | 07:45 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 07:55 | |
| *** julienvey has joined #openstack-sahara | 08:07 | |
| *** witlessb has joined #openstack-sahara | 08:17 | |
| *** Networkn3rd has quit IRC | 08:20 | |
| *** boris-42 has quit IRC | 08:21 | |
| *** skolekonov has joined #openstack-sahara | 08:25 | |
| *** boris-42 has joined #openstack-sahara | 08:31 | |
| *** Timotey has joined #openstack-sahara | 08:53 | |
| *** stannie has joined #openstack-sahara | 09:53 | |
| *** tosky has joined #openstack-sahara | 10:01 | |
| *** hochi has quit IRC | 10:07 | |
| *** tmckay has joined #openstack-sahara | 11:45 | |
| *** tmckay has left #openstack-sahara | 11:46 | |
| *** tosky has quit IRC | 12:01 | |
| *** tmckay has joined #openstack-sahara | 12:36 | |
| *** _elmiko is now known as elmiko | 12:46 | |
| *** k4n0 has quit IRC | 13:10 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Install non deprecated DB for Cloudera Manager https://review.openstack.org/117501 | 13:10 |
| *** sballe_ has quit IRC | 13:16 | |
| *** sballe_ has joined #openstack-sahara | 13:16 | |
| *** tmckay has quit IRC | 13:17 | |
| *** tosky has joined #openstack-sahara | 13:20 | |
| *** bradd1 has joined #openstack-sahara | 13:25 | |
| openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: Add notifications to Sahara https://review.openstack.org/108982 | 13:41 |
| *** tosky_ has joined #openstack-sahara | 13:46 | |
| *** tosky has quit IRC | 13:50 | |
| *** bradd1 has quit IRC | 13:53 | |
| *** tnovacik has joined #openstack-sahara | 14:00 | |
| *** tosky_ has quit IRC | 14:11 | |
| *** tosky_ has joined #openstack-sahara | 14:11 | |
| *** tmckay has joined #openstack-sahara | 14:11 | |
| *** tmckay has quit IRC | 14:11 | |
| openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: [HDP1.3.2] Fixed bug with decommissioning cluster https://review.openstack.org/114187 | 14:15 |
| *** tmckay has joined #openstack-sahara | 14:15 | |
| *** Longgeek_ has quit IRC | 14:24 | |
| *** alazarev has joined #openstack-sahara | 14:27 | |
| *** tnovacik has quit IRC | 14:36 | |
| *** bradd1 has joined #openstack-sahara | 14:38 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added several missing translations https://review.openstack.org/117034 | 14:51 |
| tmckay | aignatov, alazarev, dmitryme, SergeyLukjanov, mattf, I summarized my spark/swift investigation at the top of https://etherpad.openstack.org/p/sahara_spark_edp | 14:55 |
| tmckay | Nutshell: it works, we have proof of concept. But, there are issues to solve. Probably not a good idea to push into Juno | 14:56 |
| alazarev | tmckay: reading | 14:56 |
| tmckay | I'll send a link to the dev mail too | 14:56 |
| aignatov | reading | 14:56 |
| *** skolekonov has quit IRC | 14:58 | |
| tmckay | elmiko, forgot you ^^ | 15:00 |
| elmiko | tmckay: no worries, i got to experience most of it live ;) | 15:00 |
| tmckay | elmiko, heh | 15:00 |
| tmckay | but you didn't get it in one nice summary with logical steps | 15:01 |
| *** balles has joined #openstack-sahara | 15:01 | |
| elmiko | true | 15:01 |
| *** balles has quit IRC | 15:01 | |
| * tmckay gets snacks, brb | 15:01 | |
| tmckay | I wanted to get this up before the meeting today | 15:02 |
| elmiko | good thought | 15:03 |
| *** alazarev has quit IRC | 15:11 | |
| *** alazarev has joined #openstack-sahara | 15:24 | |
| *** Networkn3rd has joined #openstack-sahara | 15:29 | |
| *** IvanBerezovskiy has left #openstack-sahara | 15:33 | |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Updating JobBinaries to use proxy for Swift access https://review.openstack.org/117052 | 16:09 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding trust delegation and removal for proxy users https://review.openstack.org/116616 | 16:09 |
| tmckay | elmiko, comments on https://review.openstack.org/#/c/116426/. It would be cool if we could skip the extra db write (but it might be less pretty) | 16:28 |
| elmiko | tmckay: yea, i had actually thought about that. but figured i put it up and see what others thought. | 16:29 |
| tmckay | elmiko, it's minor given the scope, but the update dict could just be merged | 16:30 |
| elmiko | agreed, i'll take another pass at it | 16:30 |
| elmiko | tmckay: my thoughts on the updates for the job execution are that since this only happens during the beginning of a job execution, or when the status is updated, i figured these would be low frequency updates and it makes the code more self-contained. | 16:36 |
| tmckay | elmiko, I could agree with that. | 16:36 |
| elmiko | tmckay: in execute_job, that call to create_proxy_user_for_job_execution actually does an update, i could make the function return an update dict, but i think it might become more difficult to reuse in the future. | 16:37 |
| tmckay | elmiko, don't let me distract you from the hadoop plugin. I'll take a quick crack at it and see if I can make something pretty, intuitive, and sensible | 16:37 |
| tmckay | I won't -1 for it | 16:37 |
| elmiko | in general, i agree with you. i wouldn't have made so many update calls if i thought it would happen with any sort of high frequency | 16:38 |
| tmckay | elmiko, ack, I had the same thoughts, hence the fence | 16:38 |
| elmiko | yea | 16:38 |
| elmiko | now i'm more worried about having to update the hadoop plugin to v3 keystone | 16:38 |
| elmiko | that could become a serious boon-doggle | 16:38 |
| tmckay | doh | 16:39 |
| elmiko | i want to bring up the idea of decoupling the sahara implementation from the plugin | 16:39 |
| elmiko | currently, the proxy domain stuff is gated by a boolean conf option | 16:39 |
| elmiko | and i'm wondering if we could upgrade the hadoop component outside the openstack release schedule | 16:40 |
| tmckay | interesting. | 16:40 |
| elmiko | that way, we could get the proxy auth stuff in for J, and release the hadoop part later | 16:40 |
| elmiko | i don't think sahara-extras has to follow openstack release sched, is that true? | 16:41 |
| tmckay | elmiko, so the sahara stuff would be in there, but dormant? | 16:41 |
| elmiko | right, until we get the hadoop part working | 16:41 |
| elmiko | it's all optional anyways, by default sahara won't use the proxy user | 16:41 |
| tmckay | elmiko, I think we could probably argue that for the freeze. It's a hadoop extension, after all, not properly part of openstack | 16:42 |
| elmiko | right | 16:42 |
| tmckay | The images pull outside content anyway | 16:42 |
| tmckay | ++ | 16:42 |
| elmiko | i'm looking at sahara-extras and it looks like we have been following the openstack release sched, so i dunno. need to bring it up today | 16:42 |
| elmiko | mattf: do you have any thoughts on decoupling the sahara proxy implementation from the hadoop plugin, in terms of release timing? | 16:43 |
| tmckay | elmiko, well, even so. Even if we officially had a dormant feature in the release, and then updated hadoop for k1, it makes sense to me. And we could probably do it sooner. | 16:44 |
| elmiko | yea | 16:44 |
| tmckay | in my mind, it's equivalent to just using a different image | 16:44 |
| elmiko | it might even be done by summit, just not done for freeze | 16:44 |
| tmckay | an image that supports a particular feature | 16:44 |
| tmckay | right | 16:44 |
| elmiko | an admin is going to need to update more than a few pieces to make this security upgrade happen anyways | 16:45 |
| tmckay | elmiko, does the proxy user stuff guarantee that user/pass/trustid are all there if proxy_configs is there at all? | 16:53 |
| elmiko | tmckay: yes, unless there is an error creating the user or the trust | 16:54 |
| elmiko | and when the proxy user is deleted, proxy_configs is removed from the job_execution.job_configs, so i don't think there would be an empty dict there | 16:54 |
| elmiko | worst case scenario would be that the user is created but the trust fails, then there would be no proxy_trust_id, but i don't think that would cause a problem(aside from the general error of the trust creation failing)( | 16:56 |
| tmckay | well, the field will still be there. Right there in create_proxy_user_for_job_execution, my bad | 16:57 |
| *** tosky_ is now known as tosky | 17:01 | |
| *** ViswaV has joined #openstack-sahara | 17:06 | |
| *** ViswaV_ has joined #openstack-sahara | 17:06 | |
| *** bradd1 has quit IRC | 17:09 | |
| *** ViswaV has quit IRC | 17:11 | |
| mattf | elmiko, ask me again later | 17:11 |
| elmiko | mattf: ack | 17:13 |
| *** tosky has quit IRC | 17:30 | |
| *** bradd1 has joined #openstack-sahara | 17:33 | |
| *** shakamunyi has joined #openstack-sahara | 17:40 | |
| *** bradd1 has quit IRC | 17:48 | |
| *** tosky has joined #openstack-sahara | 17:48 | |
| *** ylobankov has joined #openstack-sahara | 17:52 | |
| SergeyLukjanov | tmckay, thank you for summarizing spark/swit stuff | 17:56 |
| tmckay | SergeyLukjanov, sure | 17:58 |
| tmckay | SergeyLukjanov, interesting, if we use a spark assembly built with hadoop-openstack, then the extra --jars arg is not needed. Brings up the question again wether to carry the swift plugin separately or not | 18:00 |
| tmckay | maybe when the swift plugin is stable, we can merge to hadoop again | 18:00 |
| SergeyLukjanov | elmiko, tmckay, for sahara-extra it's not a problem to merge something after the j3 | 18:00 |
| tmckay | (I *think* that is the case, I have not tried a custom spark build) | 18:01 |
| tmckay | that is good news! | 18:01 |
| tmckay | elmiko, so, if we have to, we can ship with default=off | 18:01 |
| tmckay | which I guess is the case anyway | 18:01 |
| SergeyLukjanov | tmckay, yup, it'll be great to merge into hadoop again | 18:02 |
| elmiko | tmckay: default is off, regardless | 18:02 |
| tmckay | I think I need to take a look at how to handle options to spark-submit, I think we will need it for flexibility. And, we probably need to be able to set spark-default.conf in the provisioning plugin too | 18:03 |
| tmckay | oo. meeting time | 18:04 |
| elmiko | tmckay: also, check out that patch to spark that g | 18:04 |
| elmiko | Gil linked in email | 18:04 |
| tmckay | elmiko, yeah, essentially the same stuff, minus the detail that arises from 1) deploy from sahara and 2) carry the plugin outside of the spark assembly | 18:05 |
| elmiko | yea | 18:05 |
| elmiko | too bad they haven't thought about v3 keystone | 18:05 |
| *** mattf is now known as _mattf | 18:17 | |
| *** julienvey has quit IRC | 18:28 | |
| *** julienvey has joined #openstack-sahara | 18:32 | |
| *** tmckay is now known as tmckay_bbl | 18:33 | |
| *** ylobankov has left #openstack-sahara | 18:34 | |
| *** julienve_ has joined #openstack-sahara | 18:35 | |
| *** julienvey has quit IRC | 18:35 | |
| *** alazarev has quit IRC | 18:59 | |
| *** bradd1 has joined #openstack-sahara | 19:04 | |
| *** alazarev has joined #openstack-sahara | 19:14 | |
| *** balles has joined #openstack-sahara | 19:15 | |
| *** balles has quit IRC | 19:18 | |
| *** sballe has joined #openstack-sahara | 19:18 | |
| *** openstackgerrit has quit IRC | 19:31 | |
| *** openstackgerrit has joined #openstack-sahara | 19:32 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara-specs: Persist sahara configuration in cluster object https://review.openstack.org/117011 | 19:54 |
| *** sballe has quit IRC | 20:14 | |
| *** tmckay_bbl is now known as tmckay | 20:28 | |
| *** julienve_ has quit IRC | 20:39 | |
| *** julienvey has joined #openstack-sahara | 20:45 | |
| openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/117429 | 20:53 |
| *** tmckay has quit IRC | 21:10 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added information about sahara settings to cluster https://review.openstack.org/117029 | 21:16 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Switched anti-affinity feature to server groups https://review.openstack.org/112159 | 21:22 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Updating JobBinaries to use proxy for Swift access https://review.openstack.org/117052 | 21:32 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding trust delegation and removal for proxy users https://review.openstack.org/116616 | 21:32 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding proxy user creation per job execution https://review.openstack.org/116426 | 21:33 |
| *** ViswaV has joined #openstack-sahara | 21:36 | |
| *** ViswaV_ has quit IRC | 21:36 | |
| *** Networkn3rd has quit IRC | 21:50 | |
| *** Networkn3rd has joined #openstack-sahara | 22:00 | |
| *** stannie has quit IRC | 22:39 | |
| *** julienvey has quit IRC | 22:56 | |
| *** witlessb has quit IRC | 23:14 | |
| *** shakamunyi has quit IRC | 23:33 | |
| *** openstackgerrit has quit IRC | 23:46 | |
| *** openstackgerrit has joined #openstack-sahara | 23:47 | |
| *** tosky has quit IRC | 23:53 | |
| *** Networkn3rd has quit IRC | 23:55 | |
| *** Networkn3rd has joined #openstack-sahara | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!