Wednesday, 2015-06-10

*** chen12 has quit IRC00:07
*** chen12 has joined #openstack-sahara00:07
*** Networkn3rd has quit IRC00:12
*** chlong-zzz has joined #openstack-sahara00:17
*** hdd has quit IRC00:41
*** egafford has joined #openstack-sahara01:16
*** egafford has quit IRC01:50
*** Nikolay_St has quit IRC03:28
*** coolsvap|afk is now known as coolsvap03:42
*** hdd has joined #openstack-sahara03:58
*** sgotliv has joined #openstack-sahara03:59
*** sgotliv has quit IRC04:03
*** stanchan has joined #openstack-sahara04:18
*** chlong-zzz is now known as chlong04:32
*** hdd has quit IRC05:17
*** nkrinner has joined #openstack-sahara05:32
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add method for geting instances with process  https://review.openstack.org/18723207:35
*** witlessb has joined #openstack-sahara07:36
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check for cinder in scenario tests  https://review.openstack.org/18756007:38
openstackgerritVitaly Gridnev proposed openstack/sahara-specs: Configure most important hadoop configs automatically  https://review.openstack.org/17629407:51
*** chlong has quit IRC07:51
openstackgerritlu huichun proposed openstack/sahara: [EDP] Add scheduling EDP jobs in sahara(oozie engine implementation)  https://review.openstack.org/18231007:53
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check sqoop2 in custom scenario  https://review.openstack.org/18928807:56
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check sqoop2 in custom scenario  https://review.openstack.org/18928808:01
*** pino|work has joined #openstack-sahara08:08
*** esikachev has joined #openstack-sahara08:10
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add custom check of impala  https://review.openstack.org/18877708:11
openstackgerritMerged stackforge/sahara-ci-config: Change username for hdp images on 'cloud-user'  https://review.openstack.org/18963608:15
*** Nikolay_St has joined #openstack-sahara08:19
openstackgerritMerged openstack/sahara: Transform configuration values into int or float when needed  https://review.openstack.org/18744408:27
openstackgerritlu huichun proposed openstack/sahara: [EDP] Add scheduling EDP jobs in sahara(oozie engine implementation)  https://review.openstack.org/18231008:36
openstackgerritlu huichun proposed openstack/sahara-specs: Add scheduling edp jobs in sahara  https://review.openstack.org/17571908:40
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check of solr in CDH 5.3  https://review.openstack.org/19007208:42
openstackgerritAlexander proposed openstack/sahara-specs: Specification for spark-jobs-for-cdh-5-3-0 added  https://review.openstack.org/19007308:44
*** vgridnev has joined #openstack-sahara08:46
openstackgerritlu huichun proposed openstack/sahara-specs: Add scheduling edp jobs in sahara  https://review.openstack.org/17571908:51
openstackgerritlu huichun proposed openstack/sahara-specs: Add scheduling edp jobs in sahara  https://review.openstack.org/17571909:03
openstackgerritAlexander proposed openstack/sahara-specs: Specification for spark-jobs-for-cdh-5-3-0 added  https://review.openstack.org/19007309:19
openstackgerritAlexander proposed openstack/sahara-specs: Specification for spark-jobs-for-cdh-5-3-0 added  https://review.openstack.org/19007309:25
*** sgotliv has joined #openstack-sahara09:27
openstackgerritAlexander proposed openstack/sahara-specs: Specification for spark-jobs-for-cdh-5-3-0 added  https://review.openstack.org/19007309:32
*** tosky has joined #openstack-sahara09:35
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add sentry check for CDH 5.3  https://review.openstack.org/19008209:40
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check of solr in CDH 5.3  https://review.openstack.org/19007209:50
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add sentry check for CDH 5.3  https://review.openstack.org/19008209:54
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check of solr in CDH 5.3  https://review.openstack.org/19007209:56
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add custom check for hbase  https://review.openstack.org/19010910:40
*** coolsvap is now known as coolsvap|afk10:45
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add sentry check for CDH 5.3  https://review.openstack.org/19008210:46
*** sgotliv_ has joined #openstack-sahara10:53
*** sgotliv has quit IRC10:54
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously  https://review.openstack.org/18801111:11
*** Longgeek has joined #openstack-sahara11:22
*** Longgeek_ has joined #openstack-sahara11:25
*** Longgeek has quit IRC11:28
openstackgerritAlexander proposed openstack/sahara: Spark jobs in Cloudera 5.3.0 plugin are now supported.  https://review.openstack.org/19012811:31
openstackgerritNikita Konovalov proposed openstack/sahara: Removed unused filtering in get_plugins  https://review.openstack.org/19012911:32
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously  https://review.openstack.org/18801111:37
*** vgridnev has quit IRC11:49
*** vgridnev has joined #openstack-sahara11:50
openstackgerritZhenguo Niu proposed openstack/sahara: Remove duplicate 'an' and 'the' in docs  https://review.openstack.org/18953211:51
openstackgerritTelles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously  https://review.openstack.org/18801111:51
openstackgerritMerged openstack/sahara-specs: Adding custom scenario tests  https://review.openstack.org/18880712:18
*** AndreyPavlov has quit IRC12:22
*** raildo has joined #openstack-sahara12:25
*** chlong has joined #openstack-sahara12:29
SergeyLukjanov[15:16:24]  <david-lyle>I'm attempting to move sahara to /contrib and define a structure and testing strategy, it needs more work12:30
SergeyLukjanovit's from today's horizon syncup12:31
SergeyLukjanov_crobertsrh, NikitaKonovalov do you know anything about it?12:31
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check sqoop2 in custom scenario  https://review.openstack.org/18928812:37
SergeyLukjanov_crobertsrh, NikitaKonovalov read horizon meeting logs for details12:38
*** AndreyPavlov has joined #openstack-sahara12:41
*** _crobertsrh is now known as crobertsrh12:49
crobertsrhSergeyLukjanov:  I knew something like this was coming, but I wasn't aware it was in progress.12:49
*** vgridnev has quit IRC12:50
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add method for geting instances with process  https://review.openstack.org/18723212:52
*** vgridnev has joined #openstack-sahara12:53
*** skolekonov has joined #openstack-sahara12:57
*** AndreyPavlov has quit IRC12:58
*** vgridnev has quit IRC12:59
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add method for geting instances with process  https://review.openstack.org/18723213:02
*** vgridnev has joined #openstack-sahara13:03
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check of solr in CDH 5.3  https://review.openstack.org/19007213:04
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add sentry check for CDH 5.3  https://review.openstack.org/19008213:06
*** tmckay has joined #openstack-sahara13:06
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add custom check for hbase  https://review.openstack.org/19010913:07
openstackgerritChad Roberts proposed openstack/python-saharaclient: Adding data source editing to library and CLI  https://review.openstack.org/18984913:26
*** egafford has joined #openstack-sahara13:51
openstackgerritNikita Konovalov proposed openstack/sahara: Removed dependency on Spark plugin in edp code  https://review.openstack.org/19019914:03
*** degorenko has quit IRC14:12
*** degorenko has joined #openstack-sahara14:19
*** nkrinner has quit IRC14:29
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add check sqoop2 in custom scenario  https://review.openstack.org/18928814:31
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Add CDH 5.4.0 jobs  https://review.openstack.org/19020614:37
*** openstackgerrit has quit IRC14:41
*** openstackgerrit has joined #openstack-sahara14:41
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add custom check of impala  https://review.openstack.org/18877714:45
*** esikachev has quit IRC14:45
elmikocrobertsrh: are you still working on https://review.openstack.org/#/c/189849/ , or is it workflowed to wait for the service patch?14:53
crobertsrhwaiting for service patch14:53
crobertsrhreviews are welcome though14:54
elmikoyea, i was looking at it last night, just wanted to make sure =)14:54
*** stanchan has quit IRC14:54
crobertsrhalso...kinda waiting for the specs to be approved too :)14:57
crobertsrhA bit of cart before horse, but it was next on my list.14:57
elmikohehe, ok14:57
elmikoit looked pretty good to me14:58
*** hdd has joined #openstack-sahara14:58
*** vgridnev has quit IRC15:22
*** skolekonov has quit IRC15:30
toskyelmiko: Keystone Session objects, does it only impact the way clients are written? Will alternative clients need to be rewritten as well (like tempest, with its separate set of clients)?15:42
elmikotosky: it will impact the way we authenticate client objects15:47
elmikowe won't need to rewrite them immediately, but we will need to ensure that all the clients can accept Session objects as their auth method15:48
elmikothen, we will start to change just the auth method for those clients on their creation15:48
elmikothat should be the only the change, but it's possible that some features won't be exposed, although this is rare15:48
elmikofor example the keystone client exposes an authenticate()15:49
elmikothat goes away with Sessions because you don't need to get a token directly15:49
toskyok, so this means that tempest client should be checked as well (no idea how all of this works)15:49
elmikoyea probably15:49
elmikothe main part of the change will be adjusting things from, for example, tempestclient.Client(auth_url=..., username=..., password=..., ...) to tempestclient.Client(session=some_session)15:50
elmikoone of the benefits is that we can abstract away the authentication method15:51
elmikobut, we can do this work in stages. so we don't need to tip the apple cart all at once ;)15:51
elmikodoes that make sense?15:54
toskyyes15:55
elmikosweet =)15:55
toskyI was trying to understand the code of the tempest client, but I miss many details15:55
elmikoit is possible that tempest does not implement session based creation yet, i haven't looked at that one15:56
*** robsparker has quit IRC15:56
toskyelmiko: client https://github.com/openstack/tempest/blob/master/tempest/services/data_processing/v1_1/data_processing_client.py15:56
toskywhich inherits from https://github.com/openstack/tempest/blob/master/tempest/common/service_client.py15:56
elmikook, so this is on the tempest side then?15:57
toskywhich in turn inherits from https://github.com/openstack/tempest-lib/blob/master/tempest_lib/common/rest_client.py15:57
toskyyes15:57
elmikook, i don't think we will need to modify that code. i was mainly targetting the sahara codebase for the first pass.15:57
toskythe normal clients are not used, for various reasons; there are own wrappers around REST APIs15:57
toskygreat!15:57
elmikofrom the outside, sahara should continue to function as normal. i think it will mainly be a transparent upgrade.15:58
toskythanks for explanation, I wanted to be sure15:58
elmikonp15:58
*** robspark_ has joined #openstack-sahara15:58
*** tosky_ has joined #openstack-sahara16:00
*** tosky has quit IRC16:01
*** Nikolay_St has quit IRC16:01
*** Networkn3rd has joined #openstack-sahara16:08
*** vgridnev has joined #openstack-sahara16:16
*** hdd has quit IRC16:33
openstackgerritChad Roberts proposed openstack/sahara: Allowing job binary objects to be updated  https://review.openstack.org/19025016:34
*** Networkn3rd has quit IRC16:51
*** Nikolay_St has joined #openstack-sahara16:52
*** AndreyPavlov has joined #openstack-sahara17:22
*** hdd has joined #openstack-sahara17:22
*** hdd has quit IRC17:23
*** stanchan has joined #openstack-sahara17:23
*** amcrn has joined #openstack-sahara17:29
*** egafford has quit IRC17:32
*** pino|work has quit IRC17:35
*** asimzafir has joined #openstack-sahara17:42
asimzafirhi fellows17:42
elmikohey asimzafir17:43
asimzafirhi elmiko , hope all is well on your end17:44
asimzafirin process of testing 20tb datapush to swift bucket. have you guys done any of that testing17:44
elmikothings are well, just getting ready to head out for san fran, spark summit next week =)17:44
elmikoasimzafir: whoa! never on 20tb lol17:44
asimzafiror let it be any number of tb17:45
asimzafirbasically want to see how do i get access to swift bucket from a cdh cluster17:45
elmikoah, cool17:45
asimzafirhave any of you used swift on cdh to backup datasets?17:47
elmikoi have not17:47
asimzafirok17:47
*** hdd has joined #openstack-sahara17:47
asimzafirwe want to provide feature to our cloud customer to backup datasets on swift  - similar to what s3 has17:48
elmikothis might be a good topic to bring up at the meeting tomorrow, there are usually some of the intel folks in attendance and they have worked with cdh a great deal.17:48
asimzafircan i be part of it ?17:48
elmikoyea definitely!17:48
asimzafirgreat17:48
asimzafirso far there is a tedious way to do it, and its not as simple as s317:48
elmikoasimzafir: https://wiki.openstack.org/wiki/Meetings/SaharaAgenda17:48
asimzafirits not cdh specific but pretty much any clustering environment should give seemless hook to s3 - specially on bigdata clsuters17:49
elmikotomorrow's meeting will be the 1800utc17:49
asimzafirand i have some performance data to share with you17:49
*** sgotliv_ has quit IRC17:49
elmikocool, it sounds like a good feature actually17:49
asimzafirwe run into the situation where the cluster was spawnned on top of emepheral local storage - it turned out that compute node went down, so the live migration didnt go successful, as a result it was pain to move 1TB volume and the way did it by letting the hdfs create underreplicated blocks, had it been something like cinder volumes, it would be much similar where we would just move a volume (i.e. detach it from one) and17:51
asimzafirso i think sahara cluster deployment should be optimized for cinder volumes then empheral , as its not very safe for customer17:52
elmikothat does sound non-ideal17:52
asimzafiri am thinking of cinder similar to that of amazon EBS optimized IOPS volume,17:54
asimzafirelmiko looking forward to see you in sf then17:58
elmikooh nice, that will be cool!17:58
elmikoasimzafir: you'll get to meet the rest of our team then lol18:00
asimzafirgreat18:00
asimzafirhad encountered some scenraios that i want to prepare to present in tokyo in colloboration if that sound cool to you guys, i will be more active on openstack-sahara - there was lot going on since i came back apparently.18:01
elmikoasimzafir: yea! that sounds great18:02
asimzafirjust got update CDH doesn't support swift as of now18:03
asimzafirthe jars aren't working and thats not a surprise :)18:03
elmikoah, ok18:04
*** robspark_ has quit IRC18:04
*** robsparker has joined #openstack-sahara18:05
*** egafford has joined #openstack-sahara18:05
asimzafirbut apparently you guys have done some work on http://docs.openstack.org/developer/sahara/userdoc/hadoop-swift.html - so i am trying to see if it works despite of what i am hearing..18:06
elmikoi think you might be able to deploy the hadoop-swift jars manually and then gain access, but it would be outside of cdh's procedures18:08
*** robspark_ has joined #openstack-sahara18:09
*** robsparker has quit IRC18:09
*** asimzafir has quit IRC18:11
*** tosky_ has quit IRC18:13
*** AndreyPavlov has quit IRC18:25
*** Networkn3rd has joined #openstack-sahara18:30
openstackgerritChad Roberts proposed openstack/sahara: Allowing data souce objects to be updated  https://review.openstack.org/18971118:33
*** Nikolay_St has quit IRC18:45
*** asimzafir has joined #openstack-sahara18:59
*** Nikolay_St has joined #openstack-sahara19:05
*** hdd has quit IRC19:05
crobertsrhI'm guessing sahara-ci is hosed?19:10
elmikodunno, did you break it?19:11
*** hdd has joined #openstack-sahara19:13
crobertsrhEither hosed or everyone became bad at coding at the same moment19:19
elmikohmm, both of those sound possible... ;)19:20
*** hdd has quit IRC19:28
*** hdd has joined #openstack-sahara19:29
elmikocrobertsrh: looks like whoever controls that gate-sahara.sh file might have some 'splainin to do19:43
*** robspark_ has quit IRC19:43
crobertsrhuh huh19:44
elmikoi'm not sure where that file is even stored though19:44
*** hdd has quit IRC19:49
SergeyLukjanovfolks, need help with understanding why dsvm on client failing20:08
SergeyLukjanovwe need to merge https://review.openstack.org/#/c/188502/ asap and release client with updated requirements20:08
SergeyLukjanovtmckay, crobertsrh, elmiko, anyone could help?20:08
elmikoi've been looking at the failure logs, but i don't know where the gate-sahara.sh file is stored20:09
elmikooh wait, this looks different.20:09
crobertsrhhttps://github.com/stackforge/sahara-ci-config/tree/master/slave-scripts20:09
crobertsrhgate-sahara.sh ^^^20:09
elmikocrobertsrh: awesome, thanks!20:10
elmikoso much for github search ...20:10
elmikoSergeyLukjanov: fyi i have to leave for the airport in about 45min. so my time on this is limited20:11
SergeyLukjanovI'm talking about check-saharaclient-dsvm-functional20:11
tmckayelmiko, any logs of interest in particular?20:11
SergeyLukjanovelmiko, have a good flight20:12
elmikotmckay: i was looking at a different issue, haven't looked at this yet20:12
elmikoSergeyLukjanov: thanks =)20:12
tmckayah, gotcha20:12
SergeyLukjanovtmckay, crobertsrh, elmiko the sahara-ci isn't so important, we should fix check-saharaclient-dsvm-functional, it blocks merging of CR20:12
SergeyLukjanovto client20:12
elmikoack20:12
SergeyLukjanovin fact we should unblock https://review.openstack.org/#/c/188502/20:12
SergeyLukjanovI'm trying to understand, but already falling asleep20:12
tmckaydoh, stdout: ERROR: Could not find Sahara endpoint in catalog20:13
*** hdd has joined #openstack-sahara20:13
SergeyLukjanovtmckay, prev. lines contains Nones in calls20:13
SergeyLukjanovtmckay, I'm not sure - is it ok20:13
elmikoSergeyLukjanov: looks like a stack issue though, ERROR: Could not find Sahara endpoint in catalog20:13
SergeyLukjanovit was rechecked few times20:14
SergeyLukjanovsahara service logs looks correct20:14
tmckayyeah, but the Nones, hmm.  I'll try to find a test that passed and see what's in there ...20:14
elmikohmm =(20:14
elmikotmckay: agreed, that seems odd20:15
SergeyLukjanovit the test, that are passing we will not see this call line :(20:15
tmckayhmm, I guess you're right20:16
elmikoSergeyLukjanov: anyway to check the config for sahara, the sahara logs are showing invalid tokens20:18
SergeyLukjanovelmiko, hm, don't see20:18
elmikohttp://logs.openstack.org/02/188502/1/check/check-saharaclient-dsvm-functional/7c69d4c/logs/screen-sahara.txt.gz20:18
elmikoactually,20:19
elmikohttp://logs.openstack.org/02/188502/1/check/check-saharaclient-dsvm-functional/7c69d4c/logs/screen-sahara.txt.gz#_2015-06-10_10_33_58_41020:19
SergeyLukjanovelmiko, I think it's wait loop that is waiting when service started20:23
elmikoSergeyLukjanov: ahh, so it's just failing while waiting then?20:23
SergeyLukjanovelmiko, because this message posted right after the sahara service started20:23
tmckayso where is the code for check-saharaclient-dsvm-functional? I have no idea20:23
SergeyLukjanovtmckay, code is in saharaclient/test/functional20:24
SergeyLukjanovtmckay, simple read-only CLI tests20:24
SergeyLukjanovelmiko, I think it's because wait loop just makes telnet localhost 838620:24
SergeyLukjanovelmiko, and it produces auth failure20:24
elmikoack20:24
*** witlessb has quit IRC20:30
tmckaySergeyLukjanov, ack, I meant the scripting around it.  Where is the stuff that launches the test?20:31
tmckaymy guess would be those Nones are bad.  Looks like  ClientTestBase uses os.environ.get() for those values, doesn't seem to be any other option20:33
SergeyLukjanovtmckay, https://github.com/openstack/python-saharaclient/blob/master/saharaclient/tests/functional/hooks/post_test_hook.sh20:34
tmckayif they aren't in the environ, the client has nothing20:34
SergeyLukjanovtmckay, agree20:34
tmckaywe need tosky :)20:35
elmikotmckay: +120:36
tmckaylooks like he's asleep (or having fun)20:36
elmikoi wonder if there is some issue sourcing the openrc script20:36
tmckayelmiko, good gues20:37
elmikoit is so tough to tell though20:37
elmikosadly i gotta get ready to leave20:38
tmckayhmm, maybe we could instrument this file20:38
*** witlessb has joined #openstack-sahara20:38
tmckaySergeyLukjanov, if we added debug code to https://github.com/openstack/python-saharaclient/blob/master/saharaclient/tests/functional/hooks/post_test_hook.sh, how hard is it to get that into the gate?20:39
tmckayI wonder if we could add some checks around the "source openrc admin admin"20:39
SergeyLukjanovtmckay, you could just create a CR with customs and it'll be executed in the gate20:40
elmikonice idea20:40
tmckaytrying to find which log "Running saharaclient functional test suite" shows up in20:43
tmckaylooks like maybe we could add echos20:43
SergeyLukjanovtmckay, devstack log IMO20:43
elmikolater guys, good luck20:46
*** elmiko has quit IRC20:46
*** crobertsrh is now known as _crobertsrh21:00
openstackgerritTrevor McKay proposed openstack/python-saharaclient: Silly test, starting point for finding errors  https://review.openstack.org/19035121:08
tmckayran tox -efunction by hand with no enviornment on my laptop, I get the same errors (Could not find Sahara endpoint in catalog)21:15
*** Networkn3rd has quit IRC21:22
openstackgerritOpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements  https://review.openstack.org/18991421:27
*** asimzafir has quit IRC21:32
*** tmckay has left #openstack-sahara21:37
*** vgridnev has quit IRC21:43
*** amcrn has quit IRC21:57
*** stanchan has quit IRC21:59
openstackgerritAndrew Lazarev proposed openstack/python-saharaclient: Test commit to check gate failure  https://review.openstack.org/19038022:18
*** witlessb has quit IRC22:25
*** hdd has quit IRC22:43
openstackgerritAndrew Lazarev proposed openstack/python-saharaclient: Test commit to check gate failure  https://review.openstack.org/19038023:08
openstackgerritOpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements  https://review.openstack.org/18991423:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!