Thursday, 2014-04-03

*** RockKuo has joined #openstack-sahara00:02
*** elmiko has quit IRC00:14
*** ErikB1 has quit IRC00:25
*** witlessb has joined #openstack-sahara01:56
*** bradd1 has joined #openstack-sahara02:03
*** bradd1 has quit IRC03:41
*** akuznetsov has joined #openstack-sahara04:19
*** IlyaE has quit IRC04:40
*** Ch00k has joined #openstack-sahara05:05
*** IlyaE has joined #openstack-sahara05:30
*** Ch00k has quit IRC06:03
openstackgerritJenkins proposed a change to openstack/sahara: Imported Translations from Transifex  https://review.openstack.org/8271906:04
*** Ch00k has joined #openstack-sahara06:16
*** IlyaE has quit IRC06:26
*** Ch00k has quit IRC06:42
*** tnovacik has joined #openstack-sahara07:18
*** witlessb has joined #openstack-sahara07:49
*** tnovacik has quit IRC07:49
*** Ch00k has joined #openstack-sahara07:51
*** akuznetsov has quit IRC08:01
*** akuznetsov has joined #openstack-sahara08:07
*** skolekonov has joined #openstack-sahara08:24
*** vrovachev has joined #openstack-sahara08:40
*** dmitryme has joined #openstack-sahara08:54
*** tosky has joined #openstack-sahara09:48
openstackgerritA change was merged to openstack/sahara: Add short info re testing  https://review.openstack.org/8471410:13
openstackgerritA change was merged to openstack/sahara: Compact all Icehouse migrations into single one  https://review.openstack.org/8449810:13
*** witlessb_ has joined #openstack-sahara10:43
*** witlessb has quit IRC10:44
*** witlessb_ is now known as witlessb10:44
openstackgerritDmitry Mescheryakov proposed a change to openstack/sahara: Fix up installation guide  https://review.openstack.org/8487111:05
*** sreshetnyak has quit IRC11:07
toskydmitryme: about his RR ^, but it's a more generic question (which impacts it): isn't Keystone V2 API deprecated now, so wouldn't it be better to enable V3 by default?11:08
dmitrymetosky: hmm, good question. Seems like we should remove this parameter and just use v3 by default. But am not sure if it is a good idea to change this at this time11:10
dmitrymewe are going to make rc1 today :-)11:11
toskydmitryme: uhm, maybe exception post-rc1? This is quite linked to an already made big keystone change, so maybe it could be granted11:11
toskyand it's not new code which didn't exist before11:12
toskywhat do you use on the gate right now?11:12
dmitrymethe gate runs Tempest which means the latest devstack11:13
dmitrymesavanna-ci runs  devstack which might be week or two outdated11:13
dmitrymeSergeyLukjanov: what do you think of making use_identity_api_v3=True by default in Icehouse?11:15
dmitrymeI am not speaking about removing it11:15
toskythis is the official blueprint for keystone: https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api11:16
SergeyLukjanovdmitryme, I think that we should make it True by default11:17
SergeyLukjanovand remove config option in Juno and cleanup corresponding code11:17
SergeyLukjanovtosky, ^^11:17
toskythere was someone asking for removal of keystone v2 tests in tempest O.o11:18
dmitrymeok, since we agreed, who will make the patch?11:18
toskyQA "managers" replied to wait for the official removal of the API in the devel cycle, which makes sense11:18
SergeyLukjanovdmitryme, not me, please ;)11:21
dmitrymetosky: will you take it?11:21
toskydmitryme: I'm not strictly a developer in my role here, but I can try11:22
dmitrymetosky: if you're interested, then sure. If not - I can make, it will not take much time11:23
toskydmitryme: it's quite easy, the option is True already by default if not specified (ba1acf80efeb98296e9dc9fa89330550f82f3a7b)11:25
toskyso it's just a matter of fixing the sample file11:25
dmitrymetosky: right, seems like  we've missed to fix sample-basic that time11:26
toskydmitryme: so, anyway, the documentation can be changed :)11:27
dmitrymetosky: right. I think it would be better to just remove the whole abstract11:28
dmitrymeseems like this is non-essential option any more11:28
dmitrymebtw, I think it makes sense to just remote that parameter from sample-basic for the same reason11:29
toskyand then .sample too11:29
dmitrymetosky: nope, the .sample should be left as is until we remove that parameter from code11:30
toskyah, no, that's the complete reference11:30
dmitrymetosky: right, besides all parameters in code are enforced by gating check to be listed in .sample11:31
toskyoh, nice11:31
openstackgerritA change was merged to openstack/sahara: Updating the vanilla image building docs  https://review.openstack.org/8454711:39
toskydmitryme: are you preparing that review for sample-basic, or should I start it?11:46
dmitrymetosky: I didn't start yet. Do you want to do it? If no, I can do it in a minute.11:47
toskydmitryme: sure, I can do it11:47
toskydo I need some "partially implements" or not?11:47
dmitrymenope, since this is not a blueprint implementation11:48
dmitrymeneither we have a bug filed for the issue11:48
toskyoki11:48
openstackgerritLuigi Toscano proposed a change to openstack/sahara: Do not document use_identity_api_v3 in the sample-basic file  https://review.openstack.org/8500211:53
openstackgerritDmitry Mescheryakov proposed a change to openstack/sahara: Fix up installation guide  https://review.openstack.org/8487111:55
openstackgerritAlexander Ignatov proposed a change to openstack/sahara: Added rackawareness to Hadoop 2 in vanilla plugin  https://review.openstack.org/8465311:57
openstackgerritA change was merged to openstack/sahara: Reserve 5 migrations for backports  https://review.openstack.org/8449912:01
toskydmitryme: question about database creation if you install through RDO: (not asking through review as I'm not sure it's true) isn't the db created on the fly if you use SQLite, as written into installation.guide.rst?12:05
*** dmitryme has quit IRC12:07
*** sballe has joined #openstack-sahara12:31
*** dmitryme has joined #openstack-sahara12:34
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Update EDP requirements for hadoop v2  https://review.openstack.org/8501712:38
dmitrymetosky: do you mean current installation.guide.rst, not the one in my CR?12:41
toskydmitryme: the one in the review12:41
dmitrymehmm, where is it written that db is created on the fly?12:42
dmitrymeafter https://review.openstack.org/#/c/82750/ was merged the DB is never created on the fly12:42
*** RockKuo has quit IRC12:43
toskydmitryme: that's why I asked; but rereading it seems ok, I was confused by "In case of SQLite, if the database file does not exist, it will be automatically created by ``sahara-db-manage``"12:43
*** ErikB1 has joined #openstack-sahara12:44
dmitrymetosky: ok12:45
*** _crobertsrh has quit IRC12:46
*** DenisEgorenko has joined #openstack-sahara12:50
*** crobertsrh has joined #openstack-sahara12:51
*** tmckay has joined #openstack-sahara12:51
*** elmiko has joined #openstack-sahara12:53
openstackgerritMichael McCune proposed a change to openstack/sahara: Updating the setup development environment docs for icehouse  https://review.openstack.org/8420613:01
*** dmitryme has quit IRC13:12
openstackgerritTrevor McKay proposed a change to openstack/sahara: Add examples of upstream files that we should not change  https://review.openstack.org/8489413:30
*** dmitryme has joined #openstack-sahara13:33
tmckaySergeyLukjanov, crobertsrh, on https://review.openstack.org/#/c/84881/, Sergey's comment.  I don't think swift credentials are needed for relaunch, because they are copied and injected from the data sources for all job types but Java actions.  For Java actions, hmm, we might have a problem.  crobertsrh, can a user specify configs on a relaunch?13:36
crobertsrhRight, swift credentials are injected from the data sources (except for java actions).  A user can definitely specify new configs (or tweak previous configs) on relaunch....that, to me, is the essence of relaunch.13:37
tmckaycrobertsrh, so, just to be clear, relaunch a Java action.  Is there a tab populated with existing configs?  So a user can re-enter swift credentials there as configs or args, depending on how the job is written13:38
elmikodmitryme: are you going to touch doc/source/horizon/installation.guide.rst?13:38
crobertsrhYeah, the existing configs are pre-populated in a tab where the user can edit them13:38
tmckayalright, so swifts will be missing.  do you think we need a heads up in the user guide for this use case?13:39
dmitrymeelmiko: hmm, I can take it13:39
crobertsrhThey could re-enter swift configs....ideally, the UI can receive some sort of marker that will let us know that something was left out so that we can make it obvious to the user that they might need to re-add the credentials.13:39
dmitrymeguys, did anyone start to work on UI installation guide?13:39
crobertsrhFor now, a user guide note is probably sufficient13:39
dmitrymeif not, I will take it13:40
crobertsrhdmitryme:  I did not work on the UI install guide13:40
elmikolet me look13:40
crobertsrhas far as I know, nobody has13:40
* crobertsrh forgot about it :)13:40
elmikowe don't have a work item for ui installation13:41
tmckaycrobertsrh, the beauty of the filter method I added is that we can just change "del" to a "replace"13:41
tmckaycrobertsrh, hmmm, what if I left them in there but made the value empty string?13:41
dmitrymeelmiko: I'll add it and assign to myself13:41
crobertsrhsimple fix :)13:41
elmikodmitryme: sure, i was just reading it and noticed a slight difference for RDO13:41
crobertsrhtmckay:  that should work13:41
tmckaycrobertsrh, okay, I'll tweak the fix.  Make sure you review it (I think I added you)13:42
crobertsrhAt some point, I could update the UI to put a marker next to fields that have a name but no value13:42
crobertsrhor rather an empty string value13:42
openstackgerritA change was merged to openstack/sahara: Do not document use_identity_api_v3 in the sample-basic file  https://review.openstack.org/8500213:45
dmitrymebtw people, if you didn't know, Jenkins builds and publishes docs for each CR. Look for 'gate-sahara-docs' in Jenkins report. Useful when reviewing someone else's CR13:45
tmckayright.13:45
aignatovtmckay: here?14:20
tmckayhi14:20
tmckayaignatov, here14:21
aignatovhi, I've an issue with uploading files to job binary internal14:22
aignatovif try to upload file about 200 KB size it crashes with Internal Server Error14:22
aignatovand traces are14:22
aignatovhttp://paste.openstack.org/show/74976/14:23
aignatovmaybe you saw it before14:23
tmckayno.  Maybe there is a size limit on a BLOB.  I didn't think there was.14:24
tmckayaignatov, we'll have to check on BLOB limits and see.14:25
aignatovI have the following column structure in job binary internal for  'data'  column14:26
toskyoh, so a job binary uploaded to internal storage (not swift) is stored into the db? Why not an external file, if I can ask?14:26
aignatov"| data       | blob        | YES  |     | NULL    |       |"14:26
aignatovtosky: I'm just verifying it, it should work in sahara for such small-size files14:27
toskyaignatov: I'm pretty sure mysql and postgresql should be able to handle even bigger files, but for sqlite, uhm (ok, it's not for production)14:28
* tosky always tried to avoid blobs for files when doing db stuff in the past14:28
tmckayaignatov, yeah, is this sqlite?  I was wondering about that14:28
tmckaytosky, I suppose we could have stored a path for the file in the database, and stored the file externally14:29
tmckayhmm, looks like mysql based on the trace14:30
SergeyLukjanovtmckay, yup, it's ok to remove credentials14:30
aignatovtmckay: yes, it's mysql14:32
aignatovjust for sure, I have job_binary_max_KB=5120 as it is by default14:33
*** bradd1 has joined #openstack-sahara14:40
tmckayaignatov, we'll have to find out max BLOB sizes for the different database types, and probably document it.14:40
tmckayanother reason for swift14:41
aignatovheh, I noticed that srehetniak use the same case and it works well in her mysql and env :)14:42
dmitrymes/her/his14:42
aignatovdmitryme: yes, thx14:43
*** bradd1 has quit IRC14:50
tmckayreally?  That's troubling14:50
tmckayI wonder why the difference14:51
*** bradd1 has joined #openstack-sahara14:52
openstackgerritTrevor McKay proposed a change to openstack/sahara: Add a custom filter method to scan wrapped dict results  https://review.openstack.org/8488114:53
*** IlyaE has joined #openstack-sahara14:56
toskyruhe: about you answer on openstack-dev: is that heat template the same one which is generated on the fly now when the heat engine is used?15:01
*** skolekonov has quit IRC15:02
elmikotosky: still working on this cluster stuff, i'm watching the sahara.log and i see it change the cluster state from InfraUpdating to Waiting, but i'm not seeing any ssh calls. is there something else i can look for, maybe possible failures from sahara?15:11
toskyelmiko: uhm, also in the other components (heat - if you are using it - and nova)15:12
elmikotosky: not using heat, afaik. i'll check nova15:12
tmckaybrb15:16
elmikotosky: and the cluster will stay in Waiting state until it is ready?15:17
toskyelmiko: iirc yes15:18
elmikoi don't think this is working, but i'll give it another 10 minutes15:18
openstackgerritYaroslav Lobankov proposed a change to openstack/sahara: Doc's update for integration tests  https://review.openstack.org/8507115:19
ruhetosky: hi15:21
ruhetosky: that's a different type of heat template :)15:21
toskyruhe: ah!15:22
toskyI didn't investigate heat too much15:22
ruheit's a template for heat to provision hadoop cluster by using Sahara15:22
ruheaignatov is working on that15:22
ruhehere is the patch https://review.openstack.org/#/c/72336/15:22
toskyI see - but on the other side, you use heat internally for the deployment of the cluster, if I understand it correctly15:23
ruhetosky: correct15:23
ruheit's like in that dicaprio movie :)15:24
toskyyeah, when the patch is completed, heat will ask sahara which will use heat engine to do the job15:24
ruheheat calls sahara to deploy hadoop. sahara calls heat to deploy infrastructure15:24
toskyinteresting15:25
*** tmckay has quit IRC15:40
*** tmckay has joined #openstack-sahara15:41
*** tmckay has left #openstack-sahara15:48
*** tmckay has joined #openstack-sahara15:49
ylobankovHi american guys :) could you please take a look at https://review.openstack.org/#/c/85071/ . Thank you.16:01
elmikoylobankov: reading it now :)16:02
ylobankov:)16:04
toskyylobankov: not a native speaker, but a small note: I know that "Hadoop version 3.0.2" and "Hadoop version 2.5.1" refers to different versions of (Intel, etc) and the context comes from the plugin, but maybe it would be better to make it explicit16:07
*** Ch00k has quit IRC16:08
aignatovtosky: where you from? :)16:10
toskyaignatov: Italy, working in Czech Republic16:10
aignatovtosky: cool :)16:10
*** IvanBerezovskiy has left #openstack-sahara16:12
ylobankovtosky: what do you mean?16:56
toskyylobankov: if you say "Hadoop", there is no 2.5.1 or 3.0.2, latest is 2.3.0 (vanilla)16:57
toskyylobankov: so I was thinking, instead of "IDH plugin with the Hadoop version 2.5.1", "IDH plugin with the Intel Hadoop version 2.5.1" (or the full name, Intel Distribution for Apache Hadoop)17:00
toskyjust to be clear17:00
vrovachevtosky: 2.5.1 dnd 3.0.2 - IDH versions17:00
vrovachevand_17:01
*** sreshetnyak has joined #openstack-sahara17:01
*** Ch00k has joined #openstack-sahara17:01
ylobankovtosky: got it :)17:06
*** dmitryme has quit IRC17:10
*** skostiuchenko has joined #openstack-sahara17:11
tmckaymeeting is in an hour, right?  Well, less17:12
tmckayI need an emergency hardware run.  Pre-school party at my house and the swing is broken :)17:13
tmckayback real soon17:13
toskyylobankov: :)17:13
ylobankovtmckay: right17:13
*** skostiuchenko has left #openstack-sahara17:17
*** skostiuchenko has joined #openstack-sahara17:17
openstackgerritA change was merged to openstack/sahara: Open Juno dev  https://review.openstack.org/8450317:22
*** tosky has quit IRC17:36
openstackgerritYaroslav Lobankov proposed a change to openstack/sahara: Doc's update for integration tests  https://review.openstack.org/8507117:42
*** bradd1 has quit IRC17:44
*** sreshetnyak has quit IRC17:59
elmikoare we having a meeting?18:00
elmikoSergeyLukjanov: ^^18:00
SergeyLukjanovhey folks, meeting time18:00
SergeyLukjanovelmiko, sure18:01
*** dmitryme has joined #openstack-sahara18:04
*** tosky has joined #openstack-sahara18:05
*** akuznetsov has quit IRC18:12
*** akuznetsov has joined #openstack-sahara18:17
openstackgerritAndreas Jaeger proposed a change to openstack/sahara: Check that all po/pot files are valid  https://review.openstack.org/8423618:18
*** akuznetsov has quit IRC18:28
*** akuznetsov has joined #openstack-sahara18:38
ylobankovruhe: hi18:46
*** bradd1 has joined #openstack-sahara18:46
ruheylobankov: hi (that was just a test)18:46
aignatovhi rune, hi ylobankov, how are you, guys?18:49
ylobankovI am fine, thanks. And you?18:49
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Removes default usage of root-passwd element from HDP image generation  https://review.openstack.org/8514318:49
tmckaylol18:50
tosky\o/ (root-password)18:50
toskyelmiko: ^^18:50
ylobankovaignatov: I am fine, thanks. And you? ruhe is fine too.18:52
elmikotosky: cool18:52
aignatovylobankov: well, did you get home? :)18:52
ylobankovNo. we still are working :)18:53
elmikotosky: i could give it a run later and see what happens18:53
ylobankovaignatov: No. we are still working :)18:54
elmikocould someone help me understand how sahara/utils/ssh_remote.py@321 works?18:55
aignatovit seems ylobankov turned on bot mode18:55
elmikoi never see it hit the _connect functions18:55
elmikoand i'm not understanding how procutils.run_in_subprocess will even run that function18:56
aignatovelmiko: dmitryme designed it18:56
elmikodmitryme: ^^18:57
aignatovdmitryme: are you here? could you please describe in a few words how subprocesess work in remote ssh calls?18:57
dmitrymeelmiko: yep, that is tricky thing, give me a minute to recall18:57
elmikook18:58
elmikobecause i put a LOG.debug in _connect and i never see it run the fun18:58
elmikofunc18:58
dmitrymeelmiko: so, that stuff runs _connect in another forked process19:00
dmitrymethat is why you see no logs, they are simply not configured in that other process19:00
elmikook19:01
elmikothat makes sense, but how does the following call use that proxy to connect?19:01
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Removes default usage of root-passwd element from HDP image generation  https://review.openstack.org/8514319:01
dmitrymethe main process does not connects, it is the forked one which does19:01
elmikohmm, ok19:02
dmitrymeafter the connection is established the same forked process is used to run remote commands19:02
dmitrymelike execute_command or read_file19:02
elmikook, how could i check that the forked process has made the connection?19:02
dmitrymein situations like that I write to a file19:03
dmitrymeinstead of log or stdin/stdout19:03
dmitrymebtw which problem do you try to solve19:04
dmitryme?19:04
elmikowell, on my machine i can run `ip netns exec <namespace> ssh user@instance`, but when sahara attempts to run `ls .ssh/authorized_keys` it times out19:04
*** bob_nettleton has joined #openstack-sahara19:05
dmitrymeaha19:05
dmitrymedid you set use_namespaces=true?19:05
elmikoyes19:05
dmitrymethat is a hack made by Hortonworks guys, but this should probably work in your case19:05
*** akuznetsov has quit IRC19:06
dmitrymehmm, are you sure you've restarted sahara-api after reconfiguration?19:06
elmikoyes19:07
elmikoand running sahara-api as root19:07
dmitrymein https://github.com/openstack/sahara/blob/master/sahara/utils/ssh_remote.py#L32019:09
dmitrymetry log self._get_conn_params()[3]19:09
elmikook, will do19:10
dmitrymethis should be neutron info, and I think namespace name should be here as well19:10
elmikook. it will take me a few minutes to get back to that point19:10
elmikouh-oh, self.get_conn_params()[3]=None19:15
elmiko self._get_conn_params()[3]=None19:15
elmikodmitryme: ^^19:16
dmitrymehmm19:16
dmitrymegot it19:16
dmitrymeset use_floating_ips=false19:16
elmikoi believe it is19:17
dmitrymesee https://github.com/openstack/sahara/blob/master/sahara/utils/ssh_remote.py#L31119:17
dmitrymeok, some more digging then19:17
elmikoyea, i have it set to false19:17
dmitrymethat seem to be impossible19:18
dmitrymecheck the code pass19:18
dmitrymethe only way 'info' could be None is if 'if' statement on line 311 was not triggered19:19
elmikook, i'll do some logging19:19
dmitrymes/code pass/code path/19:19
elmikoyea, this is weird. i'm still seeing none. i remade the node-group-template and made sure to specify no floating ips, and sahara.conf has use_floating_ips=false19:22
dmitrymewait19:23
dmitrymeI meant use_floating_ips=false in config file19:23
elmikocorrect19:23
dmitrymeah, sorry19:23
elmikoi have that set19:23
dmitrymethat 'if' depends only on config settings19:23
elmikook19:24
dmitrymecheck that you defined 'use_floating_ips'  and 'use_namespaces' only once in the config19:24
elmikook19:24
dmitrymealso check that you have these lines uncommented19:24
dmitrymeI mean sometimes I change the  value from default to something, but forget to uncomment the line19:25
elmikowait, i found a typo19:25
elmikolet me try again19:25
elmikogreat success, that part is working :)19:29
*** bob_nettleton has left #openstack-sahara19:30
elmikonow i'm getting a different error lol19:30
elmikodmitryme: thanks for helping19:32
dmitrymeelmiko: sure, you're welcome19:34
*** bradd1 has quit IRC19:36
*** IlyaE has quit IRC19:55
*** bradd1 has joined #openstack-sahara19:55
*** IlyaE has joined #openstack-sahara20:14
*** witlessb has quit IRC20:35
openstackgerritTrevor McKay proposed a change to openstack/sahara: Add a custom filter method to scan wrapped dict results  https://review.openstack.org/8488120:38
*** crobertsrh is now known as _crobertsrh21:05
*** bradd1 has quit IRC21:11
*** Ch00k has quit IRC21:17
*** dmitryme has quit IRC21:29
*** dmitryme has joined #openstack-sahara21:30
*** dmitryme has quit IRC21:30
*** bradd1 has joined #openstack-sahara21:46
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Removes default usage of root-passwd element from HDP image generation  https://review.openstack.org/8514321:47
*** bradd1 has quit IRC22:16
*** tellesnobrega has quit IRC22:30
*** tellesnobrega has joined #openstack-sahara22:30
*** IlyaE has quit IRC23:02
*** tosky has quit IRC23:24

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!