| *** esikachev has joined #openstack-sahara | 00:00 | |
| *** esikachev has quit IRC | 00:05 | |
| *** abalutoiu has quit IRC | 00:17 | |
| *** esikachev has joined #openstack-sahara | 01:00 | |
| *** esikachev has quit IRC | 01:05 | |
| openstackgerrit | zhongshengping proposed openstack/puppet-sahara master: Remove deprecated keystone authtoken signing_dir option https://review.openstack.org/481371 | 02:00 |
|---|---|---|
| *** esikachev has joined #openstack-sahara | 02:01 | |
| *** esikachev has quit IRC | 02:06 | |
| *** lucasxu has joined #openstack-sahara | 02:53 | |
| *** lucasxu has quit IRC | 02:59 | |
| *** esikachev has joined #openstack-sahara | 03:02 | |
| *** esikachev has quit IRC | 03:07 | |
| *** ukaynar has joined #openstack-sahara | 03:08 | |
| *** chlong_ has quit IRC | 03:08 | |
| *** chlong_ has joined #openstack-sahara | 03:09 | |
| *** chlong_ has quit IRC | 03:17 | |
| *** chlong_ has joined #openstack-sahara | 03:18 | |
| *** ukaynar has quit IRC | 03:55 | |
| *** esikachev has joined #openstack-sahara | 04:03 | |
| *** esikachev has quit IRC | 04:08 | |
| *** ukaynar has joined #openstack-sahara | 04:10 | |
| *** ukaynar has quit IRC | 04:26 | |
| *** ukaynar has joined #openstack-sahara | 04:26 | |
| *** ukaynar has quit IRC | 04:26 | |
| *** links has joined #openstack-sahara | 04:28 | |
| *** esikachev has joined #openstack-sahara | 05:04 | |
| *** Poornima has joined #openstack-sahara | 05:30 | |
| *** rcernin has joined #openstack-sahara | 05:46 | |
| *** pgadiya has joined #openstack-sahara | 05:50 | |
| *** Poornima has quit IRC | 06:21 | |
| *** aolwas28 has quit IRC | 06:24 | |
| *** rickflare has quit IRC | 06:37 | |
| *** htruta has quit IRC | 06:38 | |
| *** aolwas28 has joined #openstack-sahara | 06:39 | |
| *** rickflare has joined #openstack-sahara | 06:40 | |
| *** htruta has joined #openstack-sahara | 06:42 | |
| *** tesseract has joined #openstack-sahara | 07:17 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-image-elements master: Get files from tarballs.o.o if possible (extjs, policy) https://review.openstack.org/481477 | 07:42 |
| *** anshul has joined #openstack-sahara | 07:49 | |
| *** abalutoiu has joined #openstack-sahara | 08:09 | |
| *** mgoddard_ has joined #openstack-sahara | 08:16 | |
| *** tosky has joined #openstack-sahara | 09:12 | |
| *** openstackgerrit has quit IRC | 09:48 | |
| *** mgoddard_ has quit IRC | 10:19 | |
| *** tellesnobrega has quit IRC | 10:43 | |
| *** mgoddard_ has joined #openstack-sahara | 10:51 | |
| *** esikachev has quit IRC | 10:59 | |
| *** tellesnobrega has joined #openstack-sahara | 11:25 | |
| *** esikachev has joined #openstack-sahara | 11:47 | |
| *** pcaruana has joined #openstack-sahara | 11:49 | |
| *** pgadiya has quit IRC | 12:08 | |
| *** fazalkhan has joined #openstack-sahara | 12:10 | |
| *** tellesnobrega has quit IRC | 12:12 | |
| fazalkhan | Trying to deploy CDH 5.7.0 on OSP 10 using Sahara, cluster provisioning is failing at Plugin: Configure cluster (Configure Services). In the sahara log, I am getting | 12:13 |
| fazalkhan | ERROR sahara.service.ops [req-63639760-e72c-4407-8e5c-37ba8deb01f1 bad7eae8d0304f9d97987774b2da3ca9 cd860a89a6f54ec2a39c300d2b81cf37 - - -] [instance: none, cluster: e30d432a-7ff0-41d6-9671-4d88f1a9fb7e] Error during operating on cluster (reason: <urlopen error [Errno 111] ECONNREFUSED>) | 12:13 |
| fazalkhan | Traceback is going to urllib2.py line 431, in open -- line 449 in _open -- line 409 in _call_chain ---- line 1244 in http_open --- line 1214 do_open --- raise URLError(err). | 12:18 |
| fazalkhan | Can anybody help or point me to something? Thanks. | 12:18 |
| tosky | you should paste a bit more from the logs, and possibly setting them to debug level | 12:22 |
| tosky | (and using a pastebin of course) | 12:22 |
| *** esikachev has quit IRC | 12:22 | |
| *** tellesnobrega has joined #openstack-sahara | 12:25 | |
| *** esikachev has joined #openstack-sahara | 12:26 | |
| *** esikachev has quit IRC | 12:39 | |
| *** chlong_ has quit IRC | 12:43 | |
| *** esikachev has joined #openstack-sahara | 12:43 | |
| fazalkhan | @tosky Here is the current paste: https://pastebin.com/3b43YYt2 | 12:46 |
| tosky | fazalkhan: that's the stacktrace of the operation; what I was asking was for more details before | 12:48 |
| tosky | fazalkhan: did the instances spawned by sahara through nova reach the ACTIVE state? | 12:48 |
| fazalkhan | Yes, all instances are up, volumes created, attached and mounted. I can access all of them. | 12:49 |
| *** esikachev has quit IRC | 12:50 | |
| *** jeremyfreudberg has joined #openstack-sahara | 12:52 | |
| tosky | were the node group and cluster templates configured to access the instances through the public network? | 12:53 |
| fazalkhan | All instances have been auto assigned floating IPs of public network and using auto-genrated security groups. | 12:55 |
| tosky | so please paste more from the logs | 12:57 |
| jeremyfreudberg | fazalkhan, tosky, if provisioning fails at "Configuring step", then I would think it passed the "wait for instances accessible" step, so it's probably not related to network. (and if auto sec group, then that rules out sec group issue). so my guess is related to a specific CDH process not coming up properly | 12:57 |
| jeremyfreudberg | so, yes, more logs :) | 12:58 |
| fazalkhan | @tosky and @jeremyfreudberg You mean Sahara-engine and api logs with debugging set to true? | 12:59 |
| *** ukaynar has joined #openstack-sahara | 12:59 | |
| tosky | yes | 12:59 |
| jeremyfreudberg | fazalkhan give us as much context as you can now, but ideally with debug would be the best way to diagnose | 12:59 |
| *** lucasxu has joined #openstack-sahara | 12:59 | |
| *** esikachev has joined #openstack-sahara | 13:00 | |
| fazalkhan | okay. thanks. I will get back in ~10 minutes with the new logs. | 13:00 |
| *** links has quit IRC | 13:06 | |
| *** esikachev has quit IRC | 13:22 | |
| jeremyfreudberg | tosky, it's a one line change to start publishing oozie on tarballs.o.o again (some time ago we did do it), do you want me to push that? | 13:44 |
| tosky | jeremyfreudberg: yep! | 13:44 |
| tosky | the infra team would complain if there is something wrong | 13:44 |
| tosky | and the oozie tarballs are not so big | 13:44 |
| jeremyfreudberg | yep | 13:44 |
| *** openstackgerrit has joined #openstack-sahara | 13:46 | |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-extra master: Start building oozie again https://review.openstack.org/481638 | 13:46 |
| tosky | jeremyfreudberg: I wonder... why did we disable it? | 13:47 |
| tosky | I totally forgot | 13:47 |
| *** ukaynar has quit IRC | 13:47 | |
| jeremyfreudberg | tosky, according to some commit message from vitaly, we had only been building the same oozie 4.2.0 for a very long time, since we only needed that for Vanilla 2.7.1 (all other Vanillas had been removed by that time) | 13:48 |
| jeremyfreudberg | although, it still might be a good point that we don't actually need to build a new oozie each time, we could just put the binaries there once | 13:48 |
| tosky | jeremyfreudberg: did you check if the generated build matches the value used by sahara-image-elements? | 13:48 |
| jeremyfreudberg | good point too, I'll check soon | 13:49 |
| fazalkhan | @tosky @jeremyfreudberg here is the complete sahara-engine log: https://paste.ee/p/uC60L | 13:50 |
| jeremyfreudberg | thanks fazalkhan, I will take a look | 13:50 |
| fazalkhan | thank you guys. really appreciate your help. | 13:50 |
| jeremyfreudberg | looks like cloudera manager (port 7180) is our problem | 13:52 |
| tosky | fazalkhan: which flavor did you use for the host with cloudera manager? | 13:52 |
| tosky | also, did you assign other services to the host where cloudera manager is assigned? | 13:53 |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-extra master: Start building oozie again https://review.openstack.org/481638 | 13:53 |
| tosky | because cloudera manager, from my experience, should be in a separate node with 8 GiB, if no more, or RAM | 13:54 |
| tosky | of RAM* | 13:54 |
| fazalkhan | clouder manager host is using m1.medium flavor and only has cloudera_manager process assigned to it. | 13:57 |
| tosky | 4 GiB of RAM | 13:58 |
| tosky | too small | 13:58 |
| tosky | at least 8 GiB | 13:58 |
| fazalkhan | @tosky okay, I am trying again with 8GB for cloudera manager host. | 13:58 |
| *** chlong_ has joined #openstack-sahara | 14:08 | |
| *** esikachev has joined #openstack-sahara | 14:19 | |
| *** esikachev has quit IRC | 14:23 | |
| fazalkhan | @jeremyfreudberg On the cloudera-manager host, I have manually started the cloudera-scm-server but it is still not listening on port 7180. the httpd server starts listening on 80 and 443. | 14:25 |
| tosky | did you start it with the 8 GiB flavor? | 14:27 |
| fazalkhan | no I did that previously with the 4GB flavor. the deployment with 8GB flavor is still in progress. | 14:28 |
| tosky | I would suggest to postpone the testing until the 8GB deployment ends (or exits) | 14:29 |
| tosky | if cloudera manager was not started was because there was not enough memory - did you check its logs? | 14:30 |
| fazalkhan | @tosky thanks a lot sir. It has gone past the step where it was failing. | 14:35 |
| fazalkhan | @tosky it has now failed at cluster event> Plugin: start cluster (First run cluster) and logs say: Error during operating on cluster (reason: Failed to Provision Hadoop Cluster: Failed to format NameNode. | 14:41 |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-extra master: Start building oozie again https://review.openstack.org/481638 | 14:43 |
| jeremyfreudberg | fazalkhan, some debug log context around that error? | 14:43 |
| fazalkhan | @jeremyfreudberg Here sahara-enigne.log https://paste.ee/p/6Tm5L | 14:53 |
| fazalkhan | I am using Local storage on compute nodes (LVM) and assigning 4x volumes of 1TB each to each worker instance. it is failing to format those volumes. | 14:54 |
| *** anshul has quit IRC | 14:55 | |
| *** ukaynar has joined #openstack-sahara | 14:56 | |
| jeremyfreudberg | fazalkhan, I admit I do not see much solution off the top of my head, but I guess try with ephemeral instead of cinder if you can | 15:03 |
| jeremyfreudberg | sorry for the extra steps | 15:03 |
| *** lucasxu has quit IRC | 15:04 | |
| *** rcernin has quit IRC | 15:06 | |
| *** tellesnobrega has quit IRC | 15:07 | |
| fazalkhan | no problem @jeremyfreudberg. unfortunately, i have to use cinder. It is a permission related error - on the namenode instance, as "cloud-user", I cannot execute "hadoop namenode -format" but with sudo it gets executed successfully. | 15:08 |
| tosky | uhm, which images are you using? | 15:09 |
| *** sudipto has joined #openstack-sahara | 15:10 | |
| *** sudipto_ has joined #openstack-sahara | 15:10 | |
| fazalkhan | @tosky I am using sahara-newton-cloudera-5.7.0-centos7.qcow2 | 15:12 |
| tosky | we are going OT, but just to be sure, are you using OSP10 or RDO/Newton? | 15:13 |
| fazalkhan | It is OSP10 | 15:13 |
| tosky | then I tested the RHEL7-based image, rebuilt using sahara-image-elements | 15:14 |
| tosky | the instructions are part of the documentation of OSP10 | 15:14 |
| tosky | that said, the centos7 image should work as well | 15:15 |
| tosky | maybe still too low memory? | 15:20 |
| tosky | https://bugs.launchpad.net/sahara/+bug/1682359 | 15:20 |
| openstack | Launchpad bug 1682359 in Sahara "Creation of CDH Cluster fails if flavor is too small" [Undecided,Confirmed] | 15:20 |
| *** esikachev has joined #openstack-sahara | 15:20 | |
| fazalkhan | Okay, I will check the cloudera logs and will try with 16GB in the next attempt. | 15:22 |
| *** esikachev has quit IRC | 15:25 | |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-extra master: Start building oozie again https://review.openstack.org/481638 | 15:40 |
| *** abalutoiu has quit IRC | 15:42 | |
| *** anshul has joined #openstack-sahara | 15:48 | |
| *** mgoddard_ has quit IRC | 15:58 | |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-extra master: Start building oozie again https://review.openstack.org/481638 | 16:05 |
| jeremyfreudberg | tosky, I think either my brain is broken, or there is something I don't know about | 16:14 |
| jeremyfreudberg | regarding the artifacts publishing job | 16:14 |
| jeremyfreudberg | says file doesn't exist when it does, if you have some time you could take a look. (but I am afk for now) | 16:15 |
| *** jeremyfreudberg has quit IRC | 16:15 | |
| *** anshul has quit IRC | 16:19 | |
| *** esikachev has joined #openstack-sahara | 16:21 | |
| *** esikachev has quit IRC | 16:25 | |
| fazalkhan | @tosky It failed again with the error "Failed to format Namenode" with 16GB memory assigned to Cloudera Manager host. So, its probably a permission related error. | 16:31 |
| tosky | fazalkhan: try to rebuild the RHEL image according the OSP instructions | 16:31 |
| fazalkhan | @tosky you said that you tested with RHEL-7 based image, did it work fine with CDH 5.7? | 16:31 |
| fazalkhan | okay. | 16:31 |
| tosky | I did, using the RHEL image built using sahara-image-elements | 16:31 |
| fazalkhan | awesome. Thanks. | 16:32 |
| tosky | but I couldn't try with a 1 TB attached disk :) | 16:32 |
| tosky | just to be sure, while rebuilding the image, could you please try to attach smaller disks, just to be sure? | 16:32 |
| fazalkhan | I will let you know of my experiment. :) | 16:32 |
| fazalkhan | smaller as in? 200GB disks for all instances in the cluster? | 16:33 |
| tosky | and even no disk, just to be sure | 16:33 |
| tosky | try smaller than 100GB just to be sure | 16:33 |
| fazalkhan | oh okay. I can try that. I will try with no disks or 100GB first with RHEL-based image and then go on to increase the disk size. | 16:34 |
| tosky | I was suggesting to test no disk and/or small disk also with the current image, while rebuilding the new image | 16:35 |
| tosky | but whatever you can test, it's fine - it's your time | 16:35 |
| *** esikachev has joined #openstack-sahara | 16:40 | |
| *** tosky has quit IRC | 17:14 | |
| *** dave-mccowan has joined #openstack-sahara | 17:24 | |
| *** links has joined #openstack-sahara | 17:25 | |
| *** links has quit IRC | 17:29 | |
| *** links has joined #openstack-sahara | 17:30 | |
| *** tosky has joined #openstack-sahara | 17:32 | |
| *** sudipto has quit IRC | 17:35 | |
| *** sudipto_ has quit IRC | 17:35 | |
| *** links has quit IRC | 17:36 | |
| *** ukaynar has quit IRC | 17:41 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara master: Image generation for Ambari Plugin https://review.openstack.org/448714 | 18:09 |
| *** tellesnobrega has joined #openstack-sahara | 18:14 | |
| tellesnobrega | tosky, are you still around? | 18:16 |
| tellesnobrega | fazalkhan, hey, I wasn't around when you were with your problem, did you get it to work? | 18:17 |
| *** tesseract has quit IRC | 18:21 | |
| tosky | tellesnobrega: hi | 18:36 |
| tellesnobrega | did you guys figure out what was wrong with the cdh env? | 18:37 |
| tosky | the issue reported by fazalkhan ? | 18:38 |
| *** chlong_ has quit IRC | 18:40 | |
| tellesnobrega | yes | 18:48 |
| *** ukaynar has joined #openstack-sahara | 18:55 | |
| tosky | not completely | 19:14 |
| *** ukaynar has quit IRC | 19:20 | |
| *** ukaynar has joined #openstack-sahara | 19:21 | |
| *** fazalkhan has quit IRC | 19:29 | |
| *** ukaynar has quit IRC | 19:53 | |
| *** ukaynar has joined #openstack-sahara | 20:27 | |
| *** ukaynar has quit IRC | 20:38 | |
| *** ukaynar has joined #openstack-sahara | 20:39 | |
| *** abalutoiu has joined #openstack-sahara | 20:41 | |
| *** abalutoiu_ has joined #openstack-sahara | 20:43 | |
| *** abalutoiu has quit IRC | 20:46 | |
| *** abalutoiu has joined #openstack-sahara | 20:48 | |
| *** abalutoiu_ has quit IRC | 20:49 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara master: Image generation for Ambari Plugin https://review.openstack.org/448714 | 20:52 |
| *** ukaynar has quit IRC | 20:53 | |
| *** abalutoiu has quit IRC | 20:54 | |
| *** abalutoiu has joined #openstack-sahara | 20:55 | |
| *** abalutoiu_ has joined #openstack-sahara | 20:57 | |
| *** abalutoiu has quit IRC | 21:00 | |
| *** abalutoiu has joined #openstack-sahara | 21:07 | |
| *** abalutoiu_ has quit IRC | 21:09 | |
| *** abalutoiu has quit IRC | 21:21 | |
| *** dave-mccowan has quit IRC | 21:41 | |
| *** esikachev has quit IRC | 21:46 | |
| *** openstackgerrit has quit IRC | 21:47 | |
| *** openstackstatus has quit IRC | 21:56 | |
| *** openstack has joined #openstack-sahara | 21:59 | |
| *** esikachev has joined #openstack-sahara | 22:42 | |
| *** esikachev has quit IRC | 22:47 | |
| *** tosky has quit IRC | 22:53 | |
| *** abalutoiu has joined #openstack-sahara | 23:23 | |
| *** esikachev has joined #openstack-sahara | 23:43 | |
| *** abalutoiu has quit IRC | 23:45 | |
| *** abalutoiu has joined #openstack-sahara | 23:47 | |
| *** esikachev has quit IRC | 23:48 | |
| *** abalutoiu has quit IRC | 23:48 | |
| *** abalutoiu has joined #openstack-sahara | 23:49 | |
| *** abalutoiu_ has joined #openstack-sahara | 23:52 | |
| *** abalutoiu has quit IRC | 23:54 | |
| *** abalutoiu__ has joined #openstack-sahara | 23:56 | |
| *** abalutoiu has joined #openstack-sahara | 23:57 | |
| *** abalutoiu_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!