Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

-1
votes
0answers
9 views

Why does Accumulo require $ZOOKEEPER_HOME in addition to the IPs of the Zookeeper ensemble?

According to the documentation, Accumulo requires you to set $ZOOKEEPER_HOME (a local path) in the configuration files, and also requires a list of IPs for the Zookeeper ensemble. Why are IPs alone ...
1
vote
1answer
28 views

Hyper-V VLAN with static IP

Trying to setup/simulate Hadoop cluster locally via multiple (hyper-v) VMs. I'm using hortonworks sandbox image for hyper-v which runs centos linux. I can hit the vm, if I use an internal switch and ...
-1
votes
1answer
24 views

click and deploy hadoop cluster on google cloud platform

I am new to Google Cloud. I tried to click-and-deploy Hadoop cluster and I am always told that my quota are limited. But some days ago, I upgraded my free trial to a paid one ! (how can I check this ...
-2
votes
0answers
32 views

Where are the latest benchmarks on SQL-on-Hadoop performance? [closed]

I'm investigating adoption of Hadoop in my company, and a key requirement will be use of SQL for doing analytics. Hard to know exactly what the main usages will be but no doubt a mix of small and ...
0
votes
2answers
35 views

I/O and RAM limitations are important for Hadoop performance. But is disk speed related to I/O?

Hortonworks says this: "Most often performance of a Hadoop cluster will not be constrained by disk speed – I/O and RAM limitations will be more important." * How is disk speed not related to I/O ...
0
votes
0answers
12 views

Yarn autodetect slaves failure

This is something that I've found nowhere. I have a YARN cluster with some slaves. When a slave fails (chaos monkey, scale down, etc.), ResourceManager doesn't "get it". Even a rmadmin -refreshNodes ...
0
votes
0answers
11 views

Is HUE the friendliest front-end for Hadoop? [closed]

I recently became aware of HUE. It seems like a good Hadoop UI for non-technical people. Is this true? Might there be a better one? Thanks.
0
votes
1answer
11 views

store database on hadoop cluster

I'm learning hadoop and hive server. I have a confusion. Suppose i build a hadoop server with three machine. and i start storing images with my php mysql script. Now for mysql database can i install ...
0
votes
0answers
57 views

Setting dfs.data.dir when HADOOP_CONF_DIR is mounted via NFS

I have an educational hadoop cluster. I have configured HADOOP_CONF_DIR to read from NFS mounted share so I can have global configuration. Up until now my nodes were homogeneous and number of disks ...
0
votes
0answers
73 views

Flume- Error Log while using FileChannel

I am using Flume flume-ng-1.5.0 ( with CDH 5.4) to collect logs from many Servers and Sink to HDFS Here is my configuration : #Define Source , Sinks, Channel collector.sources = avro ...
0
votes
1answer
82 views

Should I deploy hadoop on physical machines or virtual machines?

We will deploy a hadoop cluster on hundreds(say 300) of physical x86 nodes. Since we have no much production deployment experience, there is a simple question as the title we want to hear response ...
1
vote
1answer
67 views

Hadoop namenode port getting blocked

I have installed 7 VM instances of Ubuntu 14.04 LTS servers. First instance runs the namenode service and all other 6 nodes run datanode service.I think my NameNode is getting crashed or blocked due ...
0
votes
0answers
23 views

in hadoop2.3.0 in windows all cmds worked correctly but it shows error

when i put the following command it shows the error like this C:\hadoop-2.3.0\bin>hadoop -namenode format Error: Could not find or load main class SUMATHY Sumathy is my pc's name. please help ...
0
votes
0answers
10 views

Application listing in hadoop resource manager

Every time resource manager restarted, previously run applications are getting cleared from Resource manager webUI (not in Mapreduce History server). Is there anyway to keep these finished application ...
0
votes
1answer
141 views

Setting up Secure Hadoop Cluster - Kerberos security

I setup a HDP 2.2 cluster successfully (1 NM, 3 DNs and 1 client). User accounts to access HDP cluster are created in client and checked these users can submit jobs, by SSH to client node and run ...
-1
votes
1answer
76 views

Hadoop on Openstack vs physical servers

I'm new to Hadoop and trying to understand how it should be installed/configured. From the documentation I see that Hadoop normally should be aware about physical servers configuration (e.g ...
-7
votes
1answer
115 views

What cause this error? [closed]

I am running two docker containers one is for hadoop basic services and other is for flume. Services are running successfully. I linked two containers env variables are automatically set by docker ...
1
vote
1answer
886 views

How to connect two docker containers running on the same host?

I have two docker containers running docker ps results CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0bfd25abbfc6 ...
0
votes
0answers
96 views

Hadoop HA connection issue

I am using the Azure VM to build a Hadoop HA cluster, I'm trying to set up an Active namenode and a Standby namenode. But when I finished my configurations, I found I can only access one namenode via ...
2
votes
2answers
700 views

Is there a way to grep gzipped content in hdfs without extracting it?

I'm looking for a way to zgrep hdfs files something like: hadoop fs -zcat hdfs://myfile.gz | grep "hi" or hadoop fs -cat hdfs://myfile.gz | zgrep "hi" it does not really work for me is there ...
0
votes
1answer
325 views

Hadoop: How to configure failover time for a datanode

I need to re-replicate blocks on my HDFS cluster in case of a datanode is failing. Actually, this appears to already happen after a period of maybe 10min. However, I want to decrease this time, but ...
-1
votes
2answers
64 views

How to specify hard disk for hadoop cluster?

I installed Hadoop on one Azure VM, and it works fine by using its OS disk. However, I attached one hard disk to my VM, and I want to know how to configure Hadoop to only use this new hard disk as its ...
0
votes
0answers
99 views

which pig version is best for hadoop 2.5.0 with yarn?

I am using hadoop 2.5.0 with yarn enabled. I have used pig versions 0.11.0, 0.13.0, 0.14.0 with hadoop 2.5.0 but getting following error. 2015-01-06 19:21:26,835 FATAL [main] ...
-2
votes
2answers
848 views

“sudo apt-get remove hadoop” is not removing package

I am trying to uninstall Cloudera Hadoop from my Ubuntu System. For this I tried sudo apt-get remove hadoop command but this command is failing with following message: ...
9
votes
2answers
200 views

Moving the SecondaryName Node in a Cloudera HBase Cluster

I deployed the secondary namenode on the same machine is my main namenode: This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy ...
0
votes
1answer
113 views

Ambari/Nagios overwrites hadoop-services.cfg file on startup

When I shut down Nagios from the Ambari Web UI, modify the file hadoop-services.cfg, save it and open it, new settings are there. However, when I start the Nagios again (from the Ambari Web UI) and ...
1
vote
0answers
163 views

How to force HDFS to use LDAP user's UID

I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP. I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
0
votes
1answer
109 views

Does Active Directory alone is not enough to secure hadoop?

I am trying to secure Hadoop environment installed in windows. So basically I started to analyse how to secure a Unix-based hadoop cluster. Have gone through various links related to Kerberos and ...
0
votes
0answers
37 views

Hadoop lib syncronisation

I'm using Cloudera CDH5. I try to work with Twitter Anlyse . After building flume-sources-1.0-SNAPSHOT.jar and hive-serdes-1.0-SNAPSHOT.jar I placed this jars in manually in directories on all ...
0
votes
1answer
112 views

How to access hadoop remotely

I have installed Hadoop on open-stack CentOS guest VM. I am able to open the site (From 192.168.0.10, VM-1) http://localhost:50070 http://192.168.0.10:50070 But not able to access the same from a ...
0
votes
0answers
299 views

Error: E0701 : E0701: XML schema error while running custom executor

I am trying to test the custom executor sample found here http://www.infoq.com/articles/ExtendingOozie and another sample about ftp action for extending Oozie. In both cases I cannot run the test. ...
0
votes
0answers
484 views

Hadoop data node start fails with error: java: cannot execute: Permission denied

I am facing an error in starting a Hadoop Data Node from Ambari WEB interface, and I need your help to resolve this. As always the error message is very comlex and long, but first thing first. Even ...
0
votes
1answer
46 views

How to know which script or executable is linked with a metric in ganglia?

I have just started to explore ganglia and my question is "How to know which script or executable is linked with a metric in ganglia?" The fact is that I don't know much about ganglia. I have good ...
0
votes
0answers
142 views

Running multiple Hive metastores against a single HDFS

I'm investigating the possiblity of implementing dedicated Hive instances, each with their own metastore residing on their own work node outside of the cluster, but accessing the same data on a shared ...
1
vote
1answer
258 views

slaves get a connection timed out with hdfs

I have 3 node instances - master, slave1 and slave2 SSHing between these nodes works fine. Here is the processes that starts on each node when I say ./sbin/start-dfs.sh master: SecondaryNameNode ...
0
votes
1answer
43 views

Unable to connect to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
0
votes
2answers
63 views

Unable to setup a connection to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
1
vote
1answer
557 views

Rhadoop hdfs.init() Error

I recently installed CDH5.1.0 along with R 3.1.*, and I got rmr2, rJava, and rhdfs all installed properly. (along with the required packages and set the required environment variables) After some ...
0
votes
0answers
114 views

Distributing Master node ssh key

For the master node to passwordless-ly ssh into the slaves, the master needs to distribute its ssh key to the slaves. Copying key using ssh-copy-id asks for the user password. If there are hundreds of ...
2
votes
2answers
490 views

hadoop-config.sh in bin/ and libexec/

While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if ...
0
votes
2answers
2k views

error while running any hadoop hdfs file system command

I am very new to hadoop and referring to the "hadoop for dummies" book. I have a VM with following specs: hadoop version 2.0.6-alpha bigtop os centos The problem is when I run any hdfs file system ...
0
votes
0answers
152 views

hadoop hbase environment variables

I tried to setup a 4 node Hadoop cluster using CDH4.7. The cluster is up and running fine and when I submit a word count MR job it completed successfully but when I am submitting an MR job to insert ...
0
votes
1answer
245 views

Best practices - Conditionals for Chef resources without guard attributes?

We are setting up a cluster using Apache Ambari. Our Chef run is interrupted by the need to use Ambari to provision the Hadoop cluster. Current installs are a three part process: Initial Chef run ...
2
votes
0answers
121 views

Hbase block locality index is always 0

I have a Hbase (v 0.94.19 with Hadoop 1.2.1) setup with one master machine and two region servers. Each region server has 16 GB heap (6.4 GB cache, 4.0 GB memstore) and 1.6 TB (2 X 800 GB) SSD disk ...
0
votes
1answer
855 views

HDFS' ZKFC service unable to start

Introduction CDH4's ZooKeeper Failover Controller (ZKFC) has been installed. Starting the ZKFC service: [vagrant@localhost ~]$ sudo service hadoop-hdfs-zkfc start Starting ...
1
vote
2answers
814 views

Unable to convert HDFS from non-HA to HA

Introduction Aim: Convert HDFS from non-HA to HA. Method: According to this documentation it should be possible to convert HFDS from non- to HA by implementing following configuration: ...
4
votes
1answer
4k views

Command to determine whether a ZooKeeper Server is a Leader or a Follower

Introduction A ZooKeeper Quorum consisting of three ZooKeeper servers has been created. The zoo.cfg located on all three ZooKeeper servers looks as follows: maxClientCnxns=50 # The number of ...
4
votes
1answer
2k views

Could not start ZK at requested port of 2181, while export HBASE_MANAGES_ZK=false

Problem The first aim was to run HBase standalone. Navigating to ip:60010/master-status is succesfull once HBase has been started. The second aim is to run a distinct ZooKeeper quorum. ZooKeeper has ...
4
votes
1answer
180 views

Does Cloudera Manager need ongoing Root Access?

When installing Cloudera Manager 4, it asks for the root password on a passwordless sudo user to install packages. Does this account need to be retained, or is it just for initial setup?
0
votes
0answers
248 views

Can I change my hive external table storage format without affecting its partitions?

I want to change my external table storage format from text file to ORC file. I already added partitions around 500 for the external table stored as 'TEXTFILE'. If I simply create another external ...