Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

0
votes
0answers
8 views

Hadoop cluster. Large number of TCP retransmissions

We have hadoop2 cluster including 2 namenodes. 13 datanodes and 1 'mediator' machine with DB for meta. Namenodes and datanodes are in the same data center, but meta DB is on remote side. When ...
0
votes
0answers
14 views

Hive 1.2.1. How to increase google.protobuf limits?

I have following error when trying to execute in Hive: Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, ...
0
votes
0answers
7 views

TEZ: DAG execution time vs Application execution time in Tez-ui

I have a situation when DAG execution time, displayed in tez-ui, is much different from Application execution time. For example DAG task may take about 2-3 minutes, but total App exec time would be ...
0
votes
0answers
7 views

Port Microsoft R Open for Hadoop to Hortonworks Cluster on Oracle Linux

Microsoft has released an enhanced version of R for Big Data, Microsoft R Open ("MRO"), and several related products and services. One of those is MRO for Hadoop - Red Hat Linux. I don't see a ...
0
votes
0answers
9 views

java process died with exit code 255 while installing hdp through ambari

I get this message when starting the ambari server...using postgres database The log file looks like .. The reference column name [resource_type_id] mapped on the element [field permissions] does ...
0
votes
0answers
13 views

SAP Lumira to HDFS Connection

I am trying to connect to HDFS filesystem from SAP Lumira but getting below error. HDFS URL is hdfs://IP-ADDR:8020/ hive jdbcdriver(500153) Error in set hive server type:null Any solution for ...
0
votes
0answers
8 views

Ambari log is filled with LdapTemplate warning

I'm running: # ambari-server --version 2.1.0 Specifically, I'm running IBM Open Platform for Hadoop 4.1 FP2. I configured Ambari to use LDAP for authentication, and the LDAP authentication is ...
0
votes
0answers
20 views

Force hadoop to use for S3 connection

I'm trying to upload file to S3 using hadoop: hadoop fs -Dfs.s3a.connection.ssl.enabled=false -Dfs.s3a.proxy.host=127.0.0.1 -Dfs.s3a.proxy.port=8123 -put pig_1421167148680.log ...
0
votes
0answers
45 views

HDP installation using ambari failing on CentOS 6.7

I am getting following error while installing HDP using ambari. Couldn't make out where the error lies,as error itself is not very clear. ========================== Creating target directory... ...
0
votes
2answers
67 views

Root directory full , can't recover consumed space

I have researched my scenario everywhere but can't find any string related to my issue. I have a datanode in Hadoop Framework , which recently went bad because all the drives on that box got umounted ...
1
vote
0answers
8 views

Hadoop Hive, Impala, Pig, and more — SQL access to Hadoop?

It appears that Hive, Impala, Pig, and others all provide SQL or SQL-like access to data stored on Hadoop clusters. They all seem to have support for HDFS, S3, and other forms. So why are there so ...
1
vote
0answers
26 views

hadoop fs commands not giving any output

I am running hadoop 1.2.1 on ubuntu 14.04 LTS in pseudo distributed mode, and the fs commands are not doing anything, i.e. niether prompt is returned nor any error message. what is the problem? Thanks ...
2
votes
2answers
152 views

How to remove RAID option from HP DL360 Gen 9 for HDFS

I am setting up a brand new DL360 G9 Server for use in a Hadoop cluster proof-of-concept. As HDFS will be taking care of the RAID, I need to bypass this option in the G9 array controller (Smart Array ...
0
votes
0answers
118 views

Missing ProgramDriver when running Hadoop 2.7.1 MapReduce examples jar file

I'm running the latest Hadoop version (2.7.1) on mint 17.2 in standalone mode. Everything seems to be working fine, but when I run the following command: ./hdfs -jar ...
8
votes
3answers
516 views

Best choice for NTP client configuration

Lets see if someone can throw a bit of light on this subject.. I'm making a server installation in the next days. My client wants to deploy a Hortonworks HDP with 2 servers as master servers and 5 ...
0
votes
1answer
142 views

Hive Server2 not impersonating HDFS

I am trying to secure Hive using storage based security. I am using Kerberos and LDAP. What I am trying to get is Hive to create directories and files as user (and their main group) in HDFS. This way ...
1
vote
1answer
808 views

Ambari server exits with no error message in the log

I have downloaded Hortonworks Data Platform 2.3.0 and installed it on Centos7. The installation was successful. When starting the server the following messages are displayed: [root@sparkperf-5360 ...
0
votes
1answer
36 views

Why does Accumulo require $ZOOKEEPER_HOME in addition to the IPs of the Zookeeper ensemble?

According to the documentation, Accumulo requires you to set $ZOOKEEPER_HOME (a local path) in the configuration files, and also requires a list of IPs for the Zookeeper ensemble. Why are IPs alone ...
1
vote
1answer
78 views

Hyper-V VLAN with static IP

Trying to setup/simulate Hadoop cluster locally via multiple (hyper-v) VMs. I'm using hortonworks sandbox image for hyper-v which runs centos linux. I can hit the vm, if I use an internal switch and ...
-1
votes
1answer
37 views

click and deploy hadoop cluster on google cloud platform

I am new to Google Cloud. I tried to click-and-deploy Hadoop cluster and I am always told that my quota are limited. But some days ago, I upgraded my free trial to a paid one ! (how can I check this ...
-2
votes
0answers
70 views

Where are the latest benchmarks on SQL-on-Hadoop performance? [closed]

I'm investigating adoption of Hadoop in my company, and a key requirement will be use of SQL for doing analytics. Hard to know exactly what the main usages will be but no doubt a mix of small and ...
0
votes
3answers
208 views

I/O and RAM limitations are important for Hadoop performance. But is disk speed related to I/O?

Hortonworks says this: "Most often performance of a Hadoop cluster will not be constrained by disk speed – I/O and RAM limitations will be more important." * How is disk speed not related to I/O ...
0
votes
0answers
15 views

Yarn autodetect slaves failure

This is something that I've found nowhere. I have a YARN cluster with some slaves. When a slave fails (chaos monkey, scale down, etc.), ResourceManager doesn't "get it". Even a rmadmin -refreshNodes ...
0
votes
0answers
56 views

Is HUE the friendliest front-end for Hadoop? [closed]

I recently became aware of HUE. It seems like a good Hadoop UI for non-technical people. Is this true? Might there be a better one? Thanks.
0
votes
1answer
23 views

store database on hadoop cluster

I'm learning Hadoop and Hive server, and I'm confused about something. Suppose I build a hadoop cluster with three machines, and I start storing images with a PHP/MySQL script. Now for a MySQL ...
0
votes
0answers
113 views

Setting dfs.data.dir when HADOOP_CONF_DIR is mounted via NFS

I have an educational hadoop cluster. I have configured HADOOP_CONF_DIR to read from NFS mounted share so I can have global configuration. Up until now my nodes were homogeneous and number of disks ...
0
votes
0answers
195 views

Flume- Error Log while using FileChannel

I am using Flume flume-ng-1.5.0 ( with CDH 5.4) to collect logs from many Servers and Sink to HDFS Here is my configuration : #Define Source , Sinks, Channel collector.sources = avro ...
1
vote
1answer
452 views

Should I deploy hadoop on physical machines or virtual machines?

We will deploy a hadoop cluster on hundreds(say 300) of physical x86 nodes. Since we have no much production deployment experience, there is a simple question as the title we want to hear response ...
1
vote
1answer
281 views

Hadoop namenode port getting blocked

I have installed 7 VM instances of Ubuntu 14.04 LTS servers. First instance runs the namenode service and all other 6 nodes run datanode service.I think my NameNode is getting crashed or blocked due ...
0
votes
0answers
38 views

in hadoop2.3.0 in windows all cmds worked correctly but it shows error

when i put the following command it shows the error like this C:\hadoop-2.3.0\bin>hadoop -namenode format Error: Could not find or load main class SUMATHY Sumathy is my pc's name. please help ...
0
votes
0answers
15 views

Application listing in hadoop resource manager

Every time resource manager restarted, previously run applications are getting cleared from Resource manager webUI (not in Mapreduce History server). Is there anyway to keep these finished application ...
0
votes
1answer
233 views

Setting up Secure Hadoop Cluster - Kerberos security

I setup a HDP 2.2 cluster successfully (1 NM, 3 DNs and 1 client). User accounts to access HDP cluster are created in client and checked these users can submit jobs, by SSH to client node and run ...
-1
votes
1answer
102 views

Hadoop on Openstack vs physical servers

I'm new to Hadoop and trying to understand how it should be installed/configured. From the documentation I see that Hadoop normally should be aware about physical servers configuration (e.g ...
-6
votes
1answer
184 views

What cause this error? [closed]

I am running two docker containers one is for hadoop basic services and other is for flume. Services are running successfully. I linked two containers env variables are automatically set by docker ...
2
votes
1answer
3k views

How to connect two docker containers running on the same host?

I have two docker containers running docker ps results CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0bfd25abbfc6 ...
3
votes
3answers
2k views

Is there a way to grep gzipped content in hdfs without extracting it?

I'm looking for a way to zgrep hdfs files something like: hadoop fs -zcat hdfs://myfile.gz | grep "hi" or hadoop fs -cat hdfs://myfile.gz | zgrep "hi" it does not really work for me is there ...
0
votes
1answer
645 views

Hadoop: How to configure failover time for a datanode

I need to re-replicate blocks on my HDFS cluster in case of a datanode is failing. Actually, this appears to already happen after a period of maybe 10min. However, I want to decrease this time, but ...
-1
votes
2answers
133 views

How to specify hard disk for hadoop cluster?

I installed Hadoop on one Azure VM, and it works fine by using its OS disk. However, I attached one hard disk to my VM, and I want to know how to configure Hadoop to only use this new hard disk as its ...
-2
votes
2answers
1k views

“sudo apt-get remove hadoop” is not removing package

I am trying to uninstall Cloudera Hadoop from my Ubuntu System. For this I tried sudo apt-get remove hadoop command but this command is failing with following message: ...
10
votes
2answers
334 views

Moving the SecondaryName Node in a Cloudera HBase Cluster

I deployed the secondary namenode on the same machine is my main namenode: This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy ...
0
votes
1answer
166 views

Ambari/Nagios overwrites hadoop-services.cfg file on startup

When I shut down Nagios from the Ambari Web UI, modify the file hadoop-services.cfg, save it and open it, new settings are there. However, when I start the Nagios again (from the Ambari Web UI) and ...
1
vote
0answers
227 views

How to force HDFS to use LDAP user's UID

I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP. I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
0
votes
1answer
133 views

Does Active Directory alone is not enough to secure hadoop?

I am trying to secure Hadoop environment installed in windows. So basically I started to analyse how to secure a Unix-based hadoop cluster. Have gone through various links related to Kerberos and ...
0
votes
1answer
306 views

How to access hadoop remotely

I have installed Hadoop on open-stack CentOS guest VM. I am able to open the site (From 192.168.0.10, VM-1) http://localhost:50070 http://192.168.0.10:50070 But not able to access the same from a ...
0
votes
1answer
56 views

How to know which script or executable is linked with a metric in ganglia?

I have just started to explore ganglia and my question is "How to know which script or executable is linked with a metric in ganglia?" The fact is that I don't know much about ganglia. I have good ...
1
vote
1answer
401 views

slaves get a connection timed out with hdfs

I have 3 node instances - master, slave1 and slave2 SSHing between these nodes works fine. Here is the processes that starts on each node when I say ./sbin/start-dfs.sh master: SecondaryNameNode ...
0
votes
1answer
53 views

Unable to connect to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
0
votes
2answers
78 views

Unable to setup a connection to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
1
vote
1answer
991 views

Rhadoop hdfs.init() Error

I recently installed CDH5.1.0 along with R 3.1.*, and I got rmr2, rJava, and rhdfs all installed properly. (along with the required packages and set the required environment variables) After some ...
0
votes
0answers
150 views

Distributing Master node ssh key

For the master node to passwordless-ly ssh into the slaves, the master needs to distribute its ssh key to the slaves. Copying key using ssh-copy-id asks for the user password. If there are hundreds of ...