Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

1
vote
0answers
288 views

How to force HDFS to use LDAP user's UID

I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP. I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
0
votes
1answer
144 views

Does Active Directory alone is not enough to secure hadoop?

I am trying to secure Hadoop environment installed in windows. So basically I started to analyse how to secure a Unix-based hadoop cluster. Have gone through various links related to Kerberos and ...
0
votes
1answer
508 views

How to access hadoop remotely

I have installed Hadoop on open-stack CentOS guest VM. I am able to open the site (From 192.168.0.10, VM-1) http://localhost:50070 http://192.168.0.10:50070 But not able to access the same from a ...
0
votes
1answer
59 views

How to know which script or executable is linked with a metric in ganglia?

I have just started to explore ganglia and my question is "How to know which script or executable is linked with a metric in ganglia?" The fact is that I don't know much about ganglia. I have good ...
1
vote
1answer
493 views

slaves get a connection timed out with hdfs

I have 3 node instances - master, slave1 and slave2 SSHing between these nodes works fine. Here is the processes that starts on each node when I say ./sbin/start-dfs.sh master: SecondaryNameNode ...
0
votes
1answer
53 views

Unable to connect to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
0
votes
2answers
83 views

Unable to setup a connection to Amazon EC2 and run pig

I have made an EC2 key pair and saved it to a location under my home directory on mac. Also I have changed permissions with 'chmod 600 /path/to/saved/keypair/file.pem'. Now I have followed the ...
1
vote
1answer
1k views

Rhadoop hdfs.init() Error

I recently installed CDH5.1.0 along with R 3.1.*, and I got rmr2, rJava, and rhdfs all installed properly. (along with the required packages and set the required environment variables) After some ...
0
votes
0answers
164 views

Distributing Master node ssh key

For the master node to passwordless-ly ssh into the slaves, the master needs to distribute its ssh key to the slaves. Copying key using ssh-copy-id asks for the user password. If there are hundreds of ...
3
votes
2answers
1k views

hadoop-config.sh in bin/ and libexec/

While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if hadoop-config....
0
votes
2answers
6k views

error while running any hadoop hdfs file system command

I am very new to hadoop and referring to the "hadoop for dummies" book. I have a VM with following specs: hadoop version 2.0.6-alpha bigtop os centos The problem is when I run any hdfs file system ...
0
votes
0answers
191 views

hadoop hbase environment variables

I tried to setup a 4 node Hadoop cluster using CDH4.7. The cluster is up and running fine and when I submit a word count MR job it completed successfully but when I am submitting an MR job to insert ...
0
votes
1answer
437 views

Best practices - Conditionals for Chef resources without guard attributes?

We are setting up a cluster using Apache Ambari. Our Chef run is interrupted by the need to use Ambari to provision the Hadoop cluster. Current installs are a three part process: Initial Chef run ...
2
votes
0answers
226 views

Hbase block locality index is always 0

I have a Hbase (v 0.94.19 with Hadoop 1.2.1) setup with one master machine and two region servers. Each region server has 16 GB heap (6.4 GB cache, 4.0 GB memstore) and 1.6 TB (2 X 800 GB) SSD disk ...
0
votes
1answer
2k views

HDFS' ZKFC service unable to start

CDH4's ZooKeeper Failover Controller (ZKFC) has been installed. Starting the ZKFC service: [vagrant@localhost ~]$ sudo service hadoop-hdfs-zkfc start Starting Hadoop zkfc: ...
1
vote
2answers
2k views

Unable to convert HDFS from non-HA to HA

Introduction Aim: Convert HDFS from non-HA to HA. Method: According to this documentation it should be possible to convert HFDS from non- to HA by implementing following configuration: /etc/hadoop/...
5
votes
1answer
4k views

Could not start ZK at requested port of 2181, while export HBASE_MANAGES_ZK=false

Problem The first aim was to run HBase standalone. Navigating to ip:60010/master-status is succesfull once HBase has been started. The second aim is to run a distinct ZooKeeper quorum. ZooKeeper has ...
4
votes
1answer
271 views

Does Cloudera Manager need ongoing Root Access?

When installing Cloudera Manager 4, it asks for the root password on a passwordless sudo user to install packages. Does this account need to be retained, or is it just for initial setup?
0
votes
0answers
333 views

Can I change my hive external table storage format without affecting its partitions?

I want to change my external table storage format from text file to ORC file. I already added partitions around 500 for the external table stored as 'TEXTFILE'. If I simply create another external ...
1
vote
1answer
1k views

Cloudera Hadoop superuser group

I'm trying to create a group on one of my datanodes that will have superuser privileges for hdfs and associated fs commands. So far I have: Checked to see that dfs.permissions.superusergroup=...
6
votes
3answers
2k views

In Hadoop, how to show current process of -copyFromLocal

I am still a newbie learner of Hadoop, and this time I was trying to process a 106GB file. I used -copyFromLocal to copy that big file to my Hadoop DFS, but since the file is big I have to wait for a ...
1
vote
0answers
44 views

When and how are initial directories created in HDFS

I have a Hadoop setup in which the configured HDFS umask is 027 instead of the default one. Some of the initially created directories have correct permissions (like tmp drwxrwxrwx) but others such as /...
0
votes
0answers
286 views

How to start hadoop on CentOS

I have installed hadoop using Yum on CentOS: yum install hadoop Now I want to start hadoop but cannot find the start-all.sh script mentioned on many blogs and wiki pages such as http://wiki.apache....
1
vote
2answers
5k views

“/usr/bin/env: bash: No such file or directory” during puppet exec command

I am setting my first steps into puppet. I am trying to setup ambari. This is my puppet config: exec { "ambari-repo": command => "curl http://public-repo-1.hortonworks.com/ambari/suse11/1.x/...
0
votes
1answer
405 views

Cannot print hdfs topology when using viewfs:/// as fs.defaultFS

I am using Rack Awareness technology in HDFS, I can use the following command to get topology: hdfs dfsadmin -printTopology Today, after setting up HDFS Federation and use viewfs instead of hdfs ...
2
votes
0answers
1k views

Encounter an error when configuer secure hadoop : org.apache.hadoop.security.AccessControlException

I try to configure secure hadoop with kerberos. I have started KDC server , generated and copy related keytab to corresponding node. kerberos can work normally (use kinit) but when i try to start ...
0
votes
0answers
101 views

Hadoop - On the Wire Performance Monitoring?

I have been tasked with implementing an 'on the wire' monitoring solution for a large Hadoop installation. The source of data will be a combination of taps and SPANs throughout the environment. My ...
0
votes
2answers
319 views

packet loss between two specific VMs

I am experiencing some high-percentage packet loss (more than 70%) between two specific VMs in a local network, in a remote cluster (I have no physical access to it). One machine is used as a hadoop ...
0
votes
1answer
556 views

Hortonworks HDP 2.x Java error

I am trying to run the following command but I am getting the below error Could anyone explain it to me hdfs fsck / -files -blocks -locations > dfs-new-fsck-1.log I am trying to install ...
1
vote
1answer
325 views

Additional Storage Options for Hadoop HDFS Nodes

We have a small production Cloudera distribution Hadoop cluster(14 nodes, but growing). As we have expanded our usage of this cluster we have found that disk storage is our biggest blocker and ...
0
votes
1answer
222 views

CDH3 rebalancer without Cloudera Manager

We recently added new nodes to the cluster and I want to run the re-balancer to distribute the data among these nodes. We are using CDH3 and whole system is cheffed - I am not even sure if there is ...
1
vote
1answer
533 views

Changing ulimit on ubuntu 12.04 Never works

I am working with hadoop and need to change number of open files ulimit -n. I have seen similar questions on stackoverflow and elsewhere and have tried everything in those answers, but it still does ...
0
votes
1answer
51 views

Grid storage for large files on multiple external machines

I have several dedicated servers with unused space on them. 16GB+ RAM, some have SSD, other HDD. All have minimum 100Mbps connection. OS is Debian 7 64bit. There is no LAN connection between machines (...
0
votes
4answers
2k views

HDFS datanode startup fails when disks are full

Our HDFS cluster is only 90% full but some datanodes have some disks that are 100% full. That means when we mass reboot the entire cluster some datanodes completely fail to start with a message like ...
0
votes
1answer
1k views

Hadoop single node cluster memory and storage requirements

I have been advised to learn Hadoop for my company and so I want to build a test environment. The company I work for has a datacentre with a VMware vCloud infrastructure which they use to sell ...
-2
votes
1answer
63 views

What tool/application (free or not) should I use to generate and write lots (TBs) of data, fast [closed]

I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I ...
0
votes
1answer
2k views

Reverse and Forward DNS set up correctly but sometimes MapReduce job fails

Ever since we switched over our cluster to communicate via private interfaces and created a DNS server with correct forward and reverse lookup zones, we get this message before the M/R job runs: ...
0
votes
1answer
90 views

Setting up a Hive Remote Metastore with Kerberos

I'm new to hive and trying to set it up in a relatively secure manner for a test environment. I want to use a remote metastore so MR jobs can access the DB. I seem to have things almost working, but ...
0
votes
1answer
178 views

Setting up multiple Hadoop clusters [closed]

Hopefully this question isn't off topic. I asked this on AskUbuntu but haven't gotten a responses: I've followed Michael Noll's excellent guide to get Hadoop set up on a single cluster: http://www....
5
votes
2answers
4k views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB disks)...
5
votes
1answer
10k views

How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused ...
0
votes
1answer
31 views

Is Elastic Workload Re-provisioning still possible with DataStax Enterprise 3.1?

In the documentation of DataStax Enterprise (DSE) 2.0 there is a chapter about "Elastic Workload Re-provisioning" (http://www.datastax.com/docs/datastax_enterprise2.0/workload_reprovisioning) but it ...
1
vote
0answers
697 views

Use Amazon SNS to send nagios alerts

Is there any way by which I can send nagios alerts to Amazon SNS ? I have tried with following steps, but its giving me this error in the Nagios log file: Jul 12 11:38:23 ip-10-134-13-204 nagios3: ...
2
votes
1answer
597 views

Hadoop - What is the purpose of the /usr/sbin/ shell scripts?

I am installing Hadoop 1.1.2 on CentOS 6.4. I read all the Hadoop documentation at http://hadoop.apache.org/docs/stable/ After installing, I noticed there are many shell scripts at /usr/sbin/. But ...
-1
votes
2answers
3k views

how do I install pdsh on centos 6? [closed]

I'm following a tutorial on how to configure a centos machine to be a node in a hadoop cluster for HortonWorks. I'm doing this on a virtual machine on VirtualBox. Sadly, since I am a linux beginner, ...
1
vote
3answers
2k views

Hadoop - /usr/bin/hadoop: line 320: /usr/bin/java/bin/java: Not a directory

I am installing Hadoop on CentOS 6.4. Following these instructions http://hadoop.apache.org/docs/stable/single_node_setup.html wget http://apache.osuosl.org/hadoop/common/hadoop-1.1.2/hadoop-1.1.2-1....
0
votes
1answer
58 views

How do I reconfigure the default port or uninstall the interactive Javascript console in HDInsight?

The Microsoft HDInsight Preview is available for download and install via the Web Platform Installer, which bundles Hadoop on Azure installer. As part of the process, it installs something that ...
0
votes
2answers
776 views

ext4 on faulty disks. How to avoid remount read-only?

The Problem: I'm in charge for a Hadoop cluster of 44 nodes. We have 1.5TB WD Green Drives with (quite unknown) the Load Cycle Count problem. These disks work fine but as they get older they show an ...
0
votes
1answer
1k views

Unable to read S3 file from interactive pig job flow

I'm unable to read a simple test file on S3 from an interactive pig job flow (hadoop, elastic map reduce), and I'm not sure why. I have two S3 buckets. Let's call them unmounted_bucket, and ...
0
votes
1answer
83 views

installing and configuring hadoop on ubuntu

I'm unable to locate the file hadoop-ec2-env.sh. I've downloaded and intalled hadoop_1.0.4-1_x86_64.deb from http://mirror.olnevhost.net/pub/apache/hadoop/common/stable/ This is the most recent ...