Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

2
votes
0answers
7 views

In Hadoop, how to show current process of -copyFromLocal

I am still a newbie learner of Hadoop, and this time I was trying to process a 106GB file. I used -copyFromLocal to copy that big file to my Hadoop DFS, but since the file is big I have to wait for a ...
1
vote
0answers
12 views

When and how are initial directories created in HDFS

I have a Hadoop setup in which the configured HDFS umask is 027 instead of the default one. Some of the initially created directories have correct permissions (like tmp drwxrwxrwx) but others such as ...
0
votes
0answers
38 views

How to start hadoop on CentOS

I have installed hadoop using Yum on CentOS: yum install hadoop Now I want to start hadoop but cannot find the start-all.sh script mentioned on many blogs and wiki pages such as ...
0
votes
2answers
58 views

“/usr/bin/env: bash: No such file or directory” during puppet exec command

I am setting my first steps into puppet. I am trying to setup ambari. This is my puppet config: exec { "ambari-repo": command => "curl ...
0
votes
1answer
25 views

Cannot print hdfs topology when using viewfs:/// as fs.defaultFS

I am using Rack Awareness technology in HDFS, I can use the following command to get topology: hdfs dfsadmin -printTopology Today, after setting up HDFS Federation and use viewfs instead of hdfs ...
1
vote
0answers
39 views

Encounter an error when configuer secure hadoop : org.apache.hadoop.security.AccessControlException

I try to configure secure hadoop with kerberos. I have started KDC server , generated and copy related keytab to corresponding node. kerberos can work normally (use kinit) but when i try to start ...
0
votes
0answers
21 views

Hadoop - On the Wire Performance Monitoring?

I have been tasked with implementing an 'on the wire' monitoring solution for a large Hadoop installation. The source of data will be a combination of taps and SPANs throughout the environment. My ...
0
votes
0answers
15 views

Apache Kafka and Apache Hadoop installation on a single node

If Apache Kafka and Apache Hadoop are installed on a single node, should they share common ZooKeeper instance? If the answer is yes, what's the proper configuration of ZooKeeper? Thanks!
0
votes
2answers
61 views

packet loss between two specific VMs

I am experiencing some high-percentage packet loss (more than 70%) between two specific VMs in a local network, in a remote cluster (I have no physical access to it). One machine is used as a hadoop ...
0
votes
1answer
47 views

Hortonworks HDP 2.x Java error

I am trying to run the following command but I am getting the below error Could anyone explain it to me hdfs fsck / -files -blocks -locations > dfs-new-fsck-1.log I am trying to install ...
1
vote
1answer
93 views

Additional Storage Options for Hadoop HDFS Nodes

We have a small production Cloudera distribution Hadoop cluster(14 nodes, but growing). As we have expanded our usage of this cluster we have found that disk storage is our biggest blocker and ...
0
votes
1answer
48 views

CDH3 rebalancer without Cloudera Manager

We recently added new nodes to the cluster and I want to run the re-balancer to distribute the data among these nodes. We are using CDH3 and whole system is cheffed - I am not even sure if there is ...
1
vote
1answer
153 views

Changing ulimit on ubuntu 12.04 Never works

I am working with hadoop and need to change number of open files ulimit -n. I have seen similar questions on stackoverflow and elsewhere and have tried everything in those answers, but it still does ...
0
votes
1answer
37 views

Grid storage for large files on multiple external machines

I have several dedicated servers with unused space on them. 16GB+ RAM, some have SSD, other HDD. All have minimum 100Mbps connection. OS is Debian 7 64bit. There is no LAN connection between machines ...
0
votes
2answers
134 views

HDFS datanode startup fails when disks are full

Our HDFS cluster is only 90% full but some datanodes have some disks that are 100% full. That means when we mass reboot the entire cluster some datanodes completely fail to start with a message like ...
0
votes
1answer
118 views

Hadoop single node cluster memory and storage requirements

I have been advised to learn Hadoop for my company and so I want to build a test environment. The company I work for has a datacentre with a VMware vCloud infrastructure which they use to sell ...
-2
votes
1answer
45 views

What tool/application (free or not) should I use to generate and write lots (TBs) of data, fast [closed]

I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I ...
0
votes
1answer
543 views

Reverse and Forward DNS set up correctly but sometimes MapReduce job fails

Ever since we switched over our cluster to communicate via private interfaces and created a DNS server with correct forward and reverse lookup zones, we get this message before the M/R job runs: ...
0
votes
1answer
17 views

Setting up a Hive Remote Metastore with Kerberos

I'm new to hive and trying to set it up in a relatively secure manner for a test environment. I want to use a remote metastore so MR jobs can access the DB. I seem to have things almost working, but ...
1
vote
0answers
59 views

Installing Hadoop, ZooKeeper, AccumuLo on a single-node Mac for testing [closed]

I am attempting to install these technologies on a OS X loaded with Mountain Lion for testing purposes. The setup is a single-node setup using 'localhost'. I am running into a few issues... 1) ...
0
votes
1answer
130 views

Setting up multiple Hadoop clusters [closed]

Hopefully this question isn't off topic. I asked this on AskUbuntu but haven't gotten a responses: I've followed Michael Noll's excellent guide to get Hadoop set up on a single cluster: ...
2
votes
1answer
291 views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB ...
2
votes
0answers
763 views

How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused ...
0
votes
1answer
21 views

Is Elastic Workload Re-provisioning still possible with DataStax Enterprise 3.1?

In the documentation of DataStax Enterprise (DSE) 2.0 there is a chapter about "Elastic Workload Re-provisioning" (http://www.datastax.com/docs/datastax_enterprise2.0/workload_reprovisioning) but it ...
0
votes
0answers
111 views

Cannot find hadoop installation: $HADOOP_HOME must be set or hadoop must be in the path

So a little background. I've been trying to setup Hive on a CentOS 6 machine. I followed the instructions of this Youtube video: http://www.youtube.com/watch?v=L2lSrHsRpOI For my case, I'm using ...
0
votes
0answers
49 views

403 AccessDenied on Amazon S3 delete proctected folder

I have one folder in amazon s3 on which i restricted the permission of deleting any folder. But, now what happened is i am not able create any folder using hive script. This hive query create one ...
1
vote
0answers
193 views

Use Amazon SNS to send nagios alerts

Is there any way by which I can send nagios alerts to Amazon SNS ? I have tried with following steps, but its giving me this error in the Nagios log file: Jul 12 11:38:23 ip-10-134-13-204 nagios3: ...
0
votes
0answers
78 views

hadoop distcp command ignoring -m option, instead using only a single mapper

I'm trying to copy some data from hdfs to s3 and am having some trouble with distcp. I'm running a command that I've run in the past, but is now giving me trouble: hadoopc distcp -m 20 ...
2
votes
1answer
164 views

Hadoop - What is the purpose of the /usr/sbin/ shell scripts?

I am installing Hadoop 1.1.2 on CentOS 6.4. I read all the Hadoop documentation at http://hadoop.apache.org/docs/stable/ After installing, I noticed there are many shell scripts at /usr/sbin/. But ...
0
votes
0answers
141 views

configuration required for HIVE to be installed on a node

I went through the process of manually installing ambari (not through SSH, because I couldnt get keyless to work) and everything installed OK, except for HIVE and GANGLIA. I got this message: ...
-1
votes
2answers
495 views

how do I install pdsh on centos 6? [closed]

I'm following a tutorial on how to configure a centos machine to be a node in a hadoop cluster for HortonWorks. I'm doing this on a virtual machine on VirtualBox. Sadly, since I am a linux beginner, ...
1
vote
1answer
161 views

Hadoop - /usr/bin/hadoop: line 320: /usr/bin/java/bin/java: Not a directory

I am installing Hadoop on CentOS 6.4. Following these instructions http://hadoop.apache.org/docs/stable/single_node_setup.html wget ...
0
votes
1answer
36 views

How do I reconfigure the default port or uninstall the interactive Javascript console in HDInsight?

The Microsoft HDInsight Preview is available for download and install via the Web Platform Installer, which bundles Hadoop on Azure installer. As part of the process, it installs something that ...
0
votes
2answers
174 views

ext4 on faulty disks. How to avoid remount read-only?

The Problem: I'm in charge for a Hadoop cluster of 44 nodes. We have 1.5TB WD Green Drives with (quite unknown) the Load Cycle Count problem. These disks work fine but as they get older they show an ...
0
votes
0answers
64 views

How can I add a new hard disk in an existing HDFS cluster?

I have a running HDFS cluster consists of 4 nodes and the initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard drive /dev/vdb of 60GB size and want to add it into my existing HDFS ...
0
votes
1answer
181 views

Unable to read S3 file from interactive pig job flow

I'm unable to read a simple test file on S3 from an interactive pig job flow (hadoop, elastic map reduce), and I'm not sure why. I have two S3 buckets. Let's call them unmounted_bucket, and ...
0
votes
1answer
69 views

installing and configuring hadoop on ubuntu

I'm unable to locate the file hadoop-ec2-env.sh. I've downloaded and intalled hadoop_1.0.4-1_x86_64.deb from http://mirror.olnevhost.net/pub/apache/hadoop/common/stable/ This is the most recent ...
0
votes
1answer
86 views

setting up environment variables for aws hadoop ec2

I've been following this book: Hadoop in Action It gives a nice guide on how to start using ec2 with hadoop. One of the first things that it says is to download the command line tools here: ...
0
votes
1answer
74 views

Hadoop: Slave nodes are not starting

I am trying to setup a Pseudo Distributed Hadoop Cluster on my machine. Env Details : Host OS: Windows Guest OS: Ubuntu Vm's Created one master and one slave. I was able to run the hadoop ...
0
votes
1answer
2k views

How do do this in apt.conf - Acquire::http::Proxy “http://server:port/”;

Below are the instruction I have to follow for setting up cloudera. I am confused about the below on ubuntu 12.04. On Debian/Ubuntu systems, add the following property to /etc/apt/apt.conf: ...
9
votes
4answers
2k views

DIY Hadoop Cluster - Heat & Dust issues?

Following are links of my DIY 6-Node Hadoop Cluster using i3 Machines, What is the best possible way to protect my design from dust & provide better heat transfer? What should I use to cover ...
0
votes
1answer
58 views

Getting exposure to big data without having to set up the environment [closed]

Is anyone aware of any sandbox where an environment is already set up for big data processing? It can be hadoop, cassandra, pig etc... I'm a sql server programmer, and trying to get into big ...
0
votes
0answers
74 views

Cannot access data from hbase with amazone ec2

I have a single node Hadoop machine in which HBase is running in the Amazon EC2 instance. Due to some reason the server got restarted. So i need to start Hadoop and Hbase again. Then it's working ...
1
vote
1answer
98 views

MapReduce job is hung after 1 of 5 reducers completed on single-node environment

I have only one Data Node on my dev environment on EC2. I ran heavy MR job and in 6 hours noticed that 100% of mappers and 20% of reducers finished (1 of reducer shows 100% competition, other ones - ...
1
vote
2answers
546 views

Cloudera VM Image CDH4 for laptop without hardware virtualization

My laptop does not have hardware virtualization(no Intel VT-X flag in BIOS). So, I cannot use the Cloudera Image for Hadoop in VirtualBox as it is 64-bit. Reference: ...
1
vote
1answer
189 views

Java process failure (hadoop, hbase)

Anytime I am running hadoop/hbase process from a command prompt I get an error: /usr/local/hadoop/bin/hadoop: line 320: /usr/lib/jvm/jdk1.7.0/bin/java: cannot execute binary file ...
1
vote
1answer
128 views

hadoop: port appears open locally but not remotelly

I am new to linux and hadoop and I am having the same issue as in this question. I think I understand what is causing it but I don't know how to solve it (Don't know what they mean by "Edit the ...
1
vote
1answer
142 views

Programmatically configure Hadoop cluster from python

Are there any python API's to configure a hadoop cluster (namenode, jobtracker, etc..)setup on an OpenStack cloud?? I have the IP addresss to the VM's and am looking for openstack api's to configure ...
0
votes
1answer
139 views

Hadoop hardware disk choice [closed]

We're evaluating the options of setting up a large hadoop cluster. Now we actually have the choice to choose from these 3 setups: 300x server with 12x 1TB disk 150x server with 12x 2TB disk 100x ...
1
vote
1answer
49 views

Data lost after Hdfs client was killed

I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon. If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS. If I run the tool in ...