Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

25
votes
3answers
22k views

What is Hadoop and what is it used for? [closed]

I have been enjoying reading ServerFault for a while and I have come across quite a few topics on Hadoop. I have had a little trouble finding out what it does from a global point of view. So my ...
10
votes
2answers
436 views

Moving the SecondaryName Node in a Cloudera HBase Cluster

I deployed the secondary namenode on the same machine is my main namenode: This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy ...
9
votes
4answers
3k views

DIY Hadoop Cluster - Heat & Dust issues?

Following are links of my DIY 6-Node Hadoop Cluster using i3 Machines, What is the best possible way to protect my design from dust & provide better heat transfer? What should I use to cover ...
8
votes
3answers
552 views

Best choice for NTP client configuration

Lets see if someone can throw a bit of light on this subject.. I'm making a server installation in the next days. My client wants to deploy a Hortonworks HDP with 2 servers as master servers and 5 ...
6
votes
1answer
1k views

Can a hadoop job be paused or suspended?

I'm using hadoop-0.20.2. Looking at hadoop fs. I am able to kill or fail an individual task. Is there anyway to pause it so that the map slots are freed up for another task?
6
votes
2answers
3k views

Hadoop disk fail, what do you do?

I would like to know about your strategies on what to do when one of the Hadoop server disk fails. Let's say, I have multiple (>15) Hadoop servers and 1 namenode, and one from 6 disks on slaves ...
5
votes
1answer
3k views

Is it possible to Managing 20 TB data using MySQL?

I am working in a project and my job is to build a database system to manage about 60,000,000,000 data entries. The project background is I have to do real-time storage for large number of messages ...
5
votes
3answers
2k views

In Hadoop, how to show current process of -copyFromLocal

I am still a newbie learner of Hadoop, and this time I was trying to process a 106GB file. I used -copyFromLocal to copy that big file to my Hadoop DFS, but since the file is big I have to wait for a ...
5
votes
1answer
4k views

Could not start ZK at requested port of 2181, while export HBASE_MANAGES_ZK=false

Problem The first aim was to run HBase standalone. Navigating to ip:60010/master-status is succesfull once HBase has been started. The second aim is to run a distinct ZooKeeper quorum. ZooKeeper has ...
5
votes
2answers
4k views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB disks)...
5
votes
1answer
925 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
5
votes
1answer
10k views

How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused ...
4
votes
4answers
10k views

Hadoop JBOD disk configuration on HP Smart Array 410/i disk controller

I'm in a evaluation phase of some hw that could be used for setting up a hadoop cluster. This hw is refurbished (hp G6 servers w/ Smart Array 410/i controller) and probably we should/must use it... we ...
4
votes
4answers
1k views

Hadoop cluster. 2 Fast, 4 Medium, 8 slower machines?

We're going to purchase some new hardware to use just for a Hadoop cluster and we're stuck on what we should purchase. Say we have a budget of $5k should we buy two super nice machines at $2500/each, ...
4
votes
1answer
265 views

Does Cloudera Manager need ongoing Root Access?

When installing Cloudera Manager 4, it asks for the root password on a passwordless sudo user to install packages. Does this account need to be retained, or is it just for initial setup?
4
votes
1answer
10k views

Can't connect to HDFS in pseudo-distributed mode

I followed the instructions here for installing hadoop in pseudo-distributed mode. However, I'm having trouble connecting to HDFS. When I execute this command : ./hadoop fs -ls / I get a ...
4
votes
1answer
3k views

Hadoop: Blacklisted tasktracker

I am running a Hadoop job (using Hadoop 0.20.2) on a 6 machine setup; one machine is the namenode / secondary node / job tracker (master) and the other 5 machines are all datanodes / tasktrackers (...
3
votes
3answers
4k views

Is there a way to grep gzipped content in hdfs without extracting it?

I'm looking for a way to zgrep hdfs files something like: hadoop fs -zcat hdfs://myfile.gz | grep "hi" or hadoop fs -cat hdfs://myfile.gz | zgrep "hi" it does not really work for me is there ...
3
votes
4answers
689 views

Interpreting exim log files after parsing

I'm parsing exim log files and, due to my processing method, lose the original order of all entries in this file. I rebuild the transactions by their transaction ID (i.e. 1OfiYX-0000Ev-7k) but still ...
3
votes
2answers
9k views

Hadoop HDFS: set file block size from commandline?

I need to set the block-size of a file when I load it into HDFS, to some value lower than the cluster block size. For example, if HDFS is using 64mb blocks, I may want a large file to be copied in ...
3
votes
1answer
657 views

Best practice for administering a (hadoop) cluster

I've recently been playing with Hadoop. I have a six node cluster up and running - with HDFS, and having run a number of MapRed jobs. So far, so good. However I'm now looking to do this more ...
3
votes
2answers
340 views

PXE boot Linux - which directories must be writable?

I'm planning to set up a small Hadoop cluster where the slave nodes boot and run from a central PXE server, to simplify deployment and updates, and to enable all of the disks on the slaves to be (...
3
votes
3answers
874 views

Hadoop ecosystem web dashboard

I am trying to find a tool, which would show me an overview of my Hadoop ecosystem - state, health, running tasks, etc. I tried to Google, but did not find any. Is there some nice useful tool?
3
votes
1answer
307 views

starting hadoop on mac os lion

I want to start hadoop on my macbook pro, I did all the steps that apache says. When I use the command "bin/start-all.sh", I get this: starting namenode, logging to /Users/alibozorgkhan/Desktop/...
3
votes
2answers
1k views

hadoop-config.sh in bin/ and libexec/

While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if hadoop-config....
2
votes
1answer
3k views

Is there any way to use arrays in a puppet module (not in template)?

I want to use puppet to manage a hadoop cluster. On the machines we have several directories which must be created and set permissions. But i'm unable to add array values for defined methods. define ...
2
votes
3answers
9k views

What is meant by “streaming data access” in HDFS?

According to the HDFS Architecture page HDFS was designed for "streaming data access". I'm not sure what that means exactly, but would guess it means an operation like seek is either disabled or has ...
2
votes
2answers
597 views

Deploy Hadoop to Openstack

I'd like to deploy Hadoop to Openstack cloud. Is there any automatic way to do that? Anyone tried to do that? I'm looking for some devops like juju. I've never used juju and right now I'm going ...
2
votes
1answer
3k views

How to connect two docker containers running on the same host?

I have two docker containers running docker ps results CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0bfd25abbfc6 ...
2
votes
1answer
344 views

Setting up hive with hadoop

I am trying to set up the hive. I am using this guide: https://cwiki.apache.org/Hive/gettingstarted.html and I'm stuck at setting up /tmp and /user/hive/warehouse dirs. First of all, it seems to me a ...
2
votes
2answers
4k views

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
2
votes
1answer
99 views

Most secure way to issue commands on ubuntu cluster with sudo'ing user?

This is sort of a follow-up question to an unanswered question I have regarding administration of Cloudera cluster, but I figure generalizing the question to all of Ubuntu may help me get an answer. ...
2
votes
1answer
556 views

Management of available file descriptors within a Hadoop cluster

I'm currently in charge of a rapidly-growing Hadoop cluster for my employer, currently built upon release 0.21.0 with CentOS as the OS for each worker and master node. I've worked through most of the ...
2
votes
2answers
188 views

Pairing MySQL and NoSQL Solutions

We have some fairly large datasets (user events and server log information - >100 GB) that's becoming fairly unwieldy for data processing. I've seen lots of activity around NoSQL/Hadoop/etc and I was ...
2
votes
2answers
296 views

How to remove RAID option from HP DL360 Gen 9 for HDFS

I am setting up a brand new DL360 G9 Server for use in a Hadoop cluster proof-of-concept. As HDFS will be taking care of the RAID, I need to bypass this option in the G9 array controller (Smart Array ...
2
votes
1answer
1k views

Does changing the default HDFS replication factor from 3 affect mapper performance?

Have a HDFS/Hadoop cluster setup and am looking into tuning. I wonder if changing the default HDFS replication factor (default:3) to something bigger will improve mapper performance, at the obvious ...
2
votes
3answers
113 views

Hadoop moving data to another user

I have a few hundreds of GB in my hdfs for userA (single node configuration). I would like to transfer all that data to userB wich will be more appropriate for the multi-node configuration I'm setting ...
2
votes
2answers
1k views

Is there a way to get a list of Hadoop cluster machines from one of the data nodes?

I have access to a data node in a Hadoop cluster, and I'd like to find out the identity of the name nodes for the same cluster. Is there a way to do this?
2
votes
1answer
646 views

Hadoop slaves file necessary?

I'm working on a team trying to create a system for creating Hadoop clusters on EC2 with minimal effort on the part of the user. Ideally, we would like slave instances to only require the hostname of ...
2
votes
2answers
3k views

Any good method for mounting Hadoop HDFS from another system?

I want to mount the Cloudera Hadoop as a Linux file system over the LAN. As a setup, I already have the hadoop cluster running on a set of Ubuntu machines. But now I need to be able to use it as a ...
2
votes
2answers
2k views

hadoop - datanode decommission

I want to remove nodes from my cluster gracefully. I added the following to my hadoop-site.xml: <property> <name>dfs.hosts.exclude</name> <value>/etc/hadoop/conf.dist/...
2
votes
0answers
221 views

Hbase block locality index is always 0

I have a Hbase (v 0.94.19 with Hadoop 1.2.1) setup with one master machine and two region servers. Each region server has 16 GB heap (6.4 GB cache, 4.0 GB memstore) and 1.6 TB (2 X 800 GB) SSD disk ...
2
votes
0answers
1k views

Encounter an error when configuer secure hadoop : org.apache.hadoop.security.AccessControlException

I try to configure secure hadoop with kerberos. I have started KDC server , generated and copy related keytab to corresponding node. kerberos can work normally (use kinit) but when i try to start ...
2
votes
1answer
592 views

Hadoop - What is the purpose of the /usr/sbin/ shell scripts?

I am installing Hadoop 1.1.2 on CentOS 6.4. I read all the Hadoop documentation at http://hadoop.apache.org/docs/stable/ After installing, I noticed there are many shell scripts at /usr/sbin/. But ...
1
vote
1answer
3k views

Raspberry Pi based Hadoop cluster [closed]

Is it at least possible to build Hadoop cluster from Raspberry Pi-based nodes? Can such a cluster meet hardware requirements of Hadoop? And if so, how much Raspberry Pi nodes are required to meet ...
1
vote
4answers
1k views

Disks for hadoop, what do you recommend? [closed]

what is you recommendation about disks for Hadoop? Do you recommend using SAS, or just attaching disk over SATA? Or maybe something else? What are pros and cons of every option? (Decision about disk ...
1
vote
1answer
8k views

How do do this in apt.conf - Acquire::http::Proxy “http://server:port/”;

Below are the instruction I have to follow for setting up cloudera. I am confused about the below on ubuntu 12.04. On Debian/Ubuntu systems, add the following property to /etc/apt/apt.conf: Acquire::...
1
vote
2answers
5k views

“/usr/bin/env: bash: No such file or directory” during puppet exec command

I am setting my first steps into puppet. I am trying to setup ambari. This is my puppet config: exec { "ambari-repo": command => "curl http://public-repo-1.hortonworks.com/ambari/suse11/1.x/...
1
vote
3answers
2k views

Hadoop - /usr/bin/hadoop: line 320: /usr/bin/java/bin/java: Not a directory

I am installing Hadoop on CentOS 6.4. Following these instructions http://hadoop.apache.org/docs/stable/single_node_setup.html wget http://apache.osuosl.org/hadoop/common/hadoop-1.1.2/hadoop-1.1.2-1....
1
vote
1answer
631 views

Implications of Multiple JobTracker nodes in a Hadoop cluster?

I get the impression that one can, potentially, have multiple JobTracker nodes configured to share the same set of MR (TaskTracker) nodes. I know that, conventionally, all the nodes in a Hadoop ...