1

I am trying a small hadoop setup (for experimentation) with just 2 machines. I am loading about 13GB of data, a table of around 39 million rows, with a replication factor of 1 using Hive. My problem is hadoop always stores all this data on a single datanode. Only if I change the dfs_replication fatcor to 2 using setrep, hadoop copies data on the other node. I also tried the balancer ($HADOOP_HOME/bin/start-balancer.sh -threshold 0). The balancer recognizes that it needs to move around 5GB to balance. But says: "No block can be moved. Exiting..." and exits.

2010-07-05 08:27:54,974 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Using a threshold of 0.0 2010-07-05 08:27:56,995 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/10.252.130.177:1036 2010-07-05 08:27:56,995 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/10.220.222.64:1036 2010-07-05 08:27:56,996 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 1 over utilized nodes: 10.220.222.64:1036 2010-07-05 08:27:56,996 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 1 under utilized nodes: 10.252.130.177:1036 2010-07-05 08:27:56,997 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 5.42 GB bytes to make the cluster balanced.

Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved No block can be moved. Exiting... Balancing took 2.222 seconds

Can anybody suggest how to achieve even distribution of data on hadoop, without replication?

2 Answers 2

1

There are a few things about hadoop that may help you:

a) the first copy of a block is always written to the local node for processes running inside the cluster. Additional replicas are made elsewhere.

b) hadoop is designed for lots of nodes. That is why the default replication is 3. With small clusters, there are lots of potential hangups. If possible, you should try working with a cluster of at least 5-10 nodes, even for testing.

Based on (a), I would recommend running hive on a node outside your cluster.

1

This might not be the most optimized way to do it but here is how I will do it if my problem statement is to distribute data from one data node to several data nodes.

Load the data just like you have loaded and then run an indentity mapper with multiple reducers (around 4 should do the trick). Since you have 2 datanodes ( and I am guessing 2 tasktrackers), both tasktackers will be used to run the job will end up distributing data.

You must log in to answer this question.