up vote 1 down vote favorite
share [g+] share [fb]

I have a single node (pseudo-distributed config) and I'm considering adding a 2nd slave node. Does it matter if the slave has less HD capacity ? Will the rebalance take of that for itself. I'm not an HADOOP expert by far.

link|improve this question
40% accept rate
feedback

migrated from stackoverflow.com Jun 2 '11 at 17:53

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 1 down vote

No it doesn't matter but HDFS will not redistribute the blocks to the new node automatically so you will have to do that on your side. The easiest way is to run bin/start-balancer.sh. Also, before you do any rebalancing, make sure you modify your conf files accordingly to accommodate moving away from a pseudo-distributed configuration to a cluster one.

Check this question on the Hadoop FAQ for more ways to rebalance.

link|improve this answer
OK looks perfect to me. So I will use the start-balancer.sh first. I thought hadoop 'duplicates' content over nodes. – millebii May 7 '11 at 10:21
Actually Hadoop in a cluster configuration will maintain each block at as a replicate of three. That is by default the dfs.replication parameter in conf/hdfs-site.xml is set to 3. In your case you should set it to 2 and adjust it as you add more datanodes. – diliop May 9 '11 at 5:24
In my case that would meand then that the smaller configuration (the new node) will determine the size of the hdfs: this is what I was worried of please confirm. Thx – millebii May 9 '11 at 6:10
If the discrepancy between hard disk sizes is sizable and you are expecting to load up HDFS to capacity then yes, you will be handicapped by using replication at all. In that case you should set it to 1 and optimize your block size to the type of files you are distributing over HDFS. You will be giving up a lot in robustness and fault tolerance but if I am understanding your issue correctly, you are in a corner case and should be willing to take those hits no matter. – diliop May 9 '11 at 21:00
If the comments plus answer solved your problem, you mind marking this as solved else let me know where you are stuck and I'll help you solve it. – diliop May 10 '11 at 18:13
show 1 more comment
feedback
up vote 0 down vote

Hadoop will balance the load. In addition you can set "dfs.replication" property to set the number of replications you want.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.