Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have nutch/hadoop pseudo distributed running fine. I want to add processing capacity by adding new nodes which are smaller than master (HD 3 times smaller) and cheaper of course.

Since the default HDFS replication is at 3, after balancing the data I will not get more space, which is not my concern first.

Do I still get more processing power ?

I don't understand how map/reduce tasks work against replication. How is it decided which nodes gets the work out of the different replica.

share|improve this question

migrated from stackoverflow.com Jun 1 '11 at 17:35

This question came from our site for professional and enthusiast programmers.

2 Answers 2

You will have to move to a cluster setup from your pseudocluster setup and by doing so, you will indeed get more processivity out of your cluster by adding more nodes i.e. you will be able to process more map and reduce tasks. The processivity increase as you would expect is linear.

Replication will determine the number of replicates that are present in your cluster for each HDFS block. So lets assume that you have a file that is split into 6 blocks, for a replication of 3, 18 blocks will be spread out in your cluster. The more nodes you have the higher coverage you will get and thus when it comes down to commencing your map phase, less data will have to be transfered between datanodes. And to answer your final question, Hadoop will always try to assign map tasks to nodes that serve as datanodes for the input to those map tasks. So in this case, the replicates will make this task easier since there will be a larger pool of tasktrackers to choose from.

share|improve this answer
    
@diliop so the task is assigned to one and only one tasktracker out of the replicates right ? The job tracker does that presumably ? –  millebii May 22 '11 at 7:53
    
You seem to be confusing the associations between split, map task input and block replicate. Every map task gets as input a single split and is run on a single tasktracker. Every split is usually (and by default) equal to the HDFS block size. Blocks are the ones that are replicated and for each file, a single replicate will only be used as input. Replication's main purpose of existence if fault tolerance. And yed, the jobtracker is responsible for generating the splits and scheduling tasks on tasktrackers. –  diliop May 23 '11 at 0:29
    
Was this helpful or are you still having issues? –  diliop May 26 '11 at 0:32
    
Yes & no, I think I need to understand the concept of split better and read a few things about it. I just have a few priorities on top before I can test and tick back. –  millebii May 26 '11 at 20:40
    
Just to help you in that process, I would strongly advise you to pick up a copy of Hadoop: The definitive guide (oreilly.com/catalog/9780596521981) and read the first 5-6 chapters. –  diliop May 26 '11 at 20:44

Your question is a bit confusing. If you're running in pseudo-distributed mode, then that's where all four processes (JobTracker, NameNode, DataNode, TaskTracker) are all launched on the same (typically development) system.

The Hadoop xxx-site.xml configuration for pseudo-distributed has everything talking to localhost, thus adding new nodes won't work.

Leaving that aside, if you are adding more nodes, and these are running both DataNodes and TaskTrackers, then you will get added storage and CPU capacity.

Note that as you fill up HDFS, eventually the 3x replication won't be possible when all of the smaller nodes are at capacity, so you'll start getting warnings/errors.

share|improve this answer
    
@kkrugler will these warnings/errors be a problem ? –  millebii May 22 '11 at 7:48
    
@kkrugler I will have to change the pseudo-distributed conf in a cluster conf, but before I do that I want to understand what will happen next. –  millebii May 22 '11 at 7:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.