Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a situation where I'd like to run Hadoop spread across 2 clusters. The first cluster (ClusterA) is normal and all nodes are publicly accessible. The second cluster (ClusterB) is behind a NAT.

Nodes in ClusterA will be running both Mapred and HDFS, while nodes in ClusterB will be running Mapred without HDFS and will not be allowed to run Reduce Tasks. The master node (jobtracker, namenode, secondary namenode) will be in ClusterA.

My question is: if I start the ClusterB TaskTrackers independently without using bin/ from the JobTracker, will this setup work? TaskTrackers in ClusterB will open their own C&C connection to the JobTracker, and should receive MapTask assignments via this connection. HDFS will be entirely in ClusterA, so all nodes should be able to access chunks fine.

The only issue I can think of is Reduce tasks running in ClusterA attempting to get intermediate data stored on ClusterB nodes. Is this a push or a pull operation? Are there any other scenarios where the NAT will cause problems?

share|improve this question
up vote 0 down vote accepted

The answer is the Reduce tasks do pull and therefore need access through the NAT. Otherwise it appears to work.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.