Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 0 down vote favorite

I have been trying to set up a Hadoop cluster; I managed to get it running in pseudo-distributed mode, and my one machine wordcounted Tolstoy's War and Peace in about thirty seconds.

I am now trying to add a second machine to my cluster; To help set it up, I created a user group Hadoop with permissions to start, stop, and run jobs on the Hadoop server (though I left editing the configuration files to root only). I ensured that all members of the group hadoop could ssh using their public keys from the master node to the slave node. I installed hadoop 1.0.0.3 using dpkg. I edited the masters and slaves files correctly on the master node and the slave node, and changed the configurations to point to the correct NameNode and JobTracker:

In core-site.xml:
fs.default.name=hdfs://$MASTER:9000

In mapred-site.xml:
mapred.job.tracker=$MASTER:9001

where $MASTER is the hostname of my master machine.

My NN, SNN, and JobTracker are starting correctly; however, my slave node is not able to connect to my master node! This is the behavior I see in my DataNode log:

2012-05-25 09:36:23,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: $MASTER/10.23.95.197:9000. Already tried 0 time(s).
2012-05-25 09:36:23,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: $MASTER/10.23.95.197:9000. Already tried 1 time(s).
...
...
connect to server: $MASTER/10.23.95.197:9000. Already tried 9 time(s).
2012-05-25 09:36:31,394 INFO org.apache.hadoop.ipc.RPC: Server at $MASTER/10.23.95.197:9000 not available yet, Zzzzz...

over and over and over again. I see the same thing in the TaskTracker log, except the port number listed there is 9001. lsof tells me that the correct processes are listening on both ports. What is going wrong???

All logs from $MASTER can be found at http://pastebin.com/ZzyKBQVJ

Thanks; please let me know if you have any quetions.

share|improve this question
What gives $ lsof -i:9000 ? – Yohann May 25 '12 at 16:58
1  
java PID 12324 TCP localhost:9000 LISTEN<br /> java PID 12324 TCP localhost:9000->localhost:52373 ESTABLISHED<br /> java PID 12598 TCP localhost:52373->localhost:9000 ESTABLISHED<br /> 12324 is my NameNode process, 12598 is a DataNode running on $MASTER. – ILikeFood May 25 '12 at 17:03
Can you past all logs of the start-all.sh command ? (edit or pastbin) ; Check this post. – Yohann May 25 '12 at 17:52
I've posted all relevant logs from my $MASTER's /var/log/hadoop directory. Note that the DataNode and the TaskTracker I have posted are the ones running on $MASTER and have no problems connecting; the issue is with the ones running on $SLAVE which have logs exactly as I specified in the OP. $MASTER = ncoiasi1 $SLAVE = ncoiasi2 – ILikeFood May 25 '12 at 18:24
Can you telnet from the machine which isn't working to 10.23.95.197:9000? – mgorven May 26 '12 at 4:40
show 1 more comment

1 Answer

up vote 0 down vote accepted

This issue is not usually caused by an issue in the Hadoop configuration, but more often in the network configuration of the cluster; in my case it was caused by this issue. If you are seeing this behavior, check your routing, /etc/hosts, etc. for problems before looking in the Hadoop files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.