Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

As I am setting up Hadoop, one question keeps popping in my mind but I can't find the answer.

Which Hadoop configuration files need to be copied to which nodes. For example, I'm making changes to the following files:

hadoop-env.sh, core-site.xml, mapred-site.xml, hdfs-site.xml, masters, slaves

Do I need to copy these files to ALL my Hadoop nodes (which is kind of a pain if I update one file). Do only certain files need to be copied? Or, do I only need to make the changes on my master nodes?

Can't seem to find the answer anywhere, so I wanted to ask here. (Up to this point, I have been mirroring all the files across every node, but that seems inefficient. My setup does work.)

share|improve this question

1 Answer 1

up vote 0 down vote accepted

In terms of what reads which files:

  • hadoop-env.sh: Everything
  • core-site.xml: Everything
  • hdfs-site.xml: HDFS (NameNode, SecondaryNameNode, DataNode)
  • mapred-site.xml: MapReduce (JobTracker, TaskTracker)
  • masters and slaves: I don't think that these are read by the applications directly, but are used by the management scripts instead.

I would however suggest that setup a deployment system so you can easily distribute all these files to all nodes, instead of trying to figure out what needs what. This could just be a script which calls ssh with public key authentication, or it could be something like Puppet or Chef.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.