While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if hadoop-config.sh is present in libexec, then it gets executed. If not then the one under bin/ is called. What's the purpose of keeping the same file in two directories?
Thanks for pointing this out.I tried in my system
[root@prashant bin]# hadoop version Hadoop 1.2.1
and you are absolutely correct
20:39:57.038706 open("/usr/local/hadoop/libexec/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000253>
When I moved this file then its referring to file inside /bin
20:42:51.024234 open("/usr/local/hadoop/bin/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000332>
Then I configured the latest version of hadoop
[root@hadoop libexec]# hadoop version Hadoop 2.4.1 [root@hadoop libexec]# pwd /usr/local/hadoop/libexec [root@hadoop libexec]# ls -l hadoop-config.sh -rwxr-xr-x 1 67974 users 9793 Jun 21 11:35 hadoop-config.sh [root@hadoop bin]# pwd /usr/local/hadoop/bin [root@hadoop bin]# ls -l hadoop-config.sh ls: cannot access hadoop-config.sh: No such file or directory
So as far I see most of the script is moved out from bin in the latest version but still I dont have answer to your question
Historical accident? Developers building out things just to make them work? There hasn't been a huge amount of effort in the past in making sure the operational shell scripts used to start up all the java programs are well thought out and designed in a functional way.
Allen Wittenauer has been undertaking a monumental task of consolidating and fixing some of these issues in HADOOP-9902. You may wish to watch that JIRA issue.
If you're using a release from Apache, you'll probably seen the changes coming "soon". If you're using one of the commercial releases, I would expect for them to get the changes much later; at the very least, they'll probably wait until Allen's work is fully committed.