While setting up hadoop, I found that hadoop-config.sh script is present in two directories, bin/ and libexec/. Both the files are identical. While looking onto scripts, I found that if hadoop-config.sh is present in libexec, then it gets executed. If not then the one under bin/ is called. What's the purpose of keeping the same file in two directories?


Thanks for pointing this out.I tried in my system

[root@prashant bin]# hadoop version
Hadoop 1.2.1

and you are absolutely correct

20:39:57.038706 open("/usr/local/hadoop/libexec/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000253>

When I moved this file then its referring to file inside /bin

20:42:51.024234 open("/usr/local/hadoop/bin/../conf/hadoop-env.sh", O_RDONLY) = 3 <0.000332>

Then I configured the latest version of hadoop

[root@hadoop libexec]# hadoop version
Hadoop 2.4.1

[root@hadoop libexec]# pwd

[root@hadoop libexec]# ls -l hadoop-config.sh
-rwxr-xr-x 1 67974 users 9793 Jun 21 11:35 hadoop-config.sh

[root@hadoop bin]# pwd

[root@hadoop bin]# ls -l hadoop-config.sh
ls: cannot access hadoop-config.sh: No such file or directory

So as far I see most of the script is moved out from bin in the latest version but still I dont have answer to your question


Historical accident? Developers building out things just to make them work? There hasn't been a huge amount of effort in the past in making sure the operational shell scripts used to start up all the java programs are well thought out and designed in a functional way.

Allen Wittenauer has been undertaking a monumental task of consolidating and fixing some of these issues in HADOOP-9902. You may wish to watch that JIRA issue.

If you're using a release from Apache, you'll probably seen the changes coming "soon". If you're using one of the commercial releases, I would expect for them to get the changes much later; at the very least, they'll probably wait until Allen's work is fully committed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.