Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am working with hadoop and need to change number of open files ulimit -n. I have seen similar questions on stackoverflow and elsewhere and have tried everything in those answers, but it still does not work. I am working with ubuntu 12.04 LTS. Here is what i have done:

change limits in /etc/security/limits.conf, i have put in settings for * and root. I have also changed limits to some number like 10000 and unlimited.

* soft nofile 1513687
* hard nofile 1513687
root soft nofile 1513687
root hard nofile 1513687

I have also tried above setting with - instead of soft and hard. After these changes, i have made changes to /etc/pam.d/ files such as:

common-session 
common-session-noninterative
login
cron
sshd
su
sudo

i have added session required pam_limits.so to the beginning of each file. I made a restart of the box in question and the settings did not take effect.

I have also found that there were files inside /etc/security/limits.d/ directory for the users hbase mapred and hdfs. I have tried changing limits in these individual files as well to no avail.

I have tried putting ulimit -S -n unlimited inside /etc/profile as well. It did not work.

Finally, i have tried putting limit nofile unlimited unlimited inside /etc/init.d/hadoop* files as first line. Did not work.

One interesting thing though is, i do not have hbase installed on the box but i do have a hbase.conf file inside /etc/security/limits.d/ directory. The settings in this file are reflected with ulimit -n. But settings from hdfs.conf and mapred.conf are not reflected. Which suggests that something is overwriting settings for hdfs and mapred.

I guess i have tried everything people suggested on several forums, is there anything else that i may have missed or done incorrectly?

I am using CDH 4.4.0 as my hadoop distribution.

share|improve this question

1 Answer 1

Did you reboot? I found the changes in /etc/security/limits.conf only work for me after reboot.

share|improve this answer
    
You can set the ulimits of running processes using prlimit on kernels >= 2.6.36. –  Matthew Ife Dec 28 '13 at 21:19
    
yes, i did reboot and it did not work. i will givr prlimit a try but ideally i want these settings to be permanent. –  Ravi Bhatt Dec 28 '13 at 22:17
    
What's the indicator that makes you determine that it does not work? –  ETL Dec 29 '13 at 1:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.