I am working with hadoop and need to change number of open files
ulimit -n. I have seen similar questions on stackoverflow and elsewhere and have tried everything in those answers, but it still does not work. I am working with
ubuntu 12.04 LTS. Here is what i have done:
change limits in
/etc/security/limits.conf, i have put in settings for * and
root. I have also changed limits to some number like
* soft nofile 1513687 * hard nofile 1513687 root soft nofile 1513687 root hard nofile 1513687
I have also tried above setting with
- instead of
hard. After these changes, i have made changes to
/etc/pam.d/ files such as:
common-session common-session-noninterative login cron sshd su sudo
i have added
session required pam_limits.so to the beginning of each file. I made a restart of the box in question and the settings did not take effect.
I have also found that there were files inside
/etc/security/limits.d/ directory for the users
hdfs. I have tried changing limits in these individual files as well to no avail.
I have tried putting
ulimit -S -n unlimited inside
/etc/profile as well. It did not work.
Finally, i have tried putting
limit nofile unlimited unlimited inside
/etc/init.d/hadoop* files as first line. Did not work.
One interesting thing though is, i do not have hbase installed on the box but i do have a
hbase.conf file inside
/etc/security/limits.d/ directory. The settings in this file are reflected with
ulimit -n. But settings from
mapred.conf are not reflected. Which suggests that something is overwriting settings for
I guess i have tried everything people suggested on several forums, is there anything else that i may have missed or done incorrectly?
I am using
CDH 4.4.0 as my hadoop distribution.