How to configure linux file descriptor limit with fs.file-max and ulimit

Question

Server applications running on linux often require large quantities of open file handlers, for ex. HBase ulimit, Hadoop epoll limit

This wiki entry should serve as documentary for Linux file limits configuration.

• What is soft vs. hard limit?
• How to control the hard limit?
• How to control the soft limit?
• Does kernel fs.file-max and user ulimit -n related?

Please describe the Linux distribution under which your configuration is valid as various vendors configure stuff differently.

---

Update Based on lstvan answer:

For people looking to automate this, at least on Ubuntu servers you can put this in your machine installation scripts:

echo 'fs.file-max = 65000' > /etc/sysctl.d/60-file-max.conf
echo '* soft nofile 65000' > /etc/security/limits.d/60-nofile-limit.conf
echo '* hard nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf
echo 'root soft nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf
echo 'root hard nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf

Answer 1

Your operating system set limits on how many files can be opened by any running application on your host. You can extend the basic values usually 1024 easily by modifying 2 configuration files:

# vi /etc/sysctl.conf

fs.file-max = 32000

# vi /etc/security/limits.conf

youruser       soft    nofile   10000
youruser       hard    nofile   30000

The hard and soft limits:

man 5 limits.conf

hard
for enforcing hard resource limits. These limits are set by the superuser and 
enforced by the Kernel. The user cannot raise his requirement of system resources
above such values.

soft
for enforcing soft resource limits. These limits are ones that the user
can move up or down within the permitted range by any pre-exisiting hard 
limits. The values specified with this token can be thought of as default values,
for normal system usage.

HTH

Answer 2

If you're increasing the file limits on Ubuntu, you may use the asterisks to indicate 'all users' for soft and hard file limits. However, this will not include the root user, and hence you have to define it separately if you want the increased limits apply to the root user as well (and so you can set different limits "for the root user", and "for everybody else"), like so:

* soft nofile 4096
* hard nofile 10240
root soft nofile 4096
root hard nofile 10240

It is perhaps cleanest to add these into a separate file, such as /etc/security/limits.d/60-nofile-limit.conf