How to configure linux file descriptor limit with fs.file-max and ulimit

Question

Server applications running on linux often require large quantities of open file handlers, for ex. HBase ulimit, Hadoop epoll limit

This wiki entry should serve as documentary for Linux file limits configuration.

• What is soft vs. hard limit?
• How to control the hard limit?
• How to control the soft limit?
• Does kernel fs.file-max and user ulimit -n related?

Please describe the Linux distribution under which your configuration is valid as various vendors configure stuff differently.

Thanks you, Maxim.

---

Update Based on lstvan answer:

For people looking to automate this, at least on Ubuntu servers you can put this in your machine installation scripts:

echo 'fs.file-max = 65000' > /etc/sysctl.d/60-file-max.conf
echo '* soft nofile 65000' > /etc/security/limits.d/60-nofile-limit.conf
echo '* hard nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf

Answer 1

Your operating system set limits on how many files can be opened by any running application on your host. You can extend the basic values usually 1024 easily by modifying 2 configuration files:

# vi /etc/sysctl.conf

fs.file-max = 32000

# vi /etc/security/limits.conf

youruser       soft    nofile   10000
youruser       hard    nofile   30000

The hard and soft limits:

man 5 limits.conf

hard
for enforcing hard resource limits. These limits are set by the superuser and 
enforced by the Kernel. The user cannot raise his requirement of system resources
above such values.

soft
for enforcing soft resource limits. These limits are ones that the user
can move up or down within the permitted range by any pre-exisiting hard 
limits. The values specified with this token can be thought of as default values,
for normal system usage.

HTH