Can no longer open files in linux server!

Question

We have a linux server running fine in prod for weeks, until our user base started to grow and our java server started to fail in many places! It turns out that all files and sockets are failing because it is returning "too many files open". I believe I have reached some sort of file opened limits.

How can I change this? The ulimit command is showing "unlimited".

Any other ideas?

---

I believe this limit is per process? Because I can still open files with say, vi, for example...

Here is the full ulimit output

ulimit  -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 4160
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Answer 1

Your limit on open files is set to 1024, not to unlimited. Change with ulimit -n for testing, and in /etc/security/limits.conf for production use.

Answer 2

SvenW is on the money.

Just to add, if you don't want to apply a new policy system wide; For instance the user spawning the Java process also performs other duties. It might well be root. Then you may wish to edit or create a wrapper for launching your Java app which performs ulimit -nX before launching. Obviously you should toy with the value of X to find a sensible number for your requirements.