Our server recently ran out of file descriptors, and in regards to that I have some questions.
ulimit -n is supposed to give me the maximum number of open file descriptors. That number is 1024. I checked the number of open file descriptors by running
lsof -u root |wc -l and got 2500 fds. That is a lot more than 1024, so I guessed that would mean the number 1024 is per process, not per user, as I though. Well, I ran
lsof -p$PidOfGlassfish|wc -l and got 1300. This is the part I don't get. If
ulimit -n is not the maximum number of processes per user or per process, then what is it good for? Does it not apply to the root user? And if so, how could I then get the error messages about running out of file descriptor?
EDIT: The only way I can make sense out of
ulimit -n is if it applies the the number of open files (as stated in the bash manual) rather than the number of file handles (different processes can open the same file). If this is the case, then simply listing the number of open files (grepping on '/', thus excluding memory mapped files) is not sufficent:
lsof -u root |grep /|sort -k9 |wc -l #prints '1738'
To actually see the number of open files, I would need to filter on the name column on only print the unique entries. Thus the following is probably more correct:
lsof -u root |grep /|sort -k9 -u |wc -l #prints '604'
The command above expects output on the following format from lsof:
java 32008 root mem REG 8,2 11942368 72721 /usr/lib64/locale/locale-archive vmtoolsd 4764 root mem REG 8,2 18624 106432 /usr/lib64/open-vm-tools/plugins/vmsvc/libguestInfo.so
This at least gives me number less than 1024 (the number reported by
ulimit -n), so this seems like a step in the right direction. "Unfortunately" I am not experiencing any problems with running out of file descriptors, so I will have a hard time validating this.