Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Good day.

Lately my VPS server (CentOS on it) started crushing with "Too many open files in system" error. I read many on the error and know that the limit is set by my hosting provider. I received a list of limits from host provider and they say that the limit is 12000 files.

I tried looking for the problem using lsof utility. When the problem occured I managed to find those responces of lsof stat:

[root@XXXXXXXX]# lsof | wc -l 
3895

sometimes it went up to 4300 or so, but I never saw it jumping higher then this.

The question is stated like this: Can the lsof utility show not complete results, or is it the host's problem? If it;s lsof than what can I alternatively use to get maximum accuracy of the number.

share|improve this question

2 Answers 2

You can monitor /proc/sys/fs/file-nr with the tool of your choice, the most simple being cat /proc/sys/fs/file-nr - the first number shows the allocated file handles, the second one the allocated but unused filehandles, and the last number shows the maximum number of file handles.

That information is provided by the kernel itself.

share|improve this answer
    
Thanks for your answer! But cat /proc/sys/fs/file-nr gave me "510 0 262144" Which is even less than lsof |wc -l with current "2394". Maybe it does not work on VPS servers? –  Taras Voynarovsky Apr 13 '12 at 13:11
    
The particular virtue of file-nr is the first number won't drop after the files are closed. So you can get a sense of the peak number of open file handles over the uptime of your system. –  Mike Apr 13 '12 at 13:31

What's important is how your host measures the number of open files. Certainly /proc/sys/fs/file-nr is a great candidate, so +1 for that.

lsof includes "files" not counted in that total, however. I would be surprised if file-nr says that more file handles are open than lsof lists.

The other thing to bear in mind is the size of the file descriptor tables. Each process has a FD table, but there is also a system file table. Your host could have made the (frankly ridiculous) decision to calculate open files by the per-process FD table. You can see this as the FDSize field in /proc/<pid>/status for each process. It has to be a multiple of 2 in size, and is increased in size to the smallest multiple of 2 that will hold all open files. We can sum all of the FDSize entries. Again, this would be an unusual way to measure open files, but other than a process rapidly opening many files that rapidly increases your usage, I can't otherwise explain why their count is so much higher.

I used a script to sum the total FDSize from all open processes, and tried all three counts on two test systems (as root):

$ cat /proc/sys/fs/file-nr
544     0   12640
$ lsof | wc -l
1377
$ find /proc/ -maxdepth 1 -type d -regex '^/proc/[0-9]+$' -exec grep -Hi FDSize '{}'/status \; | cut -f 2 | awk '{total = total + $1}END{print total}'
5888

$ cat /proc/sys/fs/file-nr
8670    0   1587168
$ sudo /usr/sbin/lsof | wc -l
12309
$ find /proc/ -maxdepth 1 -type d -regex '^/proc/[0-9]+$' -exec grep -Hi FDSize '{}'/status \; | cut -f 2 | awk '{total = total + $1}END{print total}'
33088

You may be able to simply ask your host how they measure the open files. Really the FDSize is utter nonsense, I can't imagine they are doing that for real, but it's the only way I can think of to inflate my open files count.

share|improve this answer
    
Thanks for the script. It was helpful but still doesn't give me a hint as to why I would get this error when I have 752,000 set as my system max on Redhat 4.8. I wonder what other ways there are to troubleshoot this. Maybe its a Java JVM side-effect of some kind? –  djangofan Oct 22 '12 at 21:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.