Apache crashing; "Too many open files in system"

Question

I am running Apache2 with mpm_prefork on a MediaTemple (dv) server. No significant changes have been made to my server configuration recently. httpd.conf is currently set to the following:

MaxKeepAliveRequests 200
KeepAliveTimeout 15

<IfModule prefork.c>
    StartServers           10
    MinSpareServers        10
    MaxSpareServers        10
    MaxClients            200
    MaxRequestsPerChild   4000
</IfModule>

But I am getting the following error just prior to a full-on apache crash:

[Thu Jun 04 18:30:24 2009] [warn-phpd] mmap cache can't open /var/www/vhosts/mydomain.com/httpdocs/filename.php - Too many open files in system (pid 19873)

I am getting annoyed with babysitting this server to restart it when Apache crashes. Help?

Answer 1

I suggest you try adding a line like this ulimit -n 16384 to the top of file /etc/default/apache2 and then restarting.

Also see this link.

Answer 2

from 'man proc':

/proc/sys/fs/file-max This file defines a system-wide limit on the number of open files for all processes. (See also setrlimit(2), which can be used by a process to set the per-process limit, RLIMIT_NOFILE, on the number of files it may open.) If you get lots of error messages about running out of file handles, try increasing this value:

          echo 100000 > /proc/sys/fs/file-max

          The  kernel constant NR_OPEN imposes an upper limit on the

value that may be placed in file-max.

          If you  increase  /proc/sys/fs/file-max,  be  sure  to 

increase /proc/sys/fs/inode-max to 3-4 times the new value of /proc/sys/fs/file-max, or you will run out of inodes.

   **/proc/sys/fs/file-nr**
          This (read-only)  file  gives  the  number  of  files 

presently opened. It contains three numbers: the number of allocated file handles; the number of free file handles; and the maximum number of file handles. The kernel allocates file handles dynamically, but it doesn't free them again. If the number of allocated files is close to the maximum, you should consider increasing the maximum. When the number of free file handles is large, you've encountered a peak in your usage of file handles and you probably don't need to increase the maximum.

The second number is worth looking at to see if the first number is what you need to increase. If so, you can set it in your /etc/sysctl.conf with:

fs.file-max=512000

Note that this sets the system limit; the per-user limit is set with 'ulimit', which you already indicated you were familiar with.

Answer 3

Check lsof to see what is actually going on. Exceeding an unreasonably high limit is often due to a leak or other bug.

Answer 4

MediaTemple is using Virtuozzo virtualization technology? Virtuozzo is built on top of OpenVZ. On OpenVZ there is a limit on number of open files. Maybe is Your containter hit the limit?

Run this command (if you can):

cat /proc/user_beancounters

and look at values of numfile resource.

Answer 5

Try running "ulimit -n 8192" before starting up apache. It's probably running into the max open file limit.

Answer 6

That's probably exactly what it sounds like -- your server has more files open than the kernel is configured to handle.

I assume you have a shared server? If you have root on the server, you could raise the limit for the Web server user with ulimit -n up to the max configured in /proc/sys/fs/file-max, but for a shared server you'll probably need to talk to your service provider.

It's also possible that the specific server you're on is overcrowded, assuming you share it, in which case MediaTemple will need to shuffle some of you around.

Alternatively, have you added any new features to your Web presence lately, or has your traffic increased a lot?

Answer 7

I've solved my problem by adding the following to /usr/sbin/apachectl :

ULIMIT_MAX_FILES="ulimit -n 16384"

The problem appears to have quieted down, at least for the past 12 hours.