Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running Apache2 with mpm_prefork on a MediaTemple (dv) server. No significant changes have been made to my server configuration recently. httpd.conf is currently set to the following:

MaxKeepAliveRequests 200
KeepAliveTimeout 15

<IfModule prefork.c>
    StartServers           10
    MinSpareServers        10
    MaxSpareServers        10
    MaxClients            200
    MaxRequestsPerChild   4000
</IfModule>

But I am getting the following error just prior to a full-on apache crash:

[Thu Jun 04 18:30:24 2009] [warn-phpd] mmap cache can't open /var/www/vhosts/mydomain.com/httpdocs/filename.php - Too many open files in system (pid 19873)

I am getting annoyed with babysitting this server to restart it when Apache crashes. Help?

share|improve this question
    
Does anything running on the server depend on an external web service? I had a mildly similar problem that appeared to come out of nowhere. The PHP behind a search box on my site was hitting an external web service to get its results. That web service became unreliable over time and eventually any time a user tried to search for something the PHP process would just idle indefinitely waiting for a response from the web service. Once enough people had triggered this I hit my process quota and no more pages could be fetched. –  Ben Dunlap Jun 5 '09 at 6:31

7 Answers 7

I suggest you try adding a line like this ulimit -n 16384 to the top of file /etc/default/apache2 and then restarting.

Also see this link.

share|improve this answer
    
This file does not exist on my server. What are other possible locations for it? –  martindale Jun 5 '09 at 2:03
1  
/etc/sysconfig/httpd on RHEL and Fedora –  Slashterix Jun 17 '10 at 20:37

from 'man proc':

/proc/sys/fs/file-max This file defines a system-wide limit on the number of open files for all processes. (See also setrlimit(2), which can be used by a process to set the per-process limit, RLIMIT_NOFILE, on the number of files it may open.) If you get lots of error messages about running out of file handles, try increasing this value:

          echo 100000 > /proc/sys/fs/file-max

          The  kernel constant NR_OPEN imposes an upper limit on the

value that may be placed in file-max.

          If you  increase  /proc/sys/fs/file-max,  be  sure  to

increase /proc/sys/fs/inode-max to 3-4 times the new value of /proc/sys/fs/file-max, or you will run out of inodes.

   **/proc/sys/fs/file-nr**
          This (read-only)  file  gives  the  number  of  files

presently opened. It contains three numbers: the number of allocated file handles; the number of free file handles; and the maximum number of file handles. The kernel allocates file handles dynamically, but it doesn't free them again. If the number of allocated files is close to the maximum, you should consider increasing the maximum. When the number of free file handles is large, you've encountered a peak in your usage of file handles and you probably don't need to increase the maximum.

The second number is worth looking at to see if the first number is what you need to increase. If so, you can set it in your /etc/sysctl.conf with:

fs.file-max=512000

Note that this sets the system limit; the per-user limit is set with 'ulimit', which you already indicated you were familiar with.

share|improve this answer

Check lsof to see what is actually going on. Exceeding an unreasonably high limit is often due to a leak or other bug.

share|improve this answer

MediaTemple is using Virtuozzo virtualization technology? Virtuozzo is built on top of OpenVZ. On OpenVZ there is a limit on number of open files. Maybe is Your containter hit the limit?

Run this command (if you can):

cat /proc/user_beancounters

and look at values of numfile resource.

share|improve this answer
    
Hey, thanks! This was superbly helpful. I found that MediaTemple's containers are limited to a total of 19200 file descriptors. I am adjusting my configs accordingly! –  martindale Jun 5 '09 at 12:52

Try running "ulimit -n 8192" before starting up apache. It's probably running into the max open file limit.

share|improve this answer
    
Negative, I've been through many iterations of this, including version where I've written it into my apachectl script. Thanks, though. –  martindale Jun 5 '09 at 1:44

That's probably exactly what it sounds like -- your server has more files open than the kernel is configured to handle.

I assume you have a shared server? If you have root on the server, you could raise the limit for the Web server user with ulimit -n up to the max configured in /proc/sys/fs/file-max, but for a shared server you'll probably need to talk to your service provider.

It's also possible that the specific server you're on is overcrowded, assuming you share it, in which case MediaTemple will need to shuffle some of you around.

Alternatively, have you added any new features to your Web presence lately, or has your traffic increased a lot?

share|improve this answer
    
Nope, we've got a dedicated virtual server (dv) - our resources are solely limited to our own virtual environment. No, we have not had any major increase in traffic, nor have we added any new features. –  martindale Jun 5 '09 at 2:04
    
Sounds like you've got the immediate problem figured out, but I would recommend thinking further about why you hit the limit -- you ought to be able to predict when it'll happen again by keeping an eye on your open file descriptors vs. the amount of traffic you're getting and extrapolating. –  Anonymous Jun 5 '09 at 23:00
up vote -1 down vote accepted

I've solved my problem by adding the following to /usr/sbin/apachectl :

ULIMIT_MAX_FILES="ulimit -n 16384"

The problem appears to have quieted down, at least for the past 12 hours.

share|improve this answer
    
"solved" is such a strong word. ;-) –  Ben Dunlap Jun 6 '09 at 1:21
    
Okay, I think the more appropriate term is "worked around". ;P More interestingly (and consequently, new Question coming soon...) I've noticed that we're still running into some intermittent issues relating to some tuning of MySQL. My theory is that some small tuning of MySQL caused more on-disk files to be put into use, which cascaded down into Apache issues. Don't quote me on this. –  martindale Jun 8 '09 at 17:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.