0

The system is running ubuntu 10.04, running Apache/2.2.14 (Ubuntu)

It happened today where I noticed the server was extremely unresponsive (took 5+ minutes to log into SSH). Upon viewing the system resouces, it was obvious that Apache was the culprit (with 150 processes open), and I turned Apache off so I was able to work.

Looking at the error log, I found this

[Tue Jul 26 11:55:33 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Tue Jul 26 12:49:56 2011] [warn] child process 4814 still did not exit, sending a SIGTERM
[Tue Jul 26 12:49:58 2011] [warn] child process 4814 still did not exit, sending a SIGTERM
[Tue Jul 26 12:50:00 2011] [warn] child process 4814 still did not exit, sending a SIGTERM
[Tue Jul 26 12:50:02 2011] [error] child process 4814 still did not exit, sending a SIGKILL
[Tue Jul 26 12:50:03 2011] [error] could not make child process 4814 exit, attempting to continue anyway

These errors were appearing for about 150 different PID's (I just pulled these out for clairity).

I have the following mods enabled

alias.conf       authz_default.load    autoindex.conf  deflate.load  mime.conf         php5.conf        rewrite.load   ssl.load
alias.load       authz_groupfile.load  autoindex.load  dir.conf      mime.load         php5.load        setenvif.conf  status.conf
auth_basic.load  authz_host.load       cgi.load        dir.load      negotiation.conf  reqtimeout.conf  setenvif.load  status.load
authn_file.load  authz_user.load       deflate.conf    env.load      negotiation.load  reqtimeout.load  ssl.conf

I obviously would not like a repeat of this issue, and would like some insight as to what happened, and how to prevent it.

EDIT: Around the time of the issue occuring, the access log shows the requests being relativly light (a request or 2 every few seconds), and even then the requests were very lightweight.

3 Answers 3

3

Lower your MaxClients setting. You're probably spinning up more Apache processes than you have physical memory for. Once you run out of phsyical memory, you start dipping into swap and thrashing your disk as you page stuff out to virtual memory. This is a death spiral.

4
  • Its only set to 150 (the default?). I should add that I did a system upgrade yesterday, and I did not have this issue ever before
    – Steven1350
    Jul 26, 2011 at 18:43
  • 150 can be a ton of processes, especially depending on what they are doing.
    – Brad
    Jul 26, 2011 at 18:44
  • Just to throw the question out there, what would you say is a reasonable amount? The system has 2gb of RAM avail to it (about 1700mb free as of now). The server is not heavily used, and it seems weird that 150 requests where active at once
    – Steven1350
    Jul 26, 2011 at 19:10
  • It's possible someone was scanning or attempting to DoS you by doing incomplete connects to the web server. That said, if the server isn't heavily used, then there's nothing to lose by reducing MaxClients down to something more reasonable, such as 50, and see how things go.
    – anastrophe
    Jul 27, 2011 at 0:03
1

I have a VM serving about 100 sites, fairly light traffic, about 10G/day throughput. with these settings:

<IfModule prefork.c>
        StartServers 2
        MinSpareServers 4
        MaxSpareServers 8
        ServerLimit 75
        MaxClients 75
        MaxRequestsPerChild  1000
</IfModule>

purrs along quite nicely on a 4G memory limit uses about 2G ram - so yea, check your maxclients & post your apacheconfig as well

-sean

-sean

1
  • Gonna try these settings out, hopefully it helps
    – Steven1350
    Jul 26, 2011 at 19:59
0

If a process isn't responding to a SIGKILL, as indicated by the entries in your error log, then there is something extremely odd going on -- it's in an uninterruptable wait of some kind. There isn't enough information in your question to even begin to diagnose what the cause is, but I'd be looking into disk I/O related hardware faults, and if you're using NFS, make sure you're running with the intr mount option (on kernels earlier than 2.6.25, anyway).

2
  • It shouldnt be a disk I/O fault, as it is a virtual machine
    – Steven1350
    Jul 28, 2011 at 0:36
  • And your VM runs on virtual disks that are made of... what? Unicorn tears? It's not turtles all the way down; at some point you get to real physical hardware that has real physical faults.
    – womble
    Jul 28, 2011 at 1:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .