Need help debugging an apache server that locks up

Question

I'm new to apache, and I'm looking for someone to point me in the right direction.

We currently run an apache server to handle an internal, low-volume web interface for openstack (horizon) as well as an api server.

Recently we've run into issues where apache will 'lock up' (that is, it will stop responding to any requests for a time) when it is hit with a relatively high volume of api requests in a short time window.

Nothing shows up in the apache error logs, but I do see some apache processes running with much higher resident memory sets than normal:

As soon as the processes' resident memory returns to a normal level, apache starts responding properly again.

Here is the hardware usage during a 'lock up':

and here is the config file that apache is using:

LockFile ${APACHE_LOCK_DIR}/accept.lock

PidFile ${APACHE_PID_FILE}

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 5

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_event_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

AccessFileName .htaccess

<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

DefaultType None

HoetnameLookups Off

ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel warn

Include mods-enabled/*.load
Include mods-enabled/*.conf

Include httpd.conf

Include ports.conf

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Include conf.d/

Include sites-enabled/

Is there anything else I should be looking into to get to the root of the problem? Basically I'm looking to find out what needs to be changed, be that server hardware or apache config, in order to get the webserver working smoothly under it's current load. Any help would be greatly appreciated!

Answer 1

If you copy-pasted your config here you have a typo in it (HoetnameLookups should be HostnameLookups). It should hardly be the cause for your memory eating until swapping kicks in, but it may be worth a try.