Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using CentOS 5 on a VPS with 512MB memory. I find httpd often runs out of memory, which lags the whole system. But I cannot find out the reason.

When I use top to monitor processes, sometimes one of the httpd takes out 90+% memory, then SSH hangs up for seconds. When top refreshes again, that httpd has been killed by kernel. The issue repeats a few minutes later, again and again.

Here is the system log: http://pastebin.com/bszy9ahq

My Apache prefork conf:

<IfModule prefork.c>
StartServers 1
MinSpareServers 3
MaxSpareServers 7
ServerLimit      128
MaxClients       128
MaxRequestsPerChild 4000
</IfModule>

Here is httpd -V:

Server version: Apache/2.2.3
Server built:   Mar 24 2012 13:32:42
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Is there any way to know what script cause the memory issue, or just apache related? And how to resolve...

share|improve this question
1  
Turns out 640k isn't enough anymore. Neither is 512M. –  Tom O'Connor Apr 26 '12 at 7:26
    
I migrated from another 512M VPS some weeks ago. However almost the same sites and scripts worked fine in the past. –  Clippit Apr 26 '12 at 7:32
    
Are the servers identical? If i ran a directory comparison diff against them, would I find no differences? –  Tom O'Connor Apr 26 '12 at 7:33
    
I reinstalled everything on the new server and restored site files, so... you know –  Clippit Apr 26 '12 at 7:38
    
The answer is "no, then. –  Tom O'Connor Apr 26 '12 at 8:08
add comment

2 Answers

The system has run out of memory and has decided to kill apache to make room. It's not necessarily apache which is using all the memory. 512Mb might be tight for 128 httpd processes - but that figure can vary dramatically. Have you looked to see how much memory an httpd instance uses? Don't bother trying to work it out from a static snapshot - plot the memory usage (less buffers and cache) against the number of httpd instances and do a least squares regression - the slope is the amount of memory used per instance. Then find out how much memory the system uses when Apache is switched off. Depending on the amount of content being read from disk, you should aim for a minimum of ~20% memory free for cache/buffers.

Have you tuned your httpd.conf? Removing all redundant modules? Keepalives should be enabled but the timeout should be 2 seconds at most. Enable output compression and caching for static content.

If you suspect a memory leak in Apache then try reducing MaxRequestsPerChild.

sometimes one of the httpd takes out 90+% memory

This rather suggests a runaway script / module.

any way to know what script

Not from the information you've provided - there's nothing to tell us what scripting language you are using nor how it is invoked from the httpd. If it's PHP, then try adding an autoprepend script like....

<?php

register_shutdown_function('logmem');

function logmem()
{
    apache_note('x-php-peak-mem', memory_get_peak_usage());
}

and report the value in your access log:

"%{x-php-peak-mem}n"

Obviously this is only going to report requests which complete successfully - but it's a start.

Again, if it's PHP, then setting the memory limit too high (although this also applies to other OO languages) may be counter productive by suppressing garbage collection.

(BTW if this is running out of memory after a few minutes then "StartServers 1" is just plain silly)

share|improve this answer
add comment

To troubleshoot this:

  • Is your configuration still the same?
  • Is the amount of visitors still the same?
  • Have any scripts or modules been changed?

512 MB for Apache isn't a whole lot if you have a reasonably busy site. You might want to take a look at tweaking your config.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.