Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a MediaTemple (dv) 4 server with 512mb running Apache 2.2.21 64-bit.

Within 60 seconds of starting, httpd is spawning 2-4 processes, each consuming ~150mb of memory (according to top's 'RES' column). This very quickly crashes my server. By 'crashes' I mean all urls start timing out, as a result I cannot log into the attached plask panel. Additionally, ssh no longer works (connection times out) and the real-time stats provided by MediaTemple stop working also. So far the only fix I've found is to restart the server. Apache is running PHP (Zend Engine 2.3.0) serving a simple Drupal install.

I've tried a few suggestions on the media temple wiki with no improvement.

  1. What could be causing httpd to consume this much memory?
  2. Is this atypical behaviour?
  3. What can I do to fix this and restore the server to a usable state?

I'm happy to post server details, logs etc as requested.

EDIT: I should have mentioned earlier, this server has been running fine for 4 months or so, and has only started sporadically crashing like this in the past month. I'm hoping to ascertain if this is simply due to increased Drupal loading, or some other (potentially more malicious) issue.

EDIT: As requested, here's the output of running ps_mem.py today.

-bash-3.2$ wget http://www.pixelbeat.org/scripts/ps_mem.py
-bash-3.2$ sudo python ps_mem.py
 Private  +   Shared  =  RAM used   Program 

 92.0 KiB +  12.0 KiB = 104.0 KiB   qmail-clean
 96.0 KiB +  14.0 KiB = 110.0 KiB   splogger
116.0 KiB +  23.0 KiB = 139.0 KiB   init
128.0 KiB +  12.0 KiB = 140.0 KiB   qmail-rspawn
124.0 KiB +  16.0 KiB = 140.0 KiB   syslogd
132.0 KiB +  12.0 KiB = 144.0 KiB   qmail-lspawn
148.0 KiB +  13.0 KiB = 161.0 KiB   qmail-send
208.0 KiB +  28.5 KiB = 236.5 KiB   dbus-daemon
232.0 KiB +  36.5 KiB = 268.5 KiB   xinetd
240.0 KiB +  32.5 KiB = 272.5 KiB   mysqld_safe
328.0 KiB +  20.5 KiB = 348.5 KiB   udevd
348.0 KiB +  66.0 KiB = 414.0 KiB   courierlogger (4)
444.0 KiB +  85.5 KiB = 529.5 KiB   bash
480.0 KiB +  50.0 KiB = 530.0 KiB   xfs
592.0 KiB +  36.0 KiB = 628.0 KiB   crond
544.0 KiB + 114.0 KiB = 658.0 KiB   couriertcpd (4)
  1.3 MiB +  82.5 KiB =   1.4 MiB   sw-cp-serverd
  1.2 MiB +   1.1 MiB =   2.3 MiB   sshd (3)
  3.1 MiB + 205.5 KiB =   3.3 MiB   named
  3.9 MiB +  48.2 MiB =  52.1 MiB   spamd (2)
 63.7 MiB + 387.0 KiB =  64.1 MiB   mysqld
108.3 MiB +   9.2 MiB = 117.5 MiB   httpd (7)
---------------------------------
                        245.4 MiB
=================================

A top snapshot, showing one httpd service running as root, others running as the apache user. Is this (one root process) typical?

-bash-3.2$ top -b -n 1
top - 17:09:34 up 27 days, 18:04,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  38 total,   1 running,  37 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    946344k total,   581372k used,   364972k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                     
21727 root      15   0  114m  52m 2112 S  0.0  5.6   0:09.17 spamd                                                                                                                                       
21728 popuser   18   0  114m  50m  972 S  0.0  5.5   0:00.00 spamd                                                                                                                                       
28112 apache    18   0  135m 3612  484 S  0.0  0.4   0:00.03 httpd                                                                                                                                       
28110 root      18   0  259m  15m 8284 S  0.0  1.7   0:00.21 httpd                                                                                                                                       
31854 apache    15   0  259m 9368 1492 S  0.0  1.0   0:00.00 httpd                                                                                                                                       
28113 apache    15   0  260m 9400 1508 S  0.0  1.0   0:00.03 httpd                                                                                                                                       
31853 apache    15   0  260m 9368 1484 S  0.0  1.0   0:00.00 httpd                                                                                                                                       
 5599 named     18   0  301m 5112 1976 S  0.0  0.5   0:00.51 named                                                                                                                                       
31851 apache    15   0  309m  61m 4004 S  0.0  6.7   0:01.13 httpd                                                                                                                                       
28124 apache    18   0  312m  63m 4996 S  0.0  6.8   0:03.92 httpd                                                                                                                                       
 1865 mysql     15   0  358m  70m 7112 S  0.0  7.6   5:52.36 mysqld                                                                                                                                      
 5824 root      18   0  3808  560  472 S  0.0  0.1   0:00.00 courierlogger                                                                                                                               
 5834 root      18   0  3808  556  472 S  0.0  0.1   0:00.00 courierlogger                                                                                                                               
 5842 root      18   0  3808  564  472 S  0.0  0.1   0:00.15 courierlogger                                                                                                                               
 5851 root      18   0  3808  556  472 S  0.0  0.1   0:00.00 courierlogger                                                                                                                               
 7255 qmailq    18   0  3812  428  344 S  0.0  0.0   0:00.00 qmail-clean                                                                                                                                 
 7252 qmaill    18   0  3816  556  468 S  0.0  0.1   0:00.00 splogger                                                                                                                                    
 7253 root      16   0  3856  468  352 S  0.0  0.0   0:00.00 qmail-lspawn                                                                                                                                
 7254 qmailr    15   0  3856  472  356 S  0.0  0.0   0:00.00 qmail-rspawn                                                                                                                                
 7250 qmails    15   0  3864  564  456 S  0.0  0.1   0:00.07 qmail-send                                                                                                                                  
 1418 root      15   0  5928  624  500 S  0.0  0.1   0:10.71 syslogd                                                                                                                                     
    1 root      18   0 10368  736  620 S  0.0  0.1   0:05.61 init                                                                                                                                        
 1695 root      18   0 10784 1300 1060 S  0.0  0.1   0:00.01 mysqld_safe                                                                                                                                 
31869 serverad  15   0 12080 1716 1324 S  0.0  0.2   0:00.02 bash                                                                                                                                        
31976 serverad  15   0 12624 1120  872 R  0.0  0.1   0:00.00 top  
 1123 root      15  -4 12636  684  356 S  0.0  0.1   0:00.00 udevd                                                                                                                                       
 5822 root      17   0 13188  612  476 S  0.0  0.1   0:00.00 couriertcpd                                                                                                                                 
 5832 root      15   0 13188  612  476 S  0.0  0.1   0:00.00 couriertcpd                                                                                                                                 
 5840 root      15   0 13188  612  476 S  0.0  0.1   0:00.46 couriertcpd                                                                                                                                 
 5849 root      18   0 13188  612  476 S  0.0  0.1   0:00.00 couriertcpd                                                                                                                                 
 3077 xfs       15   0 20276 1272  792 S  0.0  0.1   0:00.00 xfs                                                                                                                                         
 2035 root      15   0 20892 1180  588 S  0.0  0.1   0:04.17 crond                                                                                                                                       
 1427 dbus      19   0 21276  700  492 S  0.0  0.1   0:00.00 dbus-daemon                                                                                                                                 
 3523 root      15   0 21664  944  712 S  0.0  0.1   0:00.02 xinetd                                                                                                                                      
28213 sw-cp-se  18   0 59512 2728 1420 S  0.0  0.3   0:00.03 sw-cp-serverd                                                                                                                               
 1504 root      15   0 62680 1228  660 S  0.0  0.1   1:54.58 sshd                                                                                                                                        
31866 root      15   0 96624 3704 2852 S  0.0  0.4   0:00.02 sshd                                                                                                                                        
31868 serverad  15   0 96624 1892 1012 S  0.0  0.2   0:00.04 sshd             

... and some memory stats according to ps.

-bash-3.2$ ps aux | sort -nk +4 | tail
named     5599  0.0  0.5 308484  5112 ?        Ssl  Jul24   0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/run-root
apache   28113  0.0  0.9 266364  9400 ?        S    05:15   0:00 /usr/sbin/httpd
apache   31853  0.0  0.9 266364  9368 ?        S    16:08   0:00 /usr/sbin/httpd
apache   31854  0.0  0.9 266228  9368 ?        S    16:09   0:00 /usr/sbin/httpd
root     28110  0.0  1.6 266092 16028 ?        Ss   05:15   0:00 /usr/sbin/httpd
popuser  21728  0.0  5.5 117128 52164 ?        S    Jul07   0:00 spamd child
root     21727  0.0  5.6 117128 53304 ?        Ss   Jul07   0:09 /usr/bin/spamd --username=popuser --daemonize --nouser-config --helper-home-dir=/var/qmail --max-children 1 --pidfile=/var/run/spamd/spamd_full.pid --socketpath=/tmp/spamd_full.sock
apache   31851  0.0  6.6 317320 63324 ?        S    16:08   0:01 /usr/sbin/httpd
apache   28124  0.0  6.8 319536 64604 ?        S    05:15   0:03 /usr/sbin/httpd
mysql     1865  0.0  7.6 367604 72376 ?        Sl   Jul04   5:52 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306

Some potentially relevant snippets from httpd.conf (after adjusting prefork settings).

-bash-3.2$ head httpd.conf -n 128 | tail -60
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 20

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
  StartServers 2
  MinSpareServers 3
  MaxSpareServers 5
  ServerLimit 16
  MaxClients 16
  MaxRequestsPerChild 400000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
  StartServers 1
  MaxClients 50
  MinSpareThreads     25
  MaxSpareThreads     75 
  ThreadsPerChild     25
  MaxRequestsPerChild  0
</IfModule>

Thanks!

share|improve this question

migrated from superuser.com Jul 26 '12 at 8:33

This question came from our site for computer enthusiasts and power users.

    
You need to at least double that memory, more likely quadruple it, to get any useful work out of it as an Apache server, let alone a Drupal server (which implies a database as well). My laptop has four times that, and that's not enough. –  EJP Jul 26 '12 at 8:53
    
Your system has 512MB of physical memory as seen by the VM. Each Apache server is consuming 150MB of virtual memory as seen by the VM. You are comparing measurements of very different things. Most likely, that has nothing to do with why your server is crashing. If you want to solve the actual problem, tell us more about it. What precisely happens when your server crashes? –  David Schwartz Jul 26 '12 at 9:47
    
I'm happy to provide more info; when I say the server 'crashes', I mean all urls start timing out, as a result I cannot log into the attached plask panel. Additionally, ssh no longer works (connection times out) and the real time stats provided by MediaTemple stop working also. So far the only fix I've found is to restart the server. –  aaronsnoswell Aug 1 '12 at 23:50
    
@EJP - I was afraid of that possibility, however the sporadic memory issues have only started manifesting recently (past month); apache has been fine for at least 4 months prior. I should have mentioned this in my question. Whether this is simply due to increased Drupal load, or some other issue is what I hope to determine. –  aaronsnoswell Aug 2 '12 at 0:34
    
Any chance these issues started happening after a major plesk upgrade? –  Anthony Hatzopoulos Aug 10 '12 at 1:07
show 1 more comment

3 Answers

each consuming ~150mb of memory (according to top's 'RES' column).

That's not a good way to measure a processes memory usage, as explained in lots of places.

This very quickly crashes my server

If your system is running out of memory then you've configured it wrongly. Try measuring the memory usage properly and limit the number of servers first, then you can start to tune the installation.

What can I do to fix this

You're not going to get an answer here - there are too many questions and information that would need to be collected. Start by getting a handle on what's really happenning with the memory and the performance of the system read lots, read some more, establish a mechanism for collecting metrics, for preference set up a test environment where you can try out different configurations / loads quickly, then start tuning.

share|improve this answer
    
Until today I wasn't aware of any other ways of measuring memory than ps or top, thanks for inciting me to look for other ways (as per @deppfx's answer, I've added the results of the ps_mem.py script. I'm currently trying to set up a means for getting some better analytics going. Unfortunately, due to the limitations of my work, I don't think I'm going to be able to set up a dedicated test environment. –  aaronsnoswell Aug 2 '12 at 0:31
add comment

There could be a lot of reasons why this happens. As someone who works in a hosting data center, I would primarily doubt & ask the logs for the below:

  • There could be a memory leak in the drupal site code - Run this script & post the results here.
  • The website could have been hacked. So check the permissions of the running apache processes.

We can narrow down after you post the memory usage results.

share|improve this answer
    
Thanks for the suggestions - A memory leak or hacked server were my two suspicions. Whilst I realise it probably wouldn't make much difference to someone deliberately trying to be malicious, for what it's worth, the server is not 'public' per se - the domain is (hopefully) known only to me. I've posted the script's output, as well as some memory stats taken today. –  aaronsnoswell Aug 2 '12 at 0:08
    
@deppfx I'm also curious if you see anything suspicious or otherwise something OP could optimize after seeing his update after running the python script. –  Anthony Hatzopoulos Aug 10 '12 at 1:12
    
@AnthonyHatzopoulos Apache has been using 117 MB of memory which I think is normal. The reason why apache is dying could be something else & not memory. I would suggest to log in to the server & tail the httpd error logs and paste the output here just when it dies. If that's not possible, note the timestamp when the httpd dies, look into the logs & also check sar output for that particular time to reconfirm that its not memory that's killing apache.-- My $0.02 –  deppfx Aug 10 '12 at 6:23
add comment

This is very typical behavior which can be altered in the apache config. I had the same problem and this is what helped me:

Apache spawns couple of processes to wait for the connection - this is faster then spawning on demand.

use following directives:

 StartServers          3
 MinSpareServers       2
 MaxSpareServers       3

in the apache config to specify number of spawned processes.

Here is a similar topic about multiple apache processes -> http://stackoverflow.com/questions/501205/how-can-i-prevent-tons-of-apache-processes-spawning-when-i-start-apache-and-proc

share|improve this answer
    
Thanks for the suggestions - I went in and changed the relevant config vars - see the revised question. Not sure yet if this has helped at all. –  aaronsnoswell Aug 2 '12 at 0:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.