Help! My server is running out of memory and locking up, requiring a hard reboot

Question

I have a VPS (with 256MB of memory) that has hung more than once today. It just froze entirely, and was not accessible via SSH. I had to run my provider's "force power off" procedure to bring it back. Then the server lasted for a few hours before doing it again.

The first time there was a message about running out of memory and attempting to kill the mysqld process. The next I managed to find this in the syslog file:

Sep 23 01:47:34: [16942.757665]  [out_of_memory+0x19e/0x1e0] out_of_memory+0x19e/0x1e0
Sep 23 01:47:34: [16942.758875] Out of memory: kill process 5983 (apache2) score 57519 or a child

I've configured Apache and MySQL to be fairly mindful of the low RAM, at least to my knowledge, and I'd like to know why this is happening. Do I simply need to upgrade, or is there something wrong?

Here is an excerpt from my Apache config:

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 30

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 200

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 3

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          1
    MinSpareServers       1
    MaxSpareServers      3
    MaxClients          50
    MaxRequestsPerChild   1000
</IfModule>

And my MySQL config:

[mysqld]
#
# * Basic Settings
#

#
# * IMPORTANT
#   If you make changes to these settings and your system uses apparmor, you may
#   also need to also adjust /etc/apparmor.d/usr.sbin.mysqld.
#

user    	= mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket  	= /var/run/mysqld/mysqld.sock
port    	= 3306
basedir 	= /usr
datadir 	= /var/lib/mysql
tmpdir  	= /tmp
language    = /usr/share/mysql/english
skip-external-locking
skip-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address    	= 127.0.0.1
#
# * Fine Tuning
#
key_buffer  	= 16K
max_allowed_packet  = 1M
thread_stack    	= 64K
thread_cache_size   = 4
sort_buffer 	= 64K
net_buffer_length   = 2K
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 500k #was 1M
query_cache_size        = 8M #was 16M, lowered for RAM
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
#log    	= /var/log/mysql/mysql.log
#
# Error logging goes to syslog. This is a Debian improvement :)
#
# Here you can see queries with especially long duration
#log_slow_queries   = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id  	= 1
#log_bin    		= /var/log/mysql/mysql-bin.log
expire_logs_days    = 10
max_binlog_size         = 100M
#binlog_do_db   	= include_database_name
#binlog_ignore_db   = include_database_name
#
# * BerkeleyDB
#
# Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
skip-bdb
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
# You might want to disable InnoDB to shrink the mysqld process by circa 100MB.
skip-innodb
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet  = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer  	= 16M

And in case it helps, here's a ps aux output from about 20-30 minutes after a reboot:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3   3992   920 ?        Ss   01:53   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S<   01:53   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S<   01:53   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S<   01:53   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   01:53   0:00 [watchdog/0]
root         6  0.0  0.0      0     0 ?        S<   01:53   0:00 [events/0]
root         7  0.0  0.0      0     0 ?        S<   01:53   0:00 [khelper]
root        18  0.0  0.0      0     0 ?        S<   01:53   0:00 [xenwatch]
root        19  0.0  0.0      0     0 ?        S<   01:53   0:00 [xenbus]
root        27  0.0  0.0      0     0 ?        S<   01:53   0:00 [migration/1]
root        28  0.0  0.0      0     0 ?        S<   01:53   0:00 [ksoftirqd/1]
root        29  0.0  0.0      0     0 ?        S<   01:53   0:00 [watchdog/1]
root        30  0.0  0.0      0     0 ?        S<   01:53   0:00 [events/1]
root        54  0.0  0.0      0     0 ?        S<   01:53   0:00 [kblockd/0]
root        55  0.0  0.0      0     0 ?        S<   01:53   0:00 [kblockd/1]
root        65  0.0  0.0      0     0 ?        S<   01:53   0:00 [kseriod]
root       104  0.0  0.0      0     0 ?        S    01:53   0:00 [pdflush]
root       105  0.0  0.0      0     0 ?        S    01:53   0:00 [pdflush]
root       106  0.0  0.0      0     0 ?        S<   01:53   0:00 [kswapd0]
root       107  0.0  0.0      0     0 ?        S<   01:53   0:00 [aio/0]
root       108  0.0  0.0      0     0 ?        S<   01:53   0:00 [aio/1]
root       124  0.0  0.0      0     0 ?        S<   01:53   0:00 [accel_watch/0]
root       125  0.0  0.0      0     0 ?        S<   01:53   0:00 [accel_watch/1]
root      2031  0.0  0.0      0     0 ?        S<   01:53   0:00 [kjournald]
root      2192  0.0  0.3  16848   944 ?        S<s  01:53   0:00 /sbin/udevd --daemon
syslog    3514  0.0  0.2  12296   728 ?        Ss   01:53   0:00 /sbin/syslogd -u syslog
root      3534  0.0  0.2   8132   592 ?        S    01:53   0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg
klog      3537  0.0  0.8   5600  2292 ?        Ss   01:53   0:00 /sbin/klogd -P /var/run/klogd/kmsg
root      3556  0.0  0.4  50916  1160 ?        Ss   01:53   0:00 /usr/sbin/sshd
root      3776  0.0  0.8  36684  2152 ?        Ss   01:53   0:00 /usr/lib/postfix/master
postfix   3794  0.0  0.8  38740  2140 ?        S    01:53   0:00 pickup -l -t fifo -u -c
postfix   3795  0.0  0.8  38784  2228 ?        S    01:53   0:00 qmgr -l -t fifo -u
root      3802  0.0  0.2  12460   644 ?        Ss   01:53   0:00 /usr/sbin/dovecot
root      3815  0.0  1.0  71648  2876 ?        S    01:53   0:00 dovecot-auth
daemon    3816  0.0  0.1  16428   428 ?        Ss   01:53   0:00 /usr/sbin/atd
root      3829  0.0  0.3  18616   976 ?        Ss   01:53   0:00 /usr/sbin/cron
dovecot   3851  0.0  0.7  14148  1876 ?        S    01:53   0:00 imap-login
dovecot   3852  0.0  0.7  14148  1876 ?        S    01:53   0:00 imap-login
dovecot   3853  0.0  0.7  14148  1876 ?        S    01:53   0:00 imap-login
root      3875  0.0  0.2   3864   592 tty1     Ss+  01:53   0:00 /sbin/getty 38400 tty1
root      3913  0.0  1.1  68272  3116 ?        Ss   01:54   0:00 sshd: root@pts/0 
postfix   3917  0.0  1.1  41248  2964 ?        S    01:54   0:00 tlsmgr -l -t unix -u -c
root      3924  0.0  0.7  18908  2080 pts/0    Ss   01:54   0:00 -bash
root      3958  0.0  0.2   3864   592 ?        Ss   01:55   0:00 /sbin/getty 38400 console
root      4232  0.0  0.2   3944   608 pts/0    S    02:22   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     4271  0.1  5.0 124404 13244 pts/0    Sl   02:22   0:02 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pi
root      4272  0.0  0.2   5068   768 pts/0    S    02:22   0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
root      4415  0.0  3.6 201880  9516 ?        Ss   02:29   0:00 /usr/sbin/apache2 -k start
www-data  4527  0.5 11.9 225720 31340 ?        S    02:47   0:01 /usr/sbin/apache2 -k start
www-data  4539  0.6 12.2 226492 32092 ?        S    02:50   0:00 /usr/sbin/apache2 -k start
www-data  4541  2.3 12.1 226492 31992 ?        S    02:52   0:00 /usr/sbin/apache2 -k start
root      4544  0.0  0.4  15064  1088 pts/0    R+   02:53   0:00 ps aux

Can anyone help me out here?

Answer 1

You really need to trim up your services. Better yet, you might want to consider using inetd or xinetd (whichever applies to your installation). Rather than leave several services launched and waiting (and hogging precious RAM) you might be better with a CPU performance hit from launching a service when it's needed. And that's what inetd/xinetd were really meant for - they were intended to provide a low-memory way to allow services to launch on demand, rather than keep them hanging around all the time. Of course, your mileage may very - you can still end up with memory resource issues - but if it gets you over the rough spot for the time being...

Follow-up on where to start (from comment):

I would look at getting apache moved over first. Reduce the maximum child count to just 2 - unless you are servicing more users than yourself, it helps little to have 3 running processes.

For dovecot, specifically the imap-auth processes, you only need one. You could run it in inetd as well, although it appears to be fairly thin. If you don't use IMAP or POP on your server, turn it off. Worst case, you're back to reading mail in a command line - not pretty but functional.

You can also move sshd, although I would initially do it at an alternative port number and confirm it connects before you flip the switch on that one.

Consider moving to Exim if possible (which is the default install for Debian/Ubuntu setups). I love Postfix and I don't use Exim, but Debian chose it as the default for a reason - it requires little resources and is fairly secure.

Look at your /proc/sys/vm/swappiness setting (echo /proc/sys/vm/swappiness) and determine if it's too high or too low. A higher number will help to free up memory (by swapping to VM ahead of time) but when it's too high you'll swap like mad. Most installs say "60" but for low memory, it should probably be more like 85. I don't recommend 100 in your setup.

Lastly, think "green screen". You're talking about using techniques that have been around for a long time on much older systems (because they were resources constrained also). It's a bit old-timey to do things this way but if you can get your work done in a client program in a shell, rather than launching services, you'll stretch your computing resources that much further.

Answer 2

Keep an eye in the number of httpd processes that are running and how much memory they are using. I think your web server is swapping, maybe because the MaxClients directive is exceeding the optimal value for your server. You need to calculate average RAM you need for each apache process (according the results of ps is around 30M) and set MaxClients with a value that not exceeds the total available RAM with enough room for other processes like MySQL.

Answer 3

If you type the command 'free' when the server begins to slow down, you will see the memory usage and I am willing to bet, as HD said, that your server is hitting up the swap space. Going by your PS output, you are almost at 256M there.

You may want to move MaxSpareServers down to 2 or even 1. Also, there are alternatives to Apache, for example lighttpd which is supposed to be lightweight and use less memory.

I have a personal opinion, and I'm sure some will disagree, that you shouldn't have less than 512M or 1G ram in a server.