Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am getting rather strange high load averages on one of my VPS', I am not a server admin by nature, rather a website developer that has learnt the basics over the years. I am however very experienced with unix.

Recently one of my VPS' with exactly the same set up as the others seems to be getting weird spikes.

Do you know what could be causing the spikes? What information can I get to help diagnose the cause?

Many thanks for your time,


Server info: 1gb RAM, Ubuntu, Apache+MySQL+PHP5 with Dovecot for mail, using suPHP

Info:

Time:                    Wed May 25 11:10:21 2011 +0100
1 Min Load Avg:          12.56
5 Min Load Avg:          6.45
15 Min Load Avg:         2.83
Running/Total Processes: 2/110

PS:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root         1  0.0  0.0   2156   584 ?        Ss   Apr03   0:04 init [3]      
 root      1128  0.0  0.0   2248   324 ?        S<s  Apr03   0:00 /sbin/udevd -d
 root      1433  0.0  0.0   1812   596 ?        Ds   Apr03   0:19 syslogd -m 0
 root      1442  0.0  0.0   3448   660 ?        Ss   Apr03   0:00 /usr/sbin/dovecot
 dovecot   1461  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1462  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1463  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1465  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1467  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1468  0.0  0.0   6648  1528 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1470  0.0  0.0   6648  1528 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1471  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1472  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1476  0.0  0.0   6648  1524 ?        S    Apr03   0:00  \_ dovecot/imap-login
 dovecot   1477  0.0  0.0   3364   828 ?        S    Apr03   0:00  \_ dovecot/anvil [35 connections]
 root      1478  0.0  0.0   3356   768 ?        S    Apr03   0:00  \_ dovecot/log
 root      1481  0.0  0.0   4824  1628 ?        S    Apr03   0:00  \_ dovecot/config
 root      1490  0.0  0.0   4412  1532 ?        S    Apr03   0:00  \_ dovecot/auth [0 wait, 0 passdb, 0 userdb]
 dovecot  31765  0.0  0.0   6648  1528 ?        S    Apr05   0:00  \_ dovecot/imap-login
 dovecot  31774  0.0  0.0   6648  1528 ?        S    Apr05   0:00  \_ dovecot/imap-login
 dovecot  31778  0.0  0.0   6648  1528 ?        S    Apr05   0:00  \_ dovecot/imap-login
 dovecot  32514  0.0  0.0   6648  1528 ?        S    Apr07   0:00  \_ dovecot/imap-login
 dovecot  32522  0.0  0.0   6648  1528 ?        S    Apr07   0:00  \_ dovecot/imap-login
 dovecot  32531  0.0  0.0   6648  1524 ?        S    Apr07   0:00  \_ dovecot/imap-login
 dovecot  19608  0.0  0.1   6644  2272 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19609  0.0  0.1   6644  2256 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19611  0.0  0.1   6644  2272 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19612  0.0  0.1   6644  2276 ?        S    May23   0:00  \_ dovecot/pop3-login
 root     19660  0.0  0.0   3768  1344 ?        S    May23   0:00  \_ dovecot/auth [0 wait, 0 passdb, 0 userdb]
 dovecot  19661  0.0  0.1   6644  2276 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19662  0.0  0.1   6644  2276 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19663  0.0  0.1   6644  2276 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19664  0.0  0.1   6644  2280 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19665  0.0  0.1   6644  2280 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19666  0.0  0.1   6644  2276 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19668  0.0  0.1   6644  2280 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  19671  0.0  0.1   6644  2280 ?        S    May23   0:00  \_ dovecot/pop3-login
 dovecot  28222  0.0  0.1   6644  2280 ?        S    01:43   0:00  \_ dovecot/pop3-login
 dovecot  28242  0.0  0.1   6644  2284 ?        S    01:43   0:00  \_ dovecot/pop3-login
 dovecot  28246  0.0  0.1   6644  2284 ?        S    01:44   0:00  \_ dovecot/pop3-login
 dovecot  28248  0.0  0.1   6644  2276 ?        S    01:44   0:00  \_ dovecot/pop3-login
 root      1500  0.0  0.0   2832   588 ?        Ss   Apr03   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 root      1512  0.0  0.0   3708   976 ?        S    Apr03   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/tent.pid
 mysql     1586  0.0  0.3 133196  7972 ?        Sl   Apr03   8:44  \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/mysql-bin.err --pid-file=/var/lib/mysql/tent.pid
 root      1768  0.0  0.0   3244   352 ?        Ss   Apr03   0:06 /usr/local/directadmin/da-popb4smtp
 nobody    1776  0.0  0.0   9168   756 ?        Ss   Apr03   0:00 /usr/local/directadmin/directadmin d
 nobody    1777  0.0  0.0   9288  1172 ?        S    Apr03   0:00  \_ /usr/local/directadmin/directadmin d
 nobody    1789  0.0  0.0   9288   132 ?        S    Apr03   0:00  |   \_ /usr/local/directadmin/directadmin d
 nobody    1790  0.0  0.0   9288   132 ?        S    Apr03   0:00  |   \_ /usr/local/directadmin/directadmin d
 nobody    1791  0.0  0.0   9288   132 ?        S    Apr03   0:00  |   \_ /usr/local/directadmin/directadmin d
 nobody    1792  0.0  0.0   9288   132 ?        S    Apr03   0:00  |   \_ /usr/local/directadmin/directadmin d
 nobody    1793  0.0  0.0   9288   132 ?        S    Apr03   0:00  |   \_ /usr/local/directadmin/directadmin d
 nobody   18419  0.0  0.0   9168   192 ?        S    May19   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   18420  0.0  0.0   9168   168 ?        S    May19   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   18421  0.0  0.0   9168   168 ?        S    May19   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   18422  0.0  0.0   9168   168 ?        S    May19   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   18423  0.0  0.0   9168   168 ?        S    May19   0:00  \_ /usr/local/directadmin/directadmin d
 mail      1794  0.0  0.0   9240   712 ?        Ss   Apr03   0:02 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
 mail     14222  0.0  0.1  10836  2864 ?        D    11:01   0:00  \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
 mail     14271  0.0  0.0   9240   224 ?        D    11:08   0:00  \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
 ftp       1847  0.0  0.0   7404  1616 ?        SLs  Apr03   0:10 proftpd: (accepting connections)
 root      1860  0.0  0.0   4488   632 ?        Ss   Apr03   0:04 crond
 root     14250  0.0  0.0   5032   996 ?        D    11:06   0:00  \_ crond
 root     14254  0.0  0.0   5032   996 ?        D    11:07   0:00  \_ crond
 root     14267  0.0  0.0   5032   996 ?        D    11:08   0:00  \_ crond
 root      1868  0.0  0.0   5680   384 ?        Ss   Apr03   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
 root      1869  0.0  0.0   5680   136 ?        S    Apr03   0:00  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
 named     1888  0.0  0.0  70852  1892 ?        Ssl  Apr03   0:02 named -u named
 root      1516  0.0  0.0   7196   764 ?        Ss   May19   0:00 /usr/sbin/sshd
 root     26259  0.0  0.7  18812 15400 ?        Ss   00:00   0:03 lfd - sleeping
 root     14268  0.0  0.6  18812 14544 ?        S    11:08   0:00  \_ lfd - (child) checking load...
 root     14270  0.0  0.0   2528   884 ?        R    11:08   0:00      \_ /bin/ps axuf
 root     26419  0.0  2.0  49944 42240 ?        Ss   00:12   0:00 /usr/sbin/httpd -k start -DSSL
 apache   26424  0.0  2.0  50544 42628 ?        S    00:12   0:00  \_ /usr/sbin/httpd -k start -DSSL
 pegpro   14247  0.0  0.2  18284  4440 ?        D    11:05   0:00  |   \_ /usr/local/php5/bin/php-cgi
 apache   26425  0.0  2.0  50552 42608 ?        S    00:12   0:00  \_ /usr/sbin/httpd -k start -DSSL
 pegpro   14236  0.0  0.2  18284  4440 ?        D    11:03   0:00  |   \_ /usr/local/php5/bin/php-cgi
 apache   26426  0.0  2.0  50560 42620 ?        S    00:12   0:00  \_ /usr/sbin/httpd -k start -DSSL
 pegpro   14242  0.0  0.2  18284  4440 ?        D    11:04   0:00  |   \_ /usr/local/php5/bin/php-cgi
 apache    9842  0.0  2.0  50596 42544 ?        S    06:27   0:00  \_ /usr/sbin/httpd -k start -DSSL
 root     14258  0.0  0.0   3096  1000 ?        D    11:07   0:00  |   \_ /usr/local/suphp/sbin/suphp
 apache    9851  0.0  2.0  50560 42616 ?        S    06:28   0:00  \_ /usr/sbin/httpd -k start -DSSL
 root     14265  0.0  0.0   3096  1004 ?        D    11:08   0:00  |   \_ /usr/local/suphp/sbin/suphp
 apache   11359  0.0  2.0  50560 42548 ?        S    07:25   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   12184  0.0  2.0  50384 42432 ?        S    09:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 pegpro   14249  0.0  0.2  18284  4444 ?        D    11:05   0:00  |   \_ /usr/local/php5/bin/php-cgi
 apache   13618  0.0  2.0  50388 42372 ?        S    09:47   0:00  \_ /usr/sbin/httpd -k start -DSSL
 root     14261  0.0  0.0   3096  1000 ?        D    11:07   0:00  |   \_ /usr/local/suphp/sbin/suphp
 apache   13627  0.0  2.0  50420 42424 ?        S    09:48   0:00  \_ /usr/sbin/httpd -k start -DSSL
 root     14259  0.0  0.0   3096  1000 ?        D    11:07   0:00  |   \_ /usr/local/suphp/sbin/suphp
 apache   13761  0.0  2.0  50380 42348 ?        S    10:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 pegpro   14248  0.0  0.2  18284  4440 ?        D    11:05   0:00  |   \_ /usr/local/php5/bin/php-cgi
 apache   14253  0.0  1.9  50076 40812 ?        S    11:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   14255  0.0  1.9  50076 40812 ?        S    11:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   14260  0.0  1.9  50076 40812 ?        S    11:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   14266  0.0  1.9  50076 40812 ?        S    11:08   0:00  \_ /usr/sbin/httpd -k start -DSSL

VMSTAT:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  3 12      0 1740504      0      0    0    0     8     1    0    0  0  0 99  1  0

Apache:

Could not retrieve Apache Server Status

Edit:

Noting that crons could be the issue I checked my crontab, I can't see anything that would be running at 11:10!

The rsyncs usually only send 1/2mb of modifications per day.

0 4 * * */3 /usr/local/sysbk/sysbk -q
15 3 * * * rsync -av --delete /home/ tentbackup@SNIP.com:~/rsyncbackup/daily/
0 5 * * */14 rsync -av --delete /backup/ tentbackup@SNIP.com:~/dabackup/
35 7 * * * curl -s -o /dev/null http://SNIP/billing.php?action=autosuspend
40 7 * * * curl -s -o /dev/null http://SNIP/billing.php?action=gen_upcoming
45 7 * * * curl -s -o /dev/null http://SNIP/billing.php?action=send_pending
30 7 * * * curl -s -o /dev/null http://SNIP/alp/billing.php
0 * * * * curl -s -o /dev/null http://SNIP/gitMail/index.php

I have modified my load average email report adding iostat, mysql processes and the suggested ps. I will update this question the next time I get a message!

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Any command not in state S (sleep) will be counted as an active process. This includes those in R running state, and D blocking state. (The latter usually occurring when it's waiting from IO from a disk or network device)

To find a list of those specifically, try the following command: ps -efl | cut -c3- | egrep -v "^S". From your ps output, it would appear you've got some cron jobs that are stuck, some exim processes, your syslog daemon, and some php-cgi threds. These all may be stuck waiting on disk access. So your HDD may be swamped, or possibly throwing errors, causing processes to get hung.

Try using iostat to check on your disk IO?

--Christopher Karel

share|improve this answer
    
Thanks for the advice, I have updated my email report script and will get back to you the next time an error occurs. –  Pez Cuckow May 25 '11 at 16:04
add comment

there may be locked files, dovecot maybe locking the mbox file,

I think you can verify this by looking in /proc/locks

http://wiki1.dovecot.org/MailboxFormat/Maildir#Locking

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.