Server Load very high despite CPU usage being low

Question

When I run the top command I get this

top - 23:20:50 up  1:25,  1 user,  load average: 11.02, 11.20, 10.41
Tasks: 262 total,   3 running, 258 sleeping,   1 stopped,   0 zombie
Cpu(s): 75.6%us,  6.1%sy,  0.0%ni,  3.1%id, 14.3%wa,  0.0%hi,  0.8%si,  0.0%st
Mem:   2028800k total,  1669384k used,   359416k free,   153300k buffers
Swap:   523260k total,     2636k used,   520624k free,   749404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10221 www-data  20   0  416m  24m 5376 S   46  1.2   0:27.88 apache2
11290 www-data  20   0  420m  28m 3964 S   28  1.4   0:09.30 apache2
11844 www-data  20   0  424m  31m 5336 S   21  1.6   0:04.00 apache2
11670 www-data  20   0  410m  18m 3688 S   18  1.0   0:04.10 apache2
11147 www-data  20   0  417m  25m 5360 R   15  1.3   0:09.71 apache2
10615 www-data  20   0  418m  26m 5460 S    6  1.3   0:18.89 apache2
 3014 mysql     20   0 1316m 128m 8188 S    6  6.5   4:24.84 mysqld
10852 www-data  20   0  419m  26m 5376 S    6  1.4   0:16.05 apache2
11278 www-data  20   0  420m  28m 3984 S    3  1.5   0:10.39 apache2
 1589 root      20   0     0    0    0 D    1  0.0   1:16.40 jbd2/sda1-8
12024 www-data  20   0 81044 4732 3180 S    1  0.2   0:00.04 sendmail
 5281 root      20   0 97.9m 4696 1800 D    1  0.2   0:56.55 sendmail-mta
11927 root      20   0 17464 1452  932 R    1  0.1   0:00.32 top
12009 root      20   0 99.6m 5232 2720 D    1  0.3   0:00.06 sendmail-mta
 2929 syslog    20   0  243m 3104 1140 S    1  0.2   0:25.32 rsyslogd
 3029 bind      20   0  238m  21m 3032 S    1  1.1   0:27.77 named
 6627 root      20   0  101m 6872 2852 D    1  0.3   0:07.54 sendmail-mta
10525 root      20   0  100m 5308 1536 D    1  0.3   0:02.33 sendmail-mta
14241 root      20   0  100m 6136 2868 S    1  0.3   0:31.78 sendmail-mta
18543 root      20   0  100m 6300 2868 R    1  0.3   0:27.42 sendmail-mta
22589 root      20   0  100m 6472 2884 S    1  0.3   0:22.43 sendmail-mta
31196 root      20   0  100m 6604 2852 D    1  0.3   0:16.98 sendmail-mta
    1 root      20   0 24332 2012 1356 S    0  0.1   0:05.23 init
 1391 root      20   0     0    0    0 S    0  0.0   0:02.97 kworker/0:0
 2549 root      20   0  101m 6728 2852 D    0  0.3   0:12.15 sendmail-mta
 3395 smmsp     20   0 83048 5076 1460 S    0  0.3   0:24.24 sendmail-msp
 3661 ntp       20   0 37772 2252 1620 S    0  0.1   0:00.39 ntpd
 5382 smmsp     20   0 83048 6924 3324 S    0  0.3   0:20.41 sendmail-msp
 5483 root      20   0 97.9m 4696 1800 D    0  0.2   0:56.38 sendmail-mta
 7502 root      20   0     0    0    0 S    0  0.0   0:00.80 kworker/1:0
12025 root      20   0 99700 3956 1660 D    0  0.2   0:00.01 sendmail-mta
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.10 ksoftirqd/0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.58 ksoftirqd/1
   11 root       0 -20     0    0    0 S    0  0.0   0:00.00 cpuset

Yet when I run uptime I get

22:53:23 up 57 min,  1 user,  load average: 8.38, 9.22, 8.88

And as a result, my vBulletin forums are locking out all users.

Something obvious seems to be wrong, how can I identify and solve the issue?

Many thanks.

Answer 1

So you actually are using a lot of CPU. Either get a better server or make your forums become less popular. You also seem to be sending quite a bit of mail... is your forum hacked and is somebody using it as a spam source? Check your mail logs...

Answer 2

[Update: answer was posted before the full top output was added. While the answer is still correct, it no longer applies to the situation]

Load is not CPU usage, load is amount of processes in the run queue. Usually a high load with low CPU usage indicates an I/O problem, like sluggish or hanging I/O. I once had a load of over 9000 on a mailserver where the storage went for a walk. Hardly any CPU usage, and ssh perfectly responsive, it just didn't like being a mailserver anymore.

Answer 3

You have both an high CPU (idle time 3.1%, nice time 0%) and probably a high disk load (try looking at vmstat output, check for some out-of-scale number in the block-in/block-out queues or some high value on the wait time, which means if I'm not wrong the time spent waiting for some I/O to complete).

On a not loaded system you'll have the wait-time close to 0% and small values for the read/written blocks.

I experienced similar troubles with a site, where mysql was using a lot of disk and memory, while php/apache were mostly CPU-bound... The solution was to split it in two: the www front-end on a machine, the mysql back-end on another. Things went smoother then..

Anyway try to better understand what is causing your load - maybe your sendmail is part of the problem, I see a lot of such processes in the "D" state (waiting for device - that is, disk bound). First of all ensure it is working for you and not for others (relaying spammers' mail or such...)

Have a nice hunt! :)

Answer 4

You should just install postfix. Your mail server is probably acting as an open relay due to a configuration. Postfix defaults mitigate those problems and is probably faster than re-configuring sendmail -

Issue sendmail -bp to get a list of messages in the sendmail queue. If you have a lot of messages in /var/spool/mqueue that are not going away you could just change into that directory and rm *. If someone is sending a message at that moment and it doesn't get removed by sendmail before you do however, it will be lost. Since there is no sendmail switch to flush the queue, you may have to do that. There are other methods as well that you can find in other threads.