I'm having load problems with my server and even though I'm a somewhat experienced Linux admin I'm out of ideas now.

The problem is a slowly but steadily increasing load on the server without any apparent cause.

The Server is a AMD Athlon(tm) 64 X2 Dual Core Processor 6000+ with 6GB RAM. It is running Debian Stable with Linux gir 2.6.26-2-amd64 #1 SMP Wed Aug 19 22:33:18 UTC 2009 x86_64 GNU/Linux.

The server basically runs Lighttpd, several FastCGI PHP processes and a MySQL database. Typical webserver tasks.

The CPU is never really fully used up and memory is mainly used for buffers and cache which is fine. I tried to restart the various services to see if one of them would decrease the load again, but without luck.

Here are graphics showing load, CPU and IOStat:

So, question is: What could cause a slowly but ever increasing load? And how do I find out what's responsible?

Update: I forgot to mention, when I reboot the server, the load will be down to around 0.3 to 0.6 and will start to climb up again slowly over the next weeks.

  • 1
    The images you posted no longer exist. Please feel free to re-upload them if you still have copies. – Michael Hampton Dec 7 '14 at 21:34

Each zombie process adds 1.0 to the load. You might be seeing an accumulation of zombies.

| improve this answer | |
  • Yes. Check the "Number of Processes" graph. – Teddy Nov 3 '09 at 12:55
  • If that was correct, then typing for N in {1..100} ; do sleep 60 & done ; exec sleep 500 should be enough to cause a high load. But it doesn't. That command produces 100 zombies, but the load on my computer remained below 1. – kasperd Dec 7 '14 at 21:49

I found an excellent hint in answer to a different question.

Looking for processes in state 'D' shows four PHP processes that seem to hang for quite a while corresponding to the "steps" in the load curve:

#> ps aux | awk '$8 ~ /D/  { print $0 }'
wiki      6651  0.0  0.0      0     0 ?        D    Oct04   0:41 [php-cgi]
bugs      6731  0.0  0.0      0     0 ?        D    Oct27   0:14 [php-cgi]
manpages  7536  0.0  0.0      0     0 ?        D    Oct30   0:21 [php5-cgi]
wiki     23847  0.0  0.0      0     0 ?        D    Oct06   1:32 [php-cgi]

So these seem to be the problem. I now need to find out while those processes hang and how to fix it. Thanks everyone.

| improve this answer | |
  • This answer solved my issue. The load increase from 0.5 to 350 and kept going up. It was due to zombie processes trying to read a deleted remote folder. – Philippe Delteil Sep 26 '19 at 14:33

My guess is that server is IO starved , maybe you should add the iotop stats to the graphs

I wonder if you can have an per application io activity that is also a factor for server load


other tool is dstat

| improve this answer | |
  • I added graphics for IOStat as well. The disk IO does not increase as the load does. Is that what you were aiming at? – Andreas Gohr Nov 3 '09 at 10:50
  • Oh and dstat looks useful. I have to read a bit more about it. – Andreas Gohr Nov 3 '09 at 10:56

If it were I/O, then he would see the iowait (pink) on the cpu graphs.

| improve this answer | |

This kind of problems often came from the harddisk that is not enough fast to serve datas required by the MySQL database and the HTTP server. You should look at iostat command

| improve this answer | |
  • IO looks normal to me. And it wouldn't explain why the load is slowly increasing. – Andreas Gohr Nov 3 '09 at 13:51

In general, it's actually not a bad thing to have a high server load; it means that you're not sitting idle and doing less than you otherwise could. 80%-90% load of your total capacity (with some "burst" room) is what's usually sought after. I'd recommend checking the output of mpstat and vmstat. In particular, the first 2 numbers from vmstat can give you more meaningful info about how "backed up" you are in terms of processes in the run queue. The last column ("wa") of vmstat output can tell you if, and for how long, you're waiting for I/O completions. The run queue size and the I/O wait time are often correlated. Also check out sar (from the sysstat package): that gives you a detailed view of what's going on over a period of time; the metrics that it records are very thorough.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.