Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run a site with about 100,000 visits a day and lately the load averages sometimes hit the roof (250+). They are normally about 0.5-1.5, so I assumed a (d)dos attack. The number of apache processes reaches up to 1,000. However, the number of connections to the server does not change (around 600) and the max connections a client has is ~20. It only happens occasionally and it sorts its self out when I restart apache. Why does apache decide to spawn loads of threads and send my load averages to 250?

This is the result of top:

Normal top:

top - 15:07:16 up 1 day,  1:28,  3 users,  load average: 1.44, 2.55, 21.83
Tasks: 307 total,   3 running, 304 sleeping,   0 stopped,   0 zombie
Cpu(s): 30.2%us,  3.2%sy,  0.0%ni, 64.9%id,  1.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   2074128k total,  1836852k used,   237276k free,    21732k buffers
Swap:  2096472k total,    56440k used,  2040032k free,   278048k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
16776 apache    16   0 38752  23m 3124 S 20.4  1.2   0:01.32 httpd              
16252 apache    15   0 37760  20m 3240 S  5.6  1.0   0:00.46 httpd              
16333 apache    15   0 36152  21m 3136 S  5.0  1.1   0:01.16 httpd              
17426 apache    15   0 35732  20m 3056 S  3.2  1.0   0:00.16 httpd              
17418 apache    16   0 38536  22m 3020 S  2.8  1.1   0:00.14 httpd              
16161 apache    16   0 36800  21m 3148 S  2.0  1.1   0:00.63 httpd              
15988 apache    15   0 29284  12m 3112 S  1.8  0.6   0:00.20 httpd              
17441 mysql     16   0 66656  19m 1988 R  1.4  1.0   0:00.07 mysqld             
17442 mysql     16   0 66656  19m 1988 R  1.2  1.0   0:00.06 mysqld             
12265 apache    15   0 35916  21m 3148 S  0.8  1.0   0:01.49 httpd              
15987 apache    15   0 37616  20m 3224 S  0.6  1.0   0:00.46 httpd              
13564 root      15   0  2552 1196  804 R  0.4  0.1   0:00.83 top                
17011 apache    15   0 26668  11m 3016 S  0.4  0.6   0:00.05 httpd              
  409 root      10  -5     0    0    0 S  0.2  0.0   0:21.83 scsi_eh_0          
 3920 mysql     15   0 66656  19m 1988 S  0.2  1.0   0:26.90 mysqld             
 9357 apache    18   0 40896  23m 3228 S  0.2  1.2   0:03.36 httpd              
11971 apache    18   0 39888  23m 3868 S  0.2  1.1   0:01.55 httpd              

Abnormal top:

top - 09:12:12 up 3 days, 19:33,  1 user,  load average: 58.75, 69.17, 58.05
Tasks: 568 total,   2 running, 563 sleeping,   0 stopped,   3 zombie
Cpu(s):  5.6%us,  2.3%sy,  0.0%ni, 11.7%id, 79.9%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:   2074128k total,  2004008k used,    70120k free,     9008k buffers
Swap:  2096472k total,  1009264k used,  1087208k free,   128268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14534 mysql     16   0 79764  24m 1824 R  6.0  1.2   0:00.20 mysqld             
13903 mysql     16   0 79764  24m 1824 S  2.4  1.2   0:00.13 mysqld             
13906 mysql     16   0 79764  24m 1824 S  2.4  1.2   0:00.08 mysqld             
14068 mysql     16   0 79764  24m 1824 S  1.8  1.2   0:00.10 mysqld             
13862 apache    15   0 35532  15m 1892 D  0.9  0.8   0:00.44 httpd              
14316 apache    16   0 26012 6672 1852 S  0.9  0.3   0:00.14 httpd              
14787 apache    16   0 27992 8332 1820 D  0.9  0.4   0:00.06 httpd              
10588 apache    18   0 39480  13m 1912 D  0.6  0.7   0:01.42 httpd              
12457 apache    16   0 39332  12m 2328 D  0.6  0.6   0:00.42 httpd              
13564 root      16   0  2684 1180  648 S  0.6  0.1   5:18.00 top                
13819 apache    15   0 26688 7352 2276 S  0.6  0.4   0:00.56 httpd              
13853 mysql     16   0 79764  24m 1824 S  0.6  1.2   0:00.09 mysqld             
14335 apache    16   0 28544 8964 1872 S  0.6  0.4   0:00.30 httpd              
  442 root      10  -5     0    0    0 D  0.3  0.0   0:53.61 kjournald          
 6337 apache    15   0 39420  17m 2648 S  0.3  0.9   0:03.46 httpd              
 9976 apache    18   0 38924 7708 2316 S  0.3  0.4   0:01.44 httpd              
 9990 apache    15   0 36432  16m 2556 S  0.3  0.8   0:02.23 httpd              
share|improve this question
add comment

1 Answer

It's the value of your ServerLimit setting and others inside httpd.conf:

StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

Dial down your ServerLimit to 100 or so and decrease MaxRequestPerChild to around 1000 so the children reap faster.

share|improve this answer
    
Why would load averages suddenly rise from ~1 to 250 though? –  ddlshack Jul 18 '10 at 5:04
    
load is load - do you think it's going to spawn 200 children and just sit there? no, they were spawned to actually do work on your system and use the CPU, etc. –  troyengel Jul 18 '10 at 14:42
    
Yeah but why would those children suddenly be created and not be created again when I restart apache? –  ddlshack Jul 20 '10 at 13:18
    
Look at the swap utilization in the second top. That system is probably thrashing (doing more work swapping processes in and out of RAM/on and off disk than real process work) when the load hits the roof. Setting the ServerLimit to an amount you can hold in RAM will also prevent that. –  freiheit Jul 24 '10 at 5:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.