Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Apache threads are stacking up on one of my web servers (300-500 simultaneous requests, some taking 3-8s to process!), but CPU usage is very low (~10%). Page load time is slowing way down as a result. I have plenty of idle CPU power. How can I use more of it to handle these threads faster?

Here's the top of top...

Tasks: 469 total,   1 running, 468 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.1% us,  1.7% sy,  0.0% ni, 90.3% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   9181012k total,  7998772k used,  1182240k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                                                                 
11351 apache    15   0  364m  30m  17m S 11.9  0.3   0:00.73 httpd                                                                                                                                                                                                                                                    
 7527 apache    15   0  365m  36m  23m S  8.6  0.4   0:01.76 httpd                                                                                                                                                                                                                                                    
 7607 apache    16   0  364m  35m  22m S  2.3  0.4   0:01.47 httpd                                                                                                                                                                                                                                                    
11498 apache    17   0  359m  19m  11m S  2.3  0.2   0:00.07 httpd                                                                                                                                                                                                                                                    
11497 apache    16   0  362m  23m  13m S  1.7  0.3   0:00.05 httpd                                                                                                                                                                                                                                                    
 1840 apache    15   0  366m  44m  29m S  1.3  0.5   0:03.74 httpd                                                                                                                                                                                                                                                    
 5358 apache    15   0  364m  36m  24m S  1.3  0.4   0:02.58 httpd                                                                                                                                                                                                                                                    
 8090 apache    15   0  365m  31m  17m S  1.3  0.3   0:01.10 httpd                                                                                                                                                                                                                                                    
11346 apache    15   0  361m  28m  18m S  1.3  0.3   0:00.12 httpd                                                                                                                                                                                                                                                    
 4051 apache    16   0  365m  40m  27m S  1.0  0.5   0:01.72 httpd                                                                                                                                                                                                                                                    
32575 apache    16   0  365m  42m  28m S  0.7  0.5   0:03.62 httpd                                                                                                                                                                                                                                                    
 5145 apache    16   0  365m  37m  24m S  0.7  0.4   0:02.23 httpd                                                                                                                                                                                                                                                    
 8173 apache    16   0  363m  35m  23m S  0.7  0.4   0:00.29 httpd                                                                                                                                                                                                                                                    
 5466 apache    15   0  365m  31m  18m S  0.3  0.4   0:01.18 httpd                                                                                                                                                                                                                                                    
 7420 apache    16   0  364m  36m  23m S  0.3  0.4   0:01.24 httpd                                                                                                                                                                                                                                                    
11485 apache    16   0  362m  23m  12m S  0.3  0.3   0:00.04 httpd                                                                                                                                                                                                                                                    
    1 root      15   0 10272  612  584 S  0.0  0.0   0:02.78 init                                                                                                                                                                                                                                                     
30129 root      16  -4 12536  400  396 S  0.0  0.0   0:00.00 udevd                                                                                                                                                                                                                                                    
30402 root      16   0  5840  580  480 S  0.0  0.0   0:14.01 syslogd                                                                                                                                                                                                                                                  
30414 rpc       18   0  7992  408  404 S  0.0  0.0   0:00.00 portmap                                                                                                                                                                                                                                                  
30439 root      18   0 10088  548  544 S  0.0  0.0   0:00.00 rpc.statd                                                                                                                                                                                                                                                
30478 memcache  15   0  141m 5364  516 S  0.0  0.1   1:16.34 memcached                                                                                                                                                                                                                                                
30496 root      16   0 60604  744  636 S  0.0  0.0   0:07.31 sshd                                                                                                                                                                                                                                                     
30507 root      15   0 21572  796  688 S  0.0  0.0   0:04.56 xinetd                                                                                                                                                                                                                                                   
31817 root      15   0  166m  932  860 S  0.0  0.0   0:00.03 httpsd                                                                                                                                                                                                                                                   
31820 psaadm    15   0  175m 7992 4596 S  0.0  0.1   0:02.31 httpsd                                                                                                                                                                                                                                                   
31924 root      15   0 19704  924  552 S  0.0  0.0   0:02.50 crond                                                                                                                                                                                                                                                    
13316 root      16   0 98528 3628 2796 S  0.0  0.0   0:00.01 sshd                                                                                                                                                                                                                                                     
 1655 root      19   0  8600 1180  972 S  0.0  0.0   0:00.00 mysqld_safe                                                                                                                                                                                                                                              
 1695 mysql     16   0 4268m 464m 4684 S  0.0  5.2  10:05.19 mysqld                                                                                                                                                                                                                                                   
32564 root      16   0 98528 3612 2780 S  0.0  0.0   0:00.00 sshd                                                                                                                                                                                                                                                     
28489 root      15   0 98528 3628 2796 S  0.0  0.0   0:00.00 sshd                                                                                                                                                                                                                                                     
32152 root      16   0 98528 3612 2780 S  0.0  0.0   0:00.01 sshd                                                                                                                                                                                                                                                     
 5781 root      15   0 98528 3628 2796 S  0.0  0.0   0:00.01 sshd                                                                                                                                                                                                                                                     
 7801 root      17   0  356m  11m 5472 S  0.0  0.1   0:05.74 httpd                                                                                                                                                                                                                                                    
 7804 apache    16   0  366m  36m  21m S  0.0  0.4   0:00.35 httpd                                                                                                                                                                                                                                                    
 7805 apache    16   0  370m  31m  13m S  0.0  0.4   0:00.11 httpd                                                                                                                                                                                                                                                    
 8172 apache    15   0  366m  34m  19m S  0.0  0.4   0:00.62 httpd                                                                                                                                                                                                                                                    
 9430 apache    16   0  365m  45m  32m S  0.0  0.5   0:02.67 httpd                                                                                                                                                                                                                                                    
11393 apache    16   0  363m  37m  25m S  0.0  0.4   0:00.75 httpd                                                                                                                                                                                                                                                    
11551 apache    17   0  360m  31m  22m S  0.0  0.4   0:00.24 httpd                                                                                                                                                                                                                                                    
32345 apache    16   0  364m  39m  27m S  0.0  0.4   0:02.86 httpd                                                                                                                                                                                                                                                    
32472 apache    16   0  394m  70m  27m S  0.0  0.8   0:03.77 httpd                                                                                                                                                                                                                                                    
32488 apache    16   0  364m  42m  29m S  0.0  0.5   0:02.38 httpd                                                                                                                                                                                                                                                    
32501 apache    16   0  365m  41m  28m S  0.0  0.5   0:01.71 httpd                                                                                                                                                                                                                                                    
32644 apache    16   0  365m  36m  23m S  0.0  0.4   0:01.79 httpd                                                                                                                                                                                                                                                    
32765 apache    15   0  364m  39m  26m S  0.0  0.4   0:02.65 httpd                                                                                                                                                                                                                                                    
 1334 apache    16   0  368m  42m  26m S  0.0  0.5   0:02.77 httpd                                                                                                                                                                                                                                                    
 1339 apache    15   0  362m  39m  29m S  0.0  0.4   0:01.84 httpd                                                                                                                                                                                                                                                    
 1351 apache    15   0  364m  43m  30m S  0.0  0.5   0:02.59 httpd                                                                                                                                                                                                                                                    
 1553 apache    16   0  363m  41m  29m S  0.0  0.5   0:02.74 httpd                                                                                                                                                                                                                                                    
 1555 apache    16   0  365m  37m  24m S  0.0  0.4   0:01.59 httpd                                                                                                                                                                                                                                                    
 1564 apache    15   0  365m  40m  27m S  0.0  0.5   0:01.88 httpd                                                                                                                                                                                                                                                    
 1569 apache    16   0  364m  35m  22m S  0.0  0.4   0:00.63 httpd                                                                                                                                                                                                                                                    
 1573 apache    15   0  367m  39m  24m S  0.0  0.4   0:01.66 httpd                                                                                                                                                                                                                                                    
 1575 apache    16   0  363m  36m  24m S  0.0  0.4   0:01.88 httpd                                                                                                                                                                                                                                                    
 1583 apache    16   0  364m  34m  21m S  0.0  0.4   0:01.92 httpd                                                                                                                                                                                                                                                    
 1594 apache    15   0  367m  44m  29m S  0.0  0.5   0:03.71 httpd                                                                                                                                                                                                                                                    
 1689 apache    15   0  365m  38m  24m S  0.0  0.4   0:01.47 httpd                                                                                                                                                                                                                                                    
 1690 apache    15   0  365m  39m  26m S  0.0  0.4   0:01.18 httpd                                                                                                                                                                                                                                                    
 1710 apache    16   0  363m  34m  23m S  0.0  0.4   0:00.99 httpd                                                                                                                                                                                                                                                    
 1725 apache    15   0  364m  39m  26m S  0.0  0.4   0:01.80 httpd                                                                                                                                                                                                                                                    
 1726 apache    16   0  365m  40m  26m S  0.0  0.5   0:00.90 httpd                                                                                                                                                                                                                                                    
 1737 apache    16   0  364m  30m  17m S  0.0  0.3   0:00.46 httpd                                                                                                                                                                                                                                                    
 1919 apache    15   0  363m  34m  22m S  0.0  0.4   0:00.83 httpd                                                                                                                                                                                                                                                    
 1930 apache    16   0  364m  33m  21m S  0.0  0.4   0:00.50 httpd                                                                                                                                                                                                                                                    
 1934 apache    15   0  364m  40m  27m S  0.0  0.5   0:02.20 httpd

And critical httpd.conf settings:

Timeout 120
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 3

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>
share|improve this question
2  
Are you sure CPU is your bottleneck? –  ceejayoz Jan 20 '12 at 1:16
    
Which MPM are you using? And what's its config look like? StartServers, ServerLimit, etc.? –  Shane Madden Jan 20 '12 at 1:16
    
@ceejayoz CPU doesn't appear to be a bottleneck. Apache does. But since I have so much CPU sitting idle I'm hoping to utilize more for Apache to help speed up thread processing. –  Ryan Jan 20 '12 at 1:25
    
@ShaneMadden I just added some confs above. Does that help? (BTW, Thanks for taking a look) –  Ryan Jan 20 '12 at 1:29
3  
Are you sure Apache is your bottleneck and not, say, PHP code, or MySQL locking, etc.? –  ceejayoz Jan 20 '12 at 1:40

2 Answers 2

up vote 3 down vote accepted

To configure Apache to use more of your CPU, you need to get rid of the bottle-neck that is currently preventing Apache from using more CPU. Because requests are taking 3-8 seconds to complete, you know there is some sort of bottle-neck, you need to find it.

Things to look at are:

  • Determine if the Apache host is the bottle-neck. If you run uptime while requests are taking 3-8 seconds, does the load show as being high (in the double or triple digits)? You can't really read a lot into this, because a high load could mean the problem exists elsewhere, but if the load is low while requests are taking 3-8 seconds it probably is a remote issue.
  • If you have resources required by this service on other machines, such as a database, check the system load there as well.
  • Look at the output of "vmstat 1" on the server. This will show system utilization statistics, updated every second. Things to look at are "cpu wa" (if it's double digits it probably means the disc is saturated), and "swap si/so" (if these are non-zero it means you are out of memory), and "cpu id" (idle CPU time, if this is <10 your CPU is saturated). If "cpu sy" time is high, that may indicate that you are forking too much (perhaps Apache MaxRequests is too low?) or other kernel-level overhead. Press Control-C to end the "vmstat 1" output.
  • You can run "strace -p [AN APACHE PROCEED ID]" to get an idea of what the Apache processes are doing. This shows the system calls they are making, but if you aren't familiar with system calls you can sometimes get information just by reading the output as text. For example, last week I had a server that was acting like yours and the strace would pause for several seconds with the last thing that was displayed was a log message, and I tracked it down to a PHP file that was sending a bunch of log messages to syslog.

If you don't have "munin" installed, you probably should. If you do, look at the graphs to see how the system utilization changes when the system is responding slowly and when it's working well. If you see jumps in the graphs, those might indicate where the bottle-neck is. If you see blank areas in all the graphs, that probably means the system is saturated. If only the Apache graphs are blank, it probably means that Apache has reached it's max connections, probably a side-effect of the performance problems.

Also note that if you have multiple CPUs, but you have a single-threaded application like Zope sitting behind Apache, it could be that one of your CPUs is saturated where the others are idle. If you press "1" in top, that will show you the utilization of each individual core. Look for one that is at 0% idle all the time, where the others are much more idle.

Using these techniques I've been able to isolate and resolve most performance problems similar to this.

share|improve this answer
    
Uptime to get load? strace as a starting point? Not convinced –  symcbean Jan 20 '12 at 9:39
2  
@symcbean, your comment could be more useful. If you have something you prefer over "uptime" to see the system load average, don't keep us in suspense. strace is shockingly useful in cases like this. –  Sean Reifschneider Jan 20 '12 at 13:53

If you have a look through the many previous Q&A on apache performance then you'll get some clues.

You've omitted lots of information from your httpd.conf file that we'd need to see how the server is configured - otoh you've included both the threaded and pre-fork config - which are you using?

CPU usage is very low (~10%). Page load time is slowing way down as a result

No - you've got this completely backwards. Page load time is slowing down, but the only thing that can be discounted as contributing to the slowdown is CPU.

There may be things that you can do with the CPU which may alleviate performance issues elsewhere (using compression, software raid) but really the fault lies elsewhere.

What's the average RTT (from HTTP traffic, not ICMP)? How much disk I/O is there? What's the memory profile look like? Are you just servicing static content? If not what else is involved in generating the content? Have you monitored the network traffic in/out the server to see if there's something there you didn't expect? Have you audited the caching instructions issued?

share|improve this answer
    
Your point about being able to rule out the CPU as the bottle-neck is a good one. Otherwise, your message could be more helpful: References to the other apache performance Q&As, what information is missing from the httpd.conf file, you ask a lot of questions, without providing information on how Ryan can get those answers. –  Sean Reifschneider Jan 20 '12 at 13:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.