Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

One of my web servers is responding very slowly. Even to simple unix commands. Can you help point me in the right direction to debug?

Here's a crazy example:

# cat /dev/null

This command (printing a blank file) takes nearly 5 seconds to get to the next prompt. Similar experience with ls.

Here's the top of top:

top - 00:32:30 up  6:27,  4 users,  load average: 31.94, 26.82, 29.84
Tasks: 134 total,  16 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.6%us,  7.0%sy,  0.0%ni,  0.1%id,  0.0%wa,  0.0%hi,  0.3%si,  1.0%st
Mem:   8388608k total,  4243908k used,  4144700k free,    66700k buffers
Swap: 16777208k total,        0k used, 16777208k free,   779636k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                                                                                                                       
13068 mysql     15   0 1693m 528m 3552 S 24.2  6.5  43:10.37 mysqld                                                                                                                                                                                                                                                                                                         
28772 apache    16   0  292m  27m 4300 R 20.4  0.3   0:11.66 httpd                                                                                                                                                                                                                                                                                                          
28795 apache    15   0  275m  11m 4256 S 18.5  0.1   0:05.34 httpd                                                                                                                                                                                                                                                                                                          
28745 apache    16   0  275m  11m 4284 R 18.2  0.1   0:20.09 httpd                                                                                                                                                                                                                                                                                                          
28718 apache    16   0  274m 9740 4288 S 17.9  0.1   0:23.50 httpd                                                                                                                                                                                                                                                                                                          
28031 apache    15   0  274m 9796 4348 S 17.3  0.1   0:57.84 httpd                                                                                                                                                                                                                                                                                                          
28173 apache    16   0  292m  27m 4400 R 17.0  0.3   0:55.82 httpd                                                                                                                                                                                                                                                                                                          
28703 apache    15   0  274m 9820 4368 S 16.7  0.1   0:35.18 httpd                                                                                                                                                                                                                                                                                                          
28720 apache    15   0  275m  10m 4288 S 16.7  0.1   0:33.21 httpd                                                                                                                                                                                                                                                                                                          
28785 apache    15   0  277m  13m 4260 S 16.0  0.2   0:08.30 httpd                                                                                                                                                                                                                                                                                                          
28794 apache    16   0  292m  27m 4292 R 16.0  0.3   0:07.62 httpd                                                                                                                                                                                                                                                                                                          
27376 apache    15   0  275m  11m 4364 S 15.7  0.1   2:22.49 httpd                                                                                                                                                                                                                                                                                                          
27926 apache    15   0  292m  27m 4384 S 15.7  0.3   1:08.49 httpd                                                                                                                                                                                                                                                                                                          
28742 apache    15   0  275m  10m 4300 S 15.7  0.1   0:17.28 httpd                                                                                                                                                                                                                                                                                                          
28852 apache    15   0  275m  10m 4260 S 15.7  0.1   0:02.97 httpd                                                                                                                                                                                                                                                                                                          
28819 apache    16   0  292m  27m 4272 R 14.8  0.3   0:03.49 httpd                                                                                                                                                                                                                                                                                                          
28766 apache    16   0  291m  26m 4312 R 14.5  0.3   0:14.80 httpd                                                                                                                                                                                                                                                                                                          
27900 apache    16   0  292m  27m 4328 R 13.8  0.3   1:18.57 httpd                                                                                                                                                                                                                                                                                                          
28753 apache    16   0  288m  23m 4312 R 12.9  0.3   0:14.48 httpd                                                                                                                                                                                                                                                                                                          
28787 apache    15   0  283m  18m 4304 S 12.3  0.2   0:05.92 httpd                                                                                                                                                                                                                                                                                                          
28710 apache    15   0  275m  11m 4296 S 11.0  0.1   0:25.54 httpd                                                                                                                                                                                                                                                                                                          
28759 apache    16   0  287m  22m 4296 R 10.1  0.3   0:15.86 httpd                                                                                                                                                                                                                                                                                                          
28771 apache    16   0  284m  19m 4296 R  9.4  0.2   0:07.53 httpd                                                                                                                                                                                                                                                                                                          
28655 apache    16   0  284m  19m 4352 R  8.2  0.2   0:35.88 httpd                                                                                                                                                                                                                                                                                                          
28021 apache    16   0  285m  20m 4376 R  7.9  0.2   0:44.29 httpd                                                                                                                                                                                                                                                                                                          
27887 apache    16   0  280m  16m 4380 R  3.1  0.2   1:17.57 httpd                                                                                                                                                                                                                                                                                                          
28634 apache    15   0  275m  11m 4372 S  3.1  0.1   0:38.07 httpd                                                                                                                                                                                                                                                                                                          
28757 apache    16   0  280m  14m 4300 R  2.8  0.2   0:10.72 httpd                                                                                                                                                                                                                                                                                                          
28784 apache    16   0  279m  13m 4292 R  1.9  0.2   0:08.51 httpd                                                                                                                                                                                                                                                                                                          
28773 apache    15   0  275m  10m 4256 S  1.3  0.1   0:07.74 httpd                                                                                                                                                                                                                                                                                                          
28115 apache    16   0  280m  13m 4368 R  0.9  0.2   1:01.66 httpd                                                                                                                                                                                                                                                                                                          
28265 apache    15   0  280m  13m 4400 S  0.9  0.2   0:55.65 httpd                                                                                                                                                                                                                                                                                                          
share|improve this question

migrated from stackoverflow.com Jan 18 '12 at 0:45

This question came from our site for professional and enthusiast programmers.

    
strace -r is your friend. –  David Schwartz Jan 18 '12 at 0:51
add comment

2 Answers

up vote 12 down vote accepted

As a guess its because of your system load - you have over 30 processes currently waiting for CPU. time

share|improve this answer
    
I agree. The system is not memory or I/O limited because there's plenty of cache and system wait (wa in top output) is low. The system simply cannot provide enough CPU time to even start running cat in less than 5 seconds. If you cannot reduce the workload, the only fix is to get more CPU power. Your workload runs with very high concurrency so I would get CPU with as many cores as possible. –  Mikko Rantalainen Jan 16 '13 at 7:05
    
It seems that your httpd processes are pretty heavy because those eat way more CPU time than mysqld. I'd investigate if it's possible to optimize the code run by httpd processes (possibly PHP code?) to reduce CPU usage. If you have your own server room, it probably makes sense to move mysqld on a server of its own, run a reverse proxy with load balancer (recent apache with mod_proxy is fine) and run multiple httpd worker boxes in between. This is probably cheaper than getting one machine with very fast CPU. –  Mikko Rantalainen Jan 16 '13 at 7:08
add comment

Judging from the output of top, it looks like some fine tuning of your MySQL configuration may be in order. About half of your RAM is unused, and allowing MySQL to use some of that for caching may help decrease the wait time of your Apache processes.

Once Apache isn't hogging your iotime, you should find that your command line tests finish much quicker.

share|improve this answer
    
I don't see much evidence that this is an i/o issue. It's possible queries can be tuned, etc., but the bulk of cpu time is for user processes, not sys. The box is basically cpu-bound and is being slammed. –  cjc Jan 18 '12 at 4:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.