Unexplained server slowdown

Question

Apologies for the vague thread title, summarising the following was proving to be a little complicated after all it is, as titled, unexplained. Anyway, enough excuses...

This morning I discovered that my website is running extremely slow, now this doesn't normally happen so I am obviously attempting to track down the cause of the issue. Knowing that I have not installed or changed anything recently, my first port of call was to check the resource usage stats, these show nothing out of the ordinary:

load average: 0.35, 0.34, 0.36

Checking this over a period of about half an hour (during which time outages were reported by users) never shows anything above 1. So it is not "traditional load". So I am looking for other potential causes.

Top also shows nothing out of the ordinary:

top - 08:34:34 up  1:33,  1 user,  load average: 0.30, 0.36, 0.35               
Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie            
Cpu0  :  6.6%us,  1.3%sy,  0.0%ni, 91.1%id,  0.7%wa,  0.0%hi,  0.3%si,  0.0%st  
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 99.3%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st  
Cpu2  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st  
Cpu3  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st  
Mem:   4016884k total,  1367624k used,  2649260k free,     5324k buffers        
Swap:  3919840k total,        0k used,  3919840k free,   769024k cached         

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2593 apache    15   0  446m  66m  40m S  7.6  1.7   1:13.64 httpd              
 2450 mysql     15   0  257m  48m 5976 S  0.3  1.2   4:20.51 mysqld             
 9734 root      15   0 12740 1296  932 R  0.3  0.0   0:00.24 top                
    1 root      18   0 10348  752  628 S  0.0  0.0   0:04.91 init               
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0        
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/1      

So then I started to look at the network, the following command (which I took from a SF question about DDOS attacks):

netstat -n|grep :80|cut -c 45-|cut -f 1 -d ':'|sort|uniq -c|sort -nr|more

Gives:

534
  5     1.1.1.1
  4     2.2.2.2
  4     3.3.3.3
  3     4.4.4.4
  2     5.5.5.5
  2     6.6.6.6
  2     7.7.7.7
  1     8.8.8.8
  1     9.9.9.9
  1     10.10.10.10
  1     11.11.11.11

IP Address edited out

There doesn't seem to be anything out of the ordinary there either, although I am not certain what that 534 means. For good measure I have also rebooted the server (force of habit after using Windows for so long ;)) but this did not make any difference whatsoever.

So now I find myself at a loss, I can't explain what is happening here and this of course means that I can't fix it.

Server Details This is a dedicated server with the following spec:

• Quad-Core AMD Opteron(tm) Processor 1381
• 4GB RAM

This site server PHP pages (vbulletin only) via Apache with a MySQL backend, I am also running APC as an opcode cacher.

EDIT - More Info May or may not be useful...

Using Firebug in Firefox I have been looking at the load times for pages. What seems to be happening is that one random resource (sometimes an image, a JS file or a CSS file) takes an inordinate amount of time to complete the receive. The request completes in a matter of milliseconds but the receive sometimes takes upwards of a minute. However it is a random resource, every request I make has a different resource that takes a long time to come back. I don't have any caching etc in place for these resources, these are served as normal through apache from the filesystem.

EDIT Output from iostat:

Linux 2.6.18-164.11.1.el5 12/10/2010      

avg-cpu:  %user   %nice %system %iowait  %steal   %idle                         
           4.66    0.00    2.08    0.84    0.00   92.42                         

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn          
sda              12.48        78.48       144.52    1008089    1856500          
sda1              0.43         2.52         6.95      32354      89224          
sda2              0.01         0.11         0.00       1356          0          
sda4              0.00         0.00         0.00         10          0          
sda5              0.48         5.33         1.61      68413      20706          
sda6             11.57        70.51       135.97     905732    1746570          
sdb              12.43        78.57       144.52    1009340    1856500          
sdb1              0.43         2.24         6.95      28768      89224          
sdb2              0.00         0.08         0.00       1068          0          
sdb4              0.00         0.00         0.00         10          0          
sdb5              0.45         5.35         1.61      68729      20706          
sdb6             11.53        70.88       135.97     910533    1746570          
md1               0.91         4.72         5.96      60666      76520          
md6              14.70       141.37       126.26    1815945    1621898          
md5               0.57        10.65         1.05     136822      13474  

EDIT

It might be useful if I give you guys the URL of the site:

http://www.therev[REMOVETHIS]counter.com

Answer 1

Well, if the problem happens with static files, that's good, because at least you know to start looking at Apache. You're probably going to want to break out the debugging and profiling tools to see exactly what's wrong. Presuming you're talking about a Linux system, strace is probably the tool you want. With the -f and -c options, it will follow all forked child processes and summarize the amount of time spent in each syscall. This should hopefully help you figure out the problem.

Stop Apache, then restart it through strace:

strace -cf /usr/sbin/httpd

(strace does have a -p option to trace the pid of an existing process, but even with -f it doesn't trace child processes that were forked before strace was called.)

Let it run for awhile, bang on the site while it's running until you can trigger the slowdown a few times, then interrupt it. Analyze the results.

If it turns out that the issue is in user-mode application code rather than in something the system is doing, there's a companion program called ltrace that can be used to summarize the time spent in various shared library calls.

It probably goes without saying, but also check your server, system, and kernel logs to make sure you're not seeing any unexpected failures or hardware events.

Answer 2

What measures did you take to rule out a clientside issue? The minimal server load and intermittent latency of random resource requests would make me want to rule out a real time file scanner as the culprit. This may be way off, but it should be trivial to rule out.