Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Apologies for the vague thread title, summarising the following was proving to be a little complicated after all it is, as titled, unexplained. Anyway, enough excuses...

This morning I discovered that my website is running extremely slow, now this doesn't normally happen so I am obviously attempting to track down the cause of the issue. Knowing that I have not installed or changed anything recently, my first port of call was to check the resource usage stats, these show nothing out of the ordinary:

load average: 0.35, 0.34, 0.36

Checking this over a period of about half an hour (during which time outages were reported by users) never shows anything above 1. So it is not "traditional load". So I am looking for other potential causes.

Top also shows nothing out of the ordinary:

top - 08:34:34 up  1:33,  1 user,  load average: 0.30, 0.36, 0.35               
Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie            
Cpu0  :  6.6%us,  1.3%sy,  0.0%ni, 91.1%id,  0.7%wa,  0.0%hi,  0.3%si,  0.0%st  
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 99.3%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st  
Cpu2  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st  
Cpu3  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st  
Mem:   4016884k total,  1367624k used,  2649260k free,     5324k buffers        
Swap:  3919840k total,        0k used,  3919840k free,   769024k cached         

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2593 apache    15   0  446m  66m  40m S  7.6  1.7   1:13.64 httpd              
 2450 mysql     15   0  257m  48m 5976 S  0.3  1.2   4:20.51 mysqld             
 9734 root      15   0 12740 1296  932 R  0.3  0.0   0:00.24 top                
    1 root      18   0 10348  752  628 S  0.0  0.0   0:04.91 init               
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0        
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/1      

So then I started to look at the network, the following command (which I took from a SF question about DDOS attacks):

netstat -n|grep :80|cut -c 45-|cut -f 1 -d ':'|sort|uniq -c|sort -nr|more

Gives:

534
  5     1.1.1.1
  4     2.2.2.2
  4     3.3.3.3
  3     4.4.4.4
  2     5.5.5.5
  2     6.6.6.6
  2     7.7.7.7
  1     8.8.8.8
  1     9.9.9.9
  1     10.10.10.10
  1     11.11.11.11

IP Address edited out

There doesn't seem to be anything out of the ordinary there either, although I am not certain what that 534 means. For good measure I have also rebooted the server (force of habit after using Windows for so long ;)) but this did not make any difference whatsoever.

So now I find myself at a loss, I can't explain what is happening here and this of course means that I can't fix it.

Server Details This is a dedicated server with the following spec:

  • Quad-Core AMD Opteron(tm) Processor 1381
  • 4GB RAM

This site server PHP pages (vbulletin only) via Apache with a MySQL backend, I am also running APC as an opcode cacher.

EDIT - More Info May or may not be useful...

Using Firebug in Firefox I have been looking at the load times for pages. What seems to be happening is that one random resource (sometimes an image, a JS file or a CSS file) takes an inordinate amount of time to complete the receive. The request completes in a matter of milliseconds but the receive sometimes takes upwards of a minute. However it is a random resource, every request I make has a different resource that takes a long time to come back. I don't have any caching etc in place for these resources, these are served as normal through apache from the filesystem.

EDIT Output from iostat:

Linux 2.6.18-164.11.1.el5 12/10/2010      

avg-cpu:  %user   %nice %system %iowait  %steal   %idle                         
           4.66    0.00    2.08    0.84    0.00   92.42                         

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn          
sda              12.48        78.48       144.52    1008089    1856500          
sda1              0.43         2.52         6.95      32354      89224          
sda2              0.01         0.11         0.00       1356          0          
sda4              0.00         0.00         0.00         10          0          
sda5              0.48         5.33         1.61      68413      20706          
sda6             11.57        70.51       135.97     905732    1746570          
sdb              12.43        78.57       144.52    1009340    1856500          
sdb1              0.43         2.24         6.95      28768      89224          
sdb2              0.00         0.08         0.00       1068          0          
sdb4              0.00         0.00         0.00         10          0          
sdb5              0.45         5.35         1.61      68729      20706          
sdb6             11.53        70.88       135.97     910533    1746570          
md1               0.91         4.72         5.96      60666      76520          
md6              14.70       141.37       126.26    1815945    1621898          
md5               0.57        10.65         1.05     136822      13474  

EDIT

It might be useful if I give you guys the URL of the site:

http://www.therev[REMOVETHIS]counter.com

share|improve this question
    
Another thing I have noticed is that I cannot ping the domain the server is running on from my home PC all ping requests result in 'Request timed out' - Which is a bit odd to say the least because if I am persistent I can get to webpages and have been able to SSH to the server. –  MrEyes Dec 10 '10 at 13:53
    
Can you describe what is slow exactly? Page loads? –  mattdm Dec 10 '10 at 13:59
    
I hate to be vague, but everything. Page loads is the most obvious symptom I am using SSH from my iphone so it is difficult to tell if that is also slow. –  MrEyes Dec 10 '10 at 14:05
    
Also worth mentioning that the slow downs seem to be intermittent, for a couple of seconds everything clicks through as normal then everything slows down for a few seconds, then it is back to normal, rinse and repeat –  MrEyes Dec 10 '10 at 14:09
    
For slow page loads, is it "slow" as in the page starts loading right away but it takes a while for all of the pictures and everything to load, or does it take a while before the browser even sets the titlebar (and then is the rest fast or slow?). From what we've got so far it sounds like a network problem, but it could still be a drive IO problem if you've got heavy disk activity (check iostat?) Also check dmesg for anything that might be enlightening (drive errors, etc) –  DerfK Dec 10 '10 at 14:38

2 Answers 2

up vote 3 down vote accepted

Well, if the problem happens with static files, that's good, because at least you know to start looking at Apache. You're probably going to want to break out the debugging and profiling tools to see exactly what's wrong. Presuming you're talking about a Linux system, strace is probably the tool you want. With the -f and -c options, it will follow all forked child processes and summarize the amount of time spent in each syscall. This should hopefully help you figure out the problem.

Stop Apache, then restart it through strace:

strace -cf /usr/sbin/httpd

(strace does have a -p option to trace the pid of an existing process, but even with -f it doesn't trace child processes that were forked before strace was called.)

Let it run for awhile, bang on the site while it's running until you can trigger the slowdown a few times, then interrupt it. Analyze the results.

If it turns out that the issue is in user-mode application code rather than in something the system is doing, there's a companion program called ltrace that can be used to summarize the time spent in various shared library calls.

It probably goes without saying, but also check your server, system, and kernel logs to make sure you're not seeing any unexpected failures or hardware events.

share|improve this answer
1  
As documented, when running strace on apache httpd, start it with httpd -X to force a single-threaded mode that CAN be debugged. –  adaptr Feb 15 '12 at 15:32

What measures did you take to rule out a clientside issue? The minimal server load and intermittent latency of random resource requests would make me want to rule out a real time file scanner as the culprit. This may be way off, but it should be trivial to rule out.

share|improve this answer
    
The site is a host for a fairly tight community of people, so I have had reports from numerous members that they are experiencing exactly the same problems I am seeing. This however, does not rule out a network issue with the hosts. –  MrEyes Dec 10 '10 at 15:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.