Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server that will hang for a total of 5 seconds about every 2 to 5 minutes at peak time. Peak time will run 6+ hours.

The server specs are:

Ubuntu Linux:    12.04.1
Kernel and CPU:  Linux 3.0.0-17-server on x86_64
Processor info:  Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 8 cores
Ram: 6GB only 1.20GB used at peak

I'm running the latest LAMP package with PHP-APC. The server is located behind a protected proxy. When I run netstat -anp | grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n I get 18099 which is normal for the nature of the site. They all are legitimate connections hitting the server from 1 IP.

I have apache.conf prefork_module tuned and have changed it many times with no effect of solving the hanging.

I have watched all the proccess to see if any peak, none do. The disk IO at peak is 67% idle. I have plenty of available ram and it never swaps. The server itself runs great then it just acts like it wants to take a 5 second break and when it returns it can handle what it has missed during the 5 seconds just fine.

There is nothing in the apache error.log that would suggest anything.

I feel something is getting limited and I cannot pin point what it is. I kinda want to say its something with the amount of connections I have like its hitting some sort of connection limit. I was wondering if you guys had any ideas what it could be or something I can run to see.

Update

The server reponds fine when visiting it directly via IP not through the domain. So when I test at the same time when it hangs for the 5 seconds the domain doesn't respond but the IP will respond fine so it makes me think it's the server OS doing it

I did manage to see [apache2] <defunct> just one or two in there not sure if that is enough to cause the hang but it only shows up when it does hand and apache doesn't give me anything about it in the error log.

Update 1/20/2013

My datacenter is going to build a new server and have me switch over. I have two other servers identical on the same rack same setup same ammount of traffic and this is the only one giving me a issue.

share|improve this question
    
What type of hardware? –  ewwhite Jan 19 '13 at 22:44

3 Answers 3

up vote 1 down vote accepted

This sounds suspiciously like a failing drive. Pull a smart report for your drives and replace any that have a high number of relocated sectors or other errors.

share|improve this answer
    
I believe I ran it correct. dpaste.com/884153 it says SMART overall-health self-assessment test result: PASSED but when you scroll down it says some errors not sure what that is. –  Abby E Jan 19 '13 at 22:10
    
No RAID or RAID controller? –  ewwhite Jan 19 '13 at 22:45
    
After reading and doing a short test it Completed without error –  Abby E Jan 19 '13 at 22:45

I had an issue just like this about 5 years ago. The entire machine would just lag incredibly hard for 5 seconds (and you could do nothing at that time). It turned out that something in the motherboard had gone bad, and the warranty was still active, so I didn't look into it any further. Anyway, you might want to check there, if the drive isn't the problem.

share|improve this answer
    
I have contacted my hosting company and see what they have to say. –  Abby E Jan 19 '13 at 23:12
    
The server reponds fine when visiting it directly via IP not through the domain. So when I test at the same time when it hangs for the 5 seconds the domain doesn't respond by the server will respond through the IP so it makes me think it's the server OS doing it. –  Abby E Jan 19 '13 at 23:59

One possibilty is nameserver problem (apache amongst others may do dns lookups for logging purpose and fail silently)

Check if DNS servers are configured correctly and server can make dns lookups.
If DNS servers are unavailable check if apache2 configuration uses dns lookups.

share|improve this answer
    
I use a paid DNS service to handle all queries and it's done through vhost. :) –  Abby E Jan 20 '13 at 10:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.