Ubuntu server hangs for 5 seconds often

Question

I have a server that will hang for a total of 5 seconds about every 2 to 5 minutes at peak time. Peak time will run 6+ hours.

The server specs are:

Ubuntu Linux:    12.04.1
Kernel and CPU:  Linux 3.0.0-17-server on x86_64
Processor info:  Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 8 cores
Ram: 6GB only 1.20GB used at peak

I'm running the latest LAMP package with PHP-APC. The server is located behind a protected proxy. When I run netstat -anp | grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n I get 18099 which is normal for the nature of the site. They all are legitimate connections hitting the server from 1 IP.

I have apache.conf prefork_module tuned and have changed it many times with no effect of solving the hanging.

I have watched all the proccess to see if any peak, none do. The disk IO at peak is 67% idle. I have plenty of available ram and it never swaps. The server itself runs great then it just acts like it wants to take a 5 second break and when it returns it can handle what it has missed during the 5 seconds just fine.

There is nothing in the apache error.log that would suggest anything.

I feel something is getting limited and I cannot pin point what it is. I kinda want to say its something with the amount of connections I have like its hitting some sort of connection limit. I was wondering if you guys had any ideas what it could be or something I can run to see.

Update

The server reponds fine when visiting it directly via IP not through the domain. So when I test at the same time when it hangs for the 5 seconds the domain doesn't respond but the IP will respond fine so it makes me think it's the server OS doing it

I did manage to see [apache2] <defunct> just one or two in there not sure if that is enough to cause the hang but it only shows up when it does hand and apache doesn't give me anything about it in the error log.

Update 1/20/2013

My datacenter is going to build a new server and have me switch over. I have two other servers identical on the same rack same setup same ammount of traffic and this is the only one giving me a issue.

Answer 1

This sounds suspiciously like a failing drive. Pull a smart report for your drives and replace any that have a high number of relocated sectors or other errors.

Answer 2

I had an issue just like this about 5 years ago. The entire machine would just lag incredibly hard for 5 seconds (and you could do nothing at that time). It turned out that something in the motherboard had gone bad, and the warranty was still active, so I didn't look into it any further. Anyway, you might want to check there, if the drive isn't the problem.

Answer 3

One possibilty is nameserver problem (apache amongst others may do dns lookups for logging purpose and fail silently)

Check if DNS servers are configured correctly and server can make dns lookups.
If DNS servers are unavailable check if apache2 configuration uses dns lookups.