Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Our server occasionally refuses to serve a simple HTML page.

This is happening during a relatively high number of requests. However, the processor is not heavy loaded and there are a lot of free memory. The error seems to occure 1 out of 50 requests in average, depending on the server load.

I need to find the source of the problem and take the appropriate actions to eliminate it.

I have a suspicion that the problem source is a huge number of incoming network packets. There are 5000 packets per second on average. Traffic - 2 MBits/sec Can this be the cause of the error?

There is an interesting thing, in case the server fails to respond, the request string is not logged to access.log by Apache.

The error is repeatable from several client computers. DNS is not involved, since I have accessed the server by the IP.

I have profiled the problem case with tcpdump utility. These are the good and bad sessions traced by tcpdump. The request is the same in both experiments. Good - server returns response. Bad - no response, time-out error.

---- Bad ----
12:23:36.366292 IP > S 2125316338:2125316338(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
12:23:39.362394 IP > S 2125316338:2125316338(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
12:23:45.365567 IP > S 2125316338:2125316338(0) win 8192 <mss 1460,nop,nop,sackOK>

---- Good ----
12:27:07.632229 IP > S 3581365570:3581365570(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
12:27:10.620946 IP > S 3581365570:3581365570(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
12:27:10.620969 IP > S 2654770980:2654770980(0) ack 3581365571 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
12:27:10.838747 IP > . ack 1 win 4380
12:27:10.957143 IP > P 1:213(212) ack 1 win 4380
12:27:10.957152 IP > . ack 213 win 108
12:27:10.965543 IP > P 1:630(629) ack 213 win 108
12:27:10.965621 IP > F 630:630(0) ack 213 win 108
12:27:11.183540 IP > . ack 631 win 4222
12:27:11.185657 IP > F 213:213(0) ack 631 win 4222
12:27:11.185663 IP > . ack 214 win 108

Hoster: SuperbHosting

OS: Ubuntu

Server parameters: E6300 CONROE 1.86GHZ 2 X 1MB CACHE 1066 1GB DDR2 667MHZ

This is a link to apache configuration file we use

This is server-status report taken right after time-out error. There are only 10 Child Servers running out of 120, so enough space for new requests.


procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0   8900 725900   8468  65684    0    0     5    18   11   33  4  3 92  1
share|improve this question
Any chance of some detail about this server please? – Chopper3 Jan 27 '10 at 10:32
I have updated the question with the server details. Please ask for more details if I've missed something. – par Jan 27 '10 at 10:44
you talked about high load. could you post vmstat output and io wait statistic? – Christian Jan 27 '10 at 12:23
I have updated the post with vmstat output. I don't know how to dump "io wait statistics" Please tell me how to do it. – par Jan 28 '10 at 13:29
How many TCP connections are open on your server? Command is: 'netstat -an|grep tcp|wc -l' – Dave Drager Jan 28 '10 at 14:18
up vote 3 down vote accepted
  • There is an interesting thing, in case the server fails to respond, the request string is not logged to access.log by Apache.

This sounds like a network problem. The server should be logging any requests it receives even if it can't answer for some reason. You may want to verify that you aren't seeing packet loss on the web server.

share|improve this answer
Thanks! How do I know if the server looses packets? Is there a common tool/practice to profile this? – par Jan 28 '10 at 13:27
The ping command will report packet loss. Just leave it running and run the tests you normally do. – Chris Nava Jan 28 '10 at 16:25
Yes, it was the packet loss due to a large number of pending TCP connections. – par Feb 9 '10 at 20:09
You may have some luck upping the maximum number of child processes for your web server. Otherwise, I would look into moving some of the load to another server. – Chris Nava Feb 9 '10 at 21:44

There's a small chance that you're in a position where the available kernel buffers for TCP connections are low. I would expect some logging from that (log in to the server, test until you've had a "no response", then run dmesg and see if anything looks applicable).

To tune the network setup, this may be a starting point.

As Chris Nava said, it's probably worth making sure you're not just having packet loss across the network, so by all means start checking using ping (responding to a ping is, alas, not at all the same as dealing with a TCP packet).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.