We encounter some strange packet loss and want to know the reason for this.
We have an imageserver and a server for stressing the imageserver. Both are located in the same datacenter
First we run a load test like this (command shortened for readability):
ab -n 50 -c 5 http://testserver/img/de.png
The image has only about 300 Bytes. Results are very fast responses:
Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 0 Processing: 1 3 0.7 3 4 Waiting: 1 3 0.7 3 3 Total: 1 3 0.7 3 4
When we increase concurrency we see some lags (command shortened for readability):
sudo ab -n 500 -c 50 http://testserver/img/de.png
Results with concurrency 50:
Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 1 Processing: 2 35 101.6 12 614 Waiting: 2 35 101.6 12 614 Total: 3 36 101.7 12 615
so we see most requests are pretty fast, a few of them are pretty slow.
We dumped the whole network traffic with tcpdump and saw some strange retransmissions.
this dump was taken on the imageserver!
so you see that the initial package (No. 306) containing the GET request is arriving on the imageserver, but it seems the package get lost after tcpdump has logged it. It seems to me that this package does not arrive at the tomcat image server.
the retransmission is triggered by the requesting server 200ms later and everything runs fine afterwards.
Do you know any reason why a package can get lost after it was received?
Our machines are both:
- Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
- 8 GB RAM
- Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
- Debian version 5.0.5
So we do not have any problems concerning memory or cpu load.
We had some problems with our nic controller a while ago. We handled it by using a different driver, we are using now r8168 instead of r8169
But we had the same problems of lost packets with an Intel NIC - Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
So we see the same problem with equal machines but different ethernet cards.
Till now i thought packet loss would only happen between servers on the line, when the packet gets corrupted or things like that.
We really want to know what reasons there might be for those packet loss after tcpdump has logged them.
Your help is very appreciated.