The problem that we are facing is that some of the http connection have response time > 60s (about 5%). I discover that the problem should between the web server and loadbalancer.
Here is my finding, we have tried two set of server:
Setup A : only 1 web server (Server A), all the tcp traffic is directly point to this server.
Setup B : loadbalancer + Server A, the weight of Server A is 100. With the algorithm "Round Robin with Persistent IP"
For the Setup A, the tcp connection is really stable, the timeout ratio is less than 1% However, for the setup B, the timeout ratio is more than 5%, and here is the problem. (the connection timeout set on the client is 60s)
We have test these two setting in a common environment (in a 10 minute time-frame), which have nearest packet number (about 700,000 packet) and traffic. As a result, we got 2 set of tcpdump, i have discovered some strange log entries and counted them as follow:
Setup A Setup B
TCP Zero window size 0 611
TCP Window Full 0 3672
TCP Out-Of-Order 4147 4577
TCP Retransmission 23665 21551
TCP Dup Ack 10592 10121
For the above result, I am quite sure that this problem about TCP window, So i have tried to enable net.ipv4.tcp_window_scaling > reboot, but this does not help. I have also tried to disable iptables, does not help too. I dont know if there have any config is affecting the TCP window.
One thing worth to know is, our loadbalancer ip is xx.xx.117.128, All the packets marked as TCP Window Full are from Server A to xx.xx.117.25 And all the packets marked as TCP Zero window size are from xx.xx.117.25 to Server A
I have asked softlayer technician that what xx.xx.117.25 is and they said "xx.xx.117.25 is the address from which the load balancer will connect to your real servers" They guess this is a firewall issue, as i mentioned above, i have tested with iptables off. So we can eliminate this factor
That is what i discovered so far.
Maybe you are interested in the sysctl config and here it is:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 1000
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 20
here is a snapshot of tcp status of Server A in Setup A
604 TIME_WAIT
7 SYN_RECV
1 LISTEN
2 FIN_WAIT1
1 ESTABLISHED
1 CLOSING
Not really sure why the TIME_WAIT is so high (i have entable tcp_tw_reuse and tcp_tw_recycle) I have monitored the tcp status on Setup B as well, the number of TIME_WAIT is even more less(about 300 - 400)
for the apache config:
KeepAlive Off
<IfModule prefork.c>
StartServers 5
MinSpareServers 10
MaxSpareServers 50
ServerLimit 500
MaxClients 500
MaxRequestsPerChild 4000
</IfModule>
Please help. Thank you so much