0

The problem that we are facing is that some of the http connection have response time > 60s (about 5%). I discover that the problem should between the web server and loadbalancer.

Here is my finding, we have tried two set of server:

Setup A : only 1 web server (Server A), all the tcp traffic is directly point to this server.

Setup B : loadbalancer + Server A, the weight of Server A is 100. With the algorithm "Round Robin with Persistent IP"

For the Setup A, the tcp connection is really stable, the timeout ratio is less than 1% However, for the setup B, the timeout ratio is more than 5%, and here is the problem. (the connection timeout set on the client is 60s)

We have test these two setting in a common environment (in a 10 minute time-frame), which have nearest packet number (about 700,000 packet) and traffic. As a result, we got 2 set of tcpdump, i have discovered some strange log entries and counted them as follow:

                            Setup A                Setup B
TCP Zero window size        0                      611
TCP Window Full             0                      3672
TCP Out-Of-Order            4147                   4577
TCP Retransmission          23665                  21551
TCP Dup Ack                 10592                  10121

For the above result, I am quite sure that this problem about TCP window, So i have tried to enable net.ipv4.tcp_window_scaling > reboot, but this does not help. I have also tried to disable iptables, does not help too. I dont know if there have any config is affecting the TCP window.

One thing worth to know is, our loadbalancer ip is xx.xx.117.128, All the packets marked as TCP Window Full are from Server A to xx.xx.117.25 And all the packets marked as TCP Zero window size are from xx.xx.117.25 to Server A

I have asked softlayer technician that what xx.xx.117.25 is and they said "xx.xx.117.25 is the address from which the load balancer will connect to your real servers" They guess this is a firewall issue, as i mentioned above, i have tested with iptables off. So we can eliminate this factor

That is what i discovered so far.

Maybe you are interested in the sysctl config and here it is:

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 1000
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 20

here is a snapshot of tcp status of Server A in Setup A

604 TIME_WAIT
7 SYN_RECV
1 LISTEN
2 FIN_WAIT1
1 ESTABLISHED
1 CLOSING

Not really sure why the TIME_WAIT is so high (i have entable tcp_tw_reuse and tcp_tw_recycle) I have monitored the tcp status on Setup B as well, the number of TIME_WAIT is even more less(about 300 - 400)

for the apache config:

KeepAlive Off
<IfModule prefork.c>
StartServers       5
MinSpareServers   10
MaxSpareServers   50
ServerLimit      500
MaxClients       500
MaxRequestsPerChild  4000
</IfModule>

Please help. Thank you so much

1 Answer 1

0

Have you tried your setup without tcp_tw_recycle and tcp_tw_reuse options? At least tcp_tw_recycle can cause problems with load balancers.

Also, the number of sockets in TIME_WAIT state shouldn't be a problem, since it is nowhere near 30k, which is the default number of ports available in Linux.

If you want to be sure that there are enough ports available, you can set net.ipv4.ip_local_port_range sysctl to 1024 65535.

1
  • I have cancelled the loadbalancer service before I got this answer. So i cannot really test with tcp_tw_recycle and tcp_tw_reuse off. I think this probably cause the problem and ill have a try once i got my loadbalancer back. Thanks anyway :).
    – Chris
    Apr 28, 2014 at 9:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .