1

I am troubleshooting a connection timeout problem between two Linux boxes, where it seems the ACK to SYN-ACK was lost on the server's stack.

The tcpdump is done on the server side.

The client got syn-ack, sent ACK and data packet, and resent data 4 more times. The server resent syn-ack 4 seconds after it sent the syn-ack, indicating the ACK from the client is lost on the server's stack. The client responds with an ACK.

Then around 3s later the client resent data, and got the server's ACK. The client sent FIN at 10s because the client app has set a 10s timeout.

So the question is: the tcpdump shows the ACK to the SYN-ACK arrived at the server. Under which case could the server resend the SYN-ACK? Is it kernel or app issue on the server side? And how to debug further?

Appreciate your help.


20:31:01.159098 IP client.cport > server.sport: S 2848162415:2848162415(0) win 5840 
20:31:01.159103 IP server.sport > client.cport: S 901143055:901143055(0) ack 2848162416 win 5792 
20:31:01.159192 IP client.cport > server.sport: . ack 1 win 46 
20:31:01.159276 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:01.380395 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:01.824367 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:02.712362 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:04.488358 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:05.159038 IP server.sport > client.cport: S 901143055:901143055(0) ack 2848162416 win 5792 
20:31:05.159157 IP client.cport > server.sport: . ack 1 win 46 
20:31:08.040317 IP client.cport > server.sport: P 1:426(425) ack 1 win 46 
20:31:08.040326 IP server.sport > client.cport: . ack 426 win 27 
20:31:11.159618 IP client.cport > server.sport: F 426:426(0) ack 1 win 46 
20:31:11.199139 IP server.sport > client.cport: . ack 427 win 27 
20:31:14.724604 IP server.sport > client.cport: . 1:1449(1448) ack 427 win 27 
20:31:14.724612 IP server.sport > client.cport: P 1449:1756(307) ack 427 win 27 
20:31:14.724776 IP client.cport > server.sport: R 2848162842:2848162842(0) win 0
20:31:14.724779 IP client.cport > server.sport: R 2848162842:2848162842(0) win 0

Edit: There are tons of overflow. Could the app has a low listen() backlog leading to this problem?

$ netstat -s | grep -i list
    210473 times the listen queue of a socket overflowed
    210473 SYNs to LISTEN sockets ignored

Edit 2: this is my first post here so please stay with me. :)

Client & Server kernel: 2.6.18-92.el5

Server app: listens on sport only. It processes client request, and response back. Using strace I did find out the listen() backlog is 5.

There are 8 clients systems, each running one instance of client app. The client app sends req to the server port, get response back from server. A 10s timer is set after client succeeds in connect(). Each client instance could send multiple connection requests, each on its own client port.

There could be a burst of concurrent requests from the 8 clients.

Edit 3: The tcpdump looks very similar to the one at http://forum.openvz.org/index.php?t=msg&goto=25678 , but no root cause / solution.

6
  • If it is not completing a handshake, then I would guess that you have a firewall/filter somewhere dropping packets, or maybe a routing issue.
    – Zoredache
    Mar 29, 2013 at 19:38
  • serverfault.com/questions/336132/… does this help ?
    – user9517
    Mar 29, 2013 at 19:45
  • @Zoredache: tcpdump on server side (as posted) sees the ACK, so it's not routing. Also 'iptables -L -n' shows no firewall at all.
    – wilsonh
    Mar 29, 2013 at 20:03
  • @lain: It's different. Question 336132 was for FIN/FINACK, but this is for the ACK to SYNACK.
    – wilsonh
    Mar 29, 2013 at 20:07
  • If listen queue is overflowed, my guess is the server wouldn't respond with SYN-ACK. As it responded, it doesn't seem to be this problem. Can you look further into the 'mangle' and 'nat' table if they are both empty? What about trying different kernel/rebooting the server?
    – ondra
    Mar 30, 2013 at 9:14

2 Answers 2

2

I'd ask myself why would normal client:

  1. shrink its previously advertised data window 5840 to 46
  2. re-send the same data segment for 5 times (3 of them within 1 second!)

Also, if you need help of community, you'd better think that we've lost our telepathy abilities long ago and thus we haven't even a piece of idea what soft is involved into this (starting from Linux kernel version) and what and how it's supposed to work.

2
  • Re 2) The client TCP should be doing the retransmission, using exponential backup. It seems RFC 2988 states the min is 1 sec. The client host has multiple apps communicating to the server, could that change the TCP retransmission timtout? Re 1) I will do some research on data window. Also will edit the question with more details.
    – wilsonh
    Apr 10, 2013 at 18:06
  • Somehow the 'tcpdump' did not show the wscale = 7. Add wscale to the equation the window size is normal: 46 * 2^7 = 5888. Got the answer from stackoverflow.com/questions/3254574/…
    – wilsonh
    Apr 10, 2013 at 21:38
1

Your backlog is too small to accepts all your request. The backlog is the size of your accept queue, when the accept queue is full , the client's ACK is ignored or connection is aborted according to net.ipv4.tcp_abort_on_overflow. So increase your backlog .

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .