our server suffered a serious connection timeout problem, so we track tcp connection with tcptrack

we found out that, if the client started to connect to the server, tcptrack shows the connection, but in SYN_SENT status, and netstat -nat shows nothing. (tcptrack & netstat all runs on the server)

  1. does this mean the syn request reached the server? and no syn/ack was sent back?
  2. why the tcptrack could report this connection but netstat could not?
  3. what could be the problem that a general apache could not establish a connection with the client?

i did a bench test using ab in the same intranet, to the specified NIC, it handled 10000 concurrent connection and 400000 requests ok

ps: this doesn't happen every time, but did happened a lot

pps: is there any good tools to trace where the tcp connection was lost?

  • failed to mention that, tcptrack & netstat all runs on the server
    – xpu
    Nov 7 '11 at 6:09

It means the SYN was sent by the client and either didn't reach the server, the server didn't reply to it, or the server opted to reply to it without keeping track of it. The server does not need to keep track of every SYN reply it sends (and can use SYN cookies) because they may be spoofed and doing so creates a risk of denial of service attacks.

  • /proc/sys/net/ipv4/tcp_syncookies is set to 1
    – xpu
    Nov 7 '11 at 5:56
  • so did the syn packet reach the server anyway? tcptrack shows syn_sent connection, but netstat shows otherwise, which one is telling the truth? the tcp connection failed to establish, because of the server did not receive the syn request, or failed to reply it?
    – xpu
    Nov 7 '11 at 5:59
  • another possibility is : the server didn't reply to it yet (ie, let the time for the server to receive the packet, process it, and send the answer back) (iow, if you see a few "syn_sent" in netstat -an, it could be just that you happen to interrogate the connection juste before the server had the time to answer) Dec 22 '17 at 13:09

When I'm getting 'unwanted' traffic, as in traffic that has been specifically blocked by IPTABLES rules (as in being DROPped), the tcptrack shows the incoming IP address along with the SYN_SENT status (along with time of connection and data rate of 0b/sec). That listing stays there for a few seconds until it clears.

So - it's possible that the connections you're seeing are blocked for some reason. The IP addresses that come up with SYN_SENT could be locked out due to IPTABLES DROPs. You could disable IPTABLES for a bit and see if it continues. If so, make sure that the addresses being blocked are supposed to be.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.