4

We have a Windows Server 2012 R2 that hosts our websites on IIS. We also have an Ubuntu 16.04 server which runs Nginx 1.10.3 to proxy incoming requests to our backend Windows server. Both these servers run as VMs on ESXi.

We have noticed that our Windows Server sometimes takes too long to send SYN-ACK in response to incoming SYNs.

Here's what the windump output on the Windows server looks like (as you can see, only after 63 seconds and 7 SYNs Windows has sent the corresponding SYN-ACK):

11:26:59.080471 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60011765 ecr 0,nop,wscale 7], length 0
11:27:00.075553 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60012015 ecr 0,nop,wscale 7], length 0
11:27:02.078881 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60012516 ecr 0,nop,wscale 7], length 0
11:27:06.086875 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60013518 ecr 0,nop,wscale 7], length 0
11:27:14.094838 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60015520 ecr 0,nop,wscale 7], length 0
11:27:30.126966 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60019528 ecr 0,nop,wscale 7], length 0
11:28:02.224731 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [S], seq 3338047317, win 29200, options [mss 1460,sackOK,TS val 60027552 ecr 0,nop,wscale 7], length 0
11:28:02.224789 IP 192.168.20.2.80 > 192.168.20.129.41784: Flags [S.], seq 2819099122, ack 3338047318, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 215763098 ecr 60027552], length 0
11:28:02.225363 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 60027552 ecr 215763098], length 0
11:28:02.225900 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 60027552 ecr 215763098], length 75: HTTP: GET /ping?id=141 HTTP/1.1[!http]
11:28:02.248577 IP 192.168.20.2.80 > 192.168.20.129.41784: Flags [FP.], seq 1:224, ack 76, win 260, options [nop,nop,TS val 215763100 ecr 60027552], length 223: HTTP: HTTP/1.1 200 OK
11:28:02.253096 IP 192.168.20.129.41784 > 192.168.20.2.80: Flags [F.], seq 76, ack 225, win 237, options [nop,nop,TS val 60027559 ecr 215763100], length 0
11:28:02.253144 IP 192.168.20.2.80 > 192.168.20.129.41784: Flags [.], ack 77, win 260, options [nop,nop,TS val 215763101 ecr 60027559], length 0

The strange thing is that if we change the source IP (via Nginx's proxy_bind) or the destination port (in IIS), response times are greatly enhanced.

How can we find out what's causing this behavior?

Update 1: We changed the TcpTimedWaitDelay parameter to 30 seconds and the situation is much better now but we still have the problem.

Update 2: Here's the sum of connection states that netstats reports:

  64 CLOSE_WAIT
1371 ESTABLISHED
   1 FIN_WAIT_1
  51 LISTENING
3188 TIME_WAIT
  • How many connections does it have in SYN_RECV state? Silently dropping incoming SYN packets is a plausible reaction to too many connections in SYN_RECV state. – kasperd Dec 18 '17 at 13:11
  • @kasperd I checked it at different times and there seems to be none. – soheilpro Dec 19 '17 at 7:06
  • The number of connections in TIME_WAIT sounds high, but I don't know if that is your problem. The number of connections in ESTABLISHED is high, but not excessive. Do you know how many simultaneous connections your web server can handle? – kasperd Dec 19 '17 at 10:25
  • How did you rule out problems in 1st and 2nd ISO/OSI layers? – Michal Sokolowski Dec 21 '17 at 2:35
1

To my knowledge, it's possible that the good old Nagle is kicking your balls here. I would recommend you to test switching it off and also reduce the internal ticks-to-reply counter.

This is done in the registry:

  • In HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{YOUR-NIC}
    • Create a DWORD TCPNoDelay with the value of '1'
    • Create a DWORD TcpAckFrequency with the value of '1'
    • Create a DWORD TcpDelAckTicks with the value of '0' (more)

A reboot is required to activate those. You can check your Settings with Get-NetTCPSetting.

|improve this answer|||||
  • 1
    Nagle applies when there is unacknowledged data in flight. However the problem is seen during connection establishment thus before any data has been transmitted. So it can't be Nagle which is causing the problem. – kasperd Dec 18 '17 at 13:07
  • Yes, thats right, but proxy connections are not always stateless (since http/2). If TcpTimedWaitDelay helps (which is just dropping and reusing the connection), what is your MaxFreeTcbsset to? If all TCBs are used and more connections are requested, Windows will prematurely drop connections in TIME_WAIT state. So lowering this (<2k) could help freeing up more "unused" connections. – bjoster Dec 19 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.