0

I've been tearing my hair out for two weeks over what should be a simple networking issue. I hope some gurus on here can help me out.

Basically what's happening is that a client PC is sending a SYN packet to a linux server to establish a connection on port 80. On the first attempt the TCP handshake is successful and the connection is established. But on subsequent requests the server does not respond with a SYN-ACK. Instead it completely ignores the SYN and subsequent retransmissions. Eventually after about 5 minutes something (I don't know what) resets and the connection can be established.

What's complicating the issue is that the client PC is also making connections to port 8080 on the same server. These SYN packets are always acknowledged by the server; they are never ignored. The problem with SYN packets being ignored and the connection getting "stuck" is only related to port 80.

[UPDATE] If I shut down the webserver on port 80, CentOS sends a RST packet consistently every single time.

I noticed the client PC always uses the same source port and sequence number for it's connections which is possibly the problem? But I have enabled the tcp_tw_reuse option which should relax the restrictions.

The description is very similar to the problem reported here. However I've tried various combinations of enabling/disabling TCP window scaling and TCP timestamps but it's not made any difference.

The server is running CentOS 6.9 with kernel 2.6.32. The iptables firewall service is stopped on the server, so it's not a firewall issue. I very much doubt it's a routing problem because port 8080 always works fine. I've tried both nginx and apache with the same behaviour.

Why does the CentOS server not send a SYN-ACK in response to the SYN on subsequent requests to port 80?

Here is a link to a network diagram

...and here is my routing table:

172.18.111.193 via 10.0.47.50 dev bond0
x.x.105.72/29 dev bond0  proto kernel  scope link  src x.x.105.76
10.0.47.48/29 dev bond0  proto kernel  scope link  src 10.0.47.50
169.254.0.0/16 dev bond0  scope link  metric 1004
172.16.0.0/12 dev bond0  scope link  src 10.0.47.50
default via x.x.105.73 dev bond0

....but I really don't think routing is the issue.

Now here are some captures demonstrating the problem. Unfortunately because I'm using a libreswan VPN, I can only capture incoming traffic to the server with tcpdump. However when I capture from various other points on the network I can see the server never sends the SYN-ACK.

On the first connection attempt, connections to port 80 are successful:

15:53:53.283715 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [S], seq 576001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 6 ecr 0], length 0
15:53:53.527055 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [.], ack 138518223, win 8208, length 0
15:53:53.535034 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [P.], seq 0:48, ack 1, win 8208, length 48
15:53:54.650873 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [.], ack 144, win 8208, length 0

Looking in the conntrack table I can see the connection has transitioned to TIME_WAIT as would be expected:

tcp      6 110 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49158 dport=80 src=10.0.47.50 dst=172.18.111.193 sport=80 dport=49158 [ASSURED] mark=0 secmark=0 use=2

After a few minutes the connection table empties. However on subsequent connection attempts, SYN packets to port 80 are ignored:

15:55:22.050429 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [S], seq 512001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 4 ecr 0], length 0
15:55:29.125418 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [S], seq 512001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 15 ecr 0], length 0
15:55:53.554176 IP 172.18.111.193.49158 > 10.0.47.50.80: Flags [S], seq 512001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 63 ecr 0], length 0

Looking in the conntrack table I can see the connect to port 80 is stuck in SYN_RECV state:

tcp      6 56 SYN_RECV src=172.18.111.193 dst=10.0.47.50 sport=49158 dport=80 src=10.0.47.50 dst=172.18.111.193 sport=80 dport=49158 mark=0 secmark=0 use=2

I have to wait up to 5 minutes before connections can be established again.

During the period where SYN packets to port 80 are being ignored, SYNs sent to port 8080 are responded to absolutely fine. This seems to rule out any routing issues.

15:55:21.009836 IP 172.18.111.193.49152 > 10.0.47.50.8080: Flags [S], seq 64001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 2 ecr 0], length 0
15:55:21.149164 IP 172.18.111.193.49152 > 10.0.47.50.8080: Flags [.], ack 1394443308, win 8208, length 0
15:55:21.158766 IP 172.18.111.193.49152 > 10.0.47.50.8080: Flags [P.], seq 4294935296:4294935465, ack 1394443308, win 8208, length 169
15:55:21.159463 IP 172.18.111.193.49153 > 10.0.47.50.8080: Flags [S], seq 160001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 3 ecr 0], length 0
15:55:21.312325 IP 172.18.111.193.49153 > 10.0.47.50.8080: Flags [.], ack 1394137220, win 8208, length 0
15:55:21.327163 IP 172.18.111.193.49153 > 10.0.47.50.8080: Flags [P.], seq 4294935296:4294935573, ack 1394137220, win 8208, length 277
15:55:21.327200 IP 172.18.111.193.49152 > 10.0.47.50.8080: Flags [F.], seq 4294935465, ack 1394443658, win 8208, length 0
15:55:21.343654 IP 172.18.111.193.49154 > 10.0.47.50.8080: Flags [S], seq 224001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 3 ecr 0], length 0
15:55:21.477090 IP 172.18.111.193.49152 > 10.0.47.50.8080: Flags [.], ack 1394443659, win 8208, length 0
15:55:21.479600 IP 172.18.111.193.49154 > 10.0.47.50.8080: Flags [.], ack 1393885208, win 8208, length 0
15:55:21.505602 IP 172.18.111.193.49154 > 10.0.47.50.8080: Flags [P.], seq 4294935296:4294935567, ack 1393885208, win 8208, length 271
15:55:21.505724 IP 172.18.111.193.49153 > 10.0.47.50.8080: Flags [F.], seq 4294935573, ack 1394137598, win 8208, length 0
15:55:21.508784 IP 172.18.111.193.49155 > 10.0.47.50.8080: Flags [S], seq 288001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 3 ecr 0], length 0
15:55:21.655167 IP 172.18.111.193.49155 > 10.0.47.50.8080: Flags [.], ack 1393645647, win 8208, length 0
15:55:21.655746 IP 172.18.111.193.49153 > 10.0.47.50.8080: Flags [.], ack 1394137599, win 8208, length 0
15:55:21.666268 IP 172.18.111.193.49155 > 10.0.47.50.8080: Flags [P.], seq 4294935296:4294935485, ack 1393645647, win 8208, length 189
15:55:21.686210 IP 172.18.111.193.49156 > 10.0.47.50.8080: Flags [S], seq 384001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 4 ecr 0], length 0
15:55:21.825386 IP 172.18.111.193.49156 > 10.0.47.50.8080: Flags [.], ack 1386457421, win 8208, length 0
15:55:21.837684 IP 172.18.111.193.49156 > 10.0.47.50.8080: Flags [P.], seq 4294935296:4294935460, ack 1386457421, win 8208, length 164
15:55:21.844780 IP 172.18.111.193.49155 > 10.0.47.50.8080: Flags [F.], seq 4294935485, ack 1393646468, win 8208, length 0
15:55:21.845117 IP 172.18.111.193.49154 > 10.0.47.50.8080: Flags [F.], seq 4294935567, ack 1393886334, win 8208, length 0
15:55:21.860672 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [S], seq 448001, win 8208, options [mss 1360,nop,wscale 0,nop,nop,TS val 4 ecr 0], length 0
15:55:21.994423 IP 172.18.111.193.49155 > 10.0.47.50.8080: Flags [.], ack 1393646469, win 8208, length 0
15:55:22.005591 IP 172.18.111.193.49154 > 10.0.47.50.8080: Flags [.], ack 1393886335, win 8208, length 0
15:55:22.005792 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [.], ack 1386805434, win 8208, length 0
15:55:22.015906 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [P.], seq 4294903296:4294903481, ack 1386805434, win 8208, length 185
15:55:22.025923 IP 172.18.111.193.49156 > 10.0.47.50.8080: Flags [F.], seq 4294935460, ack 1386457708, win 8208, length 0
15:55:22.206066 IP 172.18.111.193.49156 > 10.0.47.50.8080: Flags [.], ack 1386457709, win 8208, length 0
15:55:22.595963 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [.], ack 1386806911, win 6731, length 0
15:57:24.923029 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [.], ack 1386806912, win 6731, length 0
15:57:25.132441 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [.], ack 1386806912, win 6731, length 0
15:57:54.913633 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [F.], seq 4294903481, ack 1386806912, win 8208, length 0
15:57:56.866865 IP 172.18.111.193.49157 > 10.0.47.50.8080: Flags [F.], seq 4294903481, ack 1386806912, win 8208, length 0

...and this is the connection table for requests to port 8080 which have all been successful:

tcp      6 109 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49156 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49156 [ASSURED] mark=0 secmark=0 use=2
tcp      6 108 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49152 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49152 [ASSURED] mark=0 secmark=0 use=2
tcp      6 109 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49155 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49155 [ASSURED] mark=0 secmark=0 use=2
tcp      6 109 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49154 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49154 [ASSURED] mark=0 secmark=0 use=2
tcp      6 201 ESTABLISHED src=172.18.111.193 dst=10.0.47.50 sport=49159 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49159 [ASSURED] mark=0 secmark=0 use=2
tcp      6 108 TIME_WAIT src=172.18.111.193 dst=10.0.47.50 sport=49153 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49153 [ASSURED] mark=0 secmark=0 use=2
tcp      6 431989 ESTABLISHED src=172.18.111.193 dst=10.0.47.50 sport=49157 dport=8080 src=10.0.47.50 dst=172.18.111.193 sport=8080 dport=49157 [ASSURED] mark=0 secmark=0 use=2

I have tried all sorts of setting combinations in sysctl.conf but none make any difference. These are my current settings:

net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_syncookies = 1

I even tried reducing the SYN_RECV timeout to just 4 seconds, but even after the timeout is reached, when a SYN retransmission is received from the client the connection returns to SYN_RECV state.

Really appreciate any help you can offer.

  • I would suggest concentrating on the abomination of a client that is using the same port and sequence numbers for multiple connections. It should at least be using a sequence number greater than the previous connection. – Brian Mar 10 '18 at 17:53
  • Port 80 (HTTP) is often live-scanned by UTM firewall. If you control the border/gateway firewall, give a look at its logs. – shodanshok Mar 10 '18 at 17:59
  • @Brian Yes, I know it's horrible. Unfortunately it's a legacy application from a company that's no longer trading so I'm stuck with it. :( – John Mar 10 '18 at 18:15
  • @shodanshok The traffic is not traversing a firewall. The IPSec VPN tunnel has been setup so no traffic from the internal network has to go via the public internet, or through any firewalls. – John Mar 10 '18 at 18:16
  • Interesting thought though.... if the problem actually is the sequence numbers, then how long does the kernel cache the sequence for? Perhaps it's possible to reduce the caching time to workaround the issue? – John Mar 10 '18 at 18:26

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.