0

I'm dealing with an issue in the communication between two machines. Machine A is a Windows Server 2012 box, and machine B is a Centos 8 box. Machine A is a virtual machine running under VMWare, while B is a physical x86_64 box. I don't believe there's any firewall between the two machines.

We have applications on A and B that talk to each other via TCP. Machine B opens a listening TCP port and accepts incoming connections, while machine A connects to that TCP port. This generally works fine.

However, there's a scenario which causes a failure condition. If we kill the application on A and restart it, it does establish a new connection to B. However, communication between A and B stops working on this port for some time.

What I discovered is that on A, after doing the kill/restart, a netstat shows only the newly connected TCP connection as ESTABLISHED (and doesn't show the old connection at all), while on B, a netstat shows both the old connection and the new connection as ESTABLISHED. After digging into the application running on B, I've determined that since it still sees the old connection as active (since the OS reports it as ESTABLISHED), it continues sending responses on the old connection when receiving requests on the new connection.

After some time (~2 hours?), machine B seems to detect that the old TCP connection has gone down, and starts sending responses to the correct TCP connection, so communication starts working again.

The part that's confusing to me is why the netstat output is asymmetrical. If B sees the connection as ESTABLISHED, but A doesn't see the connection at all, then my conclusion is that either A or B aren't following the TCP standard. My gut feeling is that Windows on machine A is not closing the TCP connection cleanly when the application gets killed, resulting in the connection dropping instead of closing gracefully.

If my conclusion is correct and one machine isn't doing TCP correctly, then I see a few potential solutions, but none exactly match my needs:

  1. Close the old connection on B when a new connection is detected: this is my current solution
  2. Use an application-level timeout (or TCP keepalive?) to detect when a connection goes down: this would be my preferred solution
  3. Cleanly close connections on A when the application is killed: this would be nice, but I don't control the application on A so it would be difficult to get this change made

Alternatively, is there some way to tweak a setting in Windows to make dropped connections be closed cleanly, or a configuration in CentOS to make Linux detect a dropped connection more quickly?

2
  • 1
    Is this helpful? blog.cloudflare.com/when-tcp-sockets-refuse-to-die
    – Jeff
    Commented Feb 9, 2020 at 17:32
  • 1
    I think TCP_USER_TIMEOUT actually would be a perfect solution - I might try that out. Ideally I'd like to find the root cause of the asymmetrical connection status, but if I can effectively mitigate it on machine B, then that would be good enough. Thanks!
    – millinon
    Commented Feb 9, 2020 at 18:09

0

You must log in to answer this question.

Browse other questions tagged .