We have a server that receives some data, acting as a TCP client, processes it in some way, and serving processed data to clients, acting as a TCP server. It also stores this data on disk and can serve it from files instead of real-time stream.

The problem is that this service has to be available in 24x7 mode, no interruptions allowed. Right now it is done by having two servers, one acting as a hot backup - clients maintain connections to both servers, and if something happens to the primary server, they just switch to the backup. While this solution works for about 15 years already, it is kind of inconvenient and puts a lot of failover logic on the clients.

Lately people started talking about using a cluster to ensure availability of this service, but no matter how hard I search I just can't find any clustering solutions that would allow transparent TCP connection failover so no-one would notice that something happened to the server. There are some research papers around, but I was unable to find any working implementations. Here is how I think it should work:

  1. Both servers receive the data via TCP. Ideally it should look like a single connection to the "outside" world, to save bandwidth and, more importantly, to ensure that both servers receive identical data streams.

  2. When a client connects to the cluster IP, it receives the processed data in a single connection, but both servers should see this connection and provide the data, it is just that only one of the streams actually reaches the client, the backup one goes to /dev/null, so to say.

  3. When the server fails (it doesn't transmit any data for some time, say, 5 seconds), the client should continue receiving the same stream within the same connection. It needs to happen pretty fast, so the overall streaming delay doesn't exceed approximately 10 seconds.

Reliability is the most important thing here. Quick failover is the next one. Open source Linux solutions are preferred, but if commercial and/or non-Linux near-perfect solutions exist, I would like to know about them too. Solutions that impose a lot of restrictions or require modifications of the server application software are perfectly acceptable too.

migrated from superuser.com Aug 8 '11 at 20:47

This question came from our site for computer enthusiasts and power users.

  • This sounds immensely complex.. is it OK if the servers are not completely in-sync? How would the hot backup server know that at exactly which byte to continue sending the stream (for each client)? – Karoly Horvath Aug 8 '11 at 21:49
  • This is why Sun storage servers have used UDP for so long, UDP generation is easy to failover, TCP is just not. – Steve-o Sep 1 '11 at 13:14
  • @yi_H, the only way I can think of is to sync the backup server before sending the reply to the client, possibly using some kind of virtualization, like this paper describes. But I am not aware of any real implementations of this. – Sergei Tachenov Sep 15 '11 at 13:35

You should look at HAProxy. HAProxy is usually run in HTTP mode, but it can handle raw TCP connections as well. It supports load balancing between servers and can use Heartbeat to detect if an instance is down.

If you setup needs to be totally transparent (servers getting the source IPs versus that of the HAProxy server), you might have to patch your Linux kernal for TProxy or find a Linux distribution that supports TProxy within the kernel or as a module.

That's the best open source solution. If you need something more comprehensive than that, you'd have to look at commercial offerings such as Citrix Netscaler for F5's BigIP.

  • HAProxy looks like an interesting project, but more for load balancing (which we absolutely don't need). It says something about high availability there too, but I only see more points of failure introduced instead, and it doesn't seem to solve transparent failover problem at all. – Sergei Tachenov Sep 15 '11 at 15:30

You could get a PhD in this stuff -- it's an immensely complicated problem. Or, you could take the easy approach and fix the protocol so it wasn't so temperamental about connection failure. SMTP is a decent model for how to avoid most forms of failure-induced data loss.

  • Is it much easier to implement with connection failure allowed? The user would have to detect connection failure pretty quickly and reconnect when the failover is complete, so the overall delay doesn't exceed 5-10 seconds. What if it attempts to reconnect during the failover? – Sergei Tachenov Sep 15 '11 at 15:23
  • Keep trying to connect until you get through. – womble Sep 17 '11 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.