approach to re-route (plain-text) protocols for machines behind a router that are down

Question

I've been looking for some time now, how to achieve that, but I can't seem to find similar solutions or synthesize one from my knowledge.

Here are the details about my setup: I have one gateway/router that has the uplink to the internet. "Behind" that are several different servers. Some of those have a public IP, others don't. Either way, all the services I wanted to expose are exposed by either unblocking the respective port on the target IP or DNATing the port through to one of the machines with no public IP. That whole setup works nicely (for the bigger part it's a netfilter-based ruleset in combination with a few routing table entries).

Now the problem: sometimes machines behind the router can be down (I can't afford extra redundancy at the moment). If that happens, I would like the user of a service - in particular HTTP - to have a "better" user experience than a network timeout and offer a "backup" server that would only display a short notice about the fact that the service behind the router is down at the moment. So if the router can't reach the service, it should re-route the traffic dynamically to the "backup".

Does anyone know of a solution to this problem or can suggest a way how to approach it? Suggestions for search terms are also welcome ...

Answer 1

One solution would be to use a loadbalancer like HAProxy. It supports the concept of a backup server, so you can route your traffic to your normal server as long as it answers to the regular checks the loadbalancer does. If it doesn't answer anymore, traffic can get routed to backup servers which in your case could simply display a "sorry" note.

The downside of this approach is of course that you have to maintain the loadbalancer, which might be another single point of failure. If your gateway/router is an actual linux system (as opposed to some plastic box), this is probably not an issue, as it already is a single point of failure and can probably also run the loadbalancer :)

More general, the problem to solve is not to redirect the traffic (there are many solutions for that) but to actually detect that a host is down and then to change the routing/redirection accordingly. The same is needed when the server goes up again, so that traffic is routed to the actual server agin.

Answer 2

The cheapest solution, you'd have to write a script which polls the backend server and updates your port forwarding when it goes down. You'd forward the traffic to a different server which contains your error message.

The details of the script depend completely on the specific details of your systems. Some very rough pseudo bash code:

wget backendserver > /dev/null
test_for_failure
if failed; then
  iptables -D <port forward rule>
  iptables -A <new forward rule>
else
  verify iptables is sending to the backend
fi

In our environment, all traffic is forwarded to one server, which then proxies to the backends. Its setup to display proper messages during outages. However it can still fail causing timeouts. Its dedicated to its task though, making outages rare. The next step would be to make it part of a cluster, to increase uptime; for us the cost/benefit isn't there.