4

I am running into issues with a site that is using DNS round robin for failover.

There are two webservers. When I turn off apache on one box, everything is fine. When browsers try to go to the webserver with apache turned off, it gets a connection refused and it automatically tries the other one.

However, when rebooting, I run into problems. When I reboot the server, instead of connection refused, the browsers now wait for a timeout leading to a really slow site until the server comes back up.

Is there a way to set up the network (or CentOS) such that during the reboots, that IP continues to reject instead of taking a long time to timeout?

1

As an ad hoc workaround (only in the regretful absence of a loadbalancer), you could accompany the server reboot with suitable configurations at one of its next hops (a switch, a router, a firewall), e.g.,

  1. stop apache
  2. Make the next hop respond with explicit "not reachable" messages
  3. reboot the server
  4. Undo changes made step 2

Specifically, step 2 might consist of one of the following

  • adding the web server ip as secondary ip address to one of its interfaces (provided there is no web server running on it)
  • or create a static arp entry (i.e., ip to mac translation) that refers to a runnnng host that does not run a webserver
  • or make that single ip non-routable
  • or add a firewall rule to reject (not drop!) port 80 traffic to the ip in question
  • or modify an existing SNAT entry for your webserver to refer to a host that is running but not a webserver
  • or attach a dummy host with the same MAC address to the same lan

Possibly other variants exist as well. There is of course one important point: Don't shoot yourself in the foot! For example,

  • You might accidentally break other things when introducing such misconfigurations
  • You might forget to undo the changes
  • your server might not be able to properly close existing connections
  • Your rebooted server might be unhappy if it cannot see the internet immediately due to this method (it might desperately want to check for updates, or licences, or time, or ...)

Also, i did not test any of these methods myself, I just came up with them and think they should work.

| improve this answer | |
  • 2
    Actually, why the downvotes? I'm aware that these methods are not the fine art. But as I said, these methods are intended only for the case that a better and automatic solution is not available. And instead of suggesting aditional systems (ok, the last suggestion with a dummy host with the same MAC would be an additional system), this does answer the OP's question whether there is a way to set up (i.e., merely reconfigure) the existing network such that during the reboots, that IP continues to reject instead of taking a long time to timeout. – Hagen von Eitzen Jan 18 '15 at 11:01
9

Use a Loadbalancer Software with healthcheck like Zen-Loadbalancer or HA-Proxy.

There you can configure a healthcheck which pings or open a file on your Webserver and remove them from the Load-Balancing if the file is not available for example.

DNS Round-robin is not made for such things because your DNS doesn't know if a server is not available. The DNS give the requests alternately to every ip.

| improve this answer | |
5

Just to explain: DNS Round Robin is not designed for the purpose you are using it for. It simply tells the DNS to hand out the available IP-addresses in an alternating manner; there is no feedback between the failing server and the DNS (and if there were, timing problems would prevent it to be of much help, because of the way the DNS is designed). So you can use DNS-Round-Robin for loadbalancing (in a limited way) but not for failover.

See this quote: "If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service." from http://en.wikipedia.org/wiki/Round-robin_DNS

The reason you did not have any problems with the webserver shut down, is probably, that your browser sticks to the webserver that works, maybe because of some session cookie or the like. You should try your experiment with the web server shut down several times, using completely new sessions - you should experience the same problems as with reboots then.

Besides from that I would second the answer already given: a loadbalancer solution would do what you want.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.