We're in the planning phase to make our web application services redundant. Right now we've got our own physical servers at a colo, running a VMWare cluster connected to an EqualLogic SAN. It's a LAMP setup. We want to set up a second site, either for load balancing or for as an active/passive failover (I think we've been leaning towards the latter, but no decisions have been made).

On the drawing board, we came up with what we thought was a simple solution. ourdomain.com resolves to, which is the IP address of a load balancer, or a failover service or a something like apache's reverse proxy.. where requests come in to it and it forwards the requests to the appropriate data center. That way if data center A goes down, we simply change the load balancer to send all requests to data center B.

We have failed at being able to find ANYONE that offers this type of service. Everyone we ask about it (like X0 and L3) say they don't really know if we'll find something like that. Our end goal is to have redundancy between two sites to minimize downtime, whether it be hardware failure or an entire data center goes offline due to natural disaster. We describe this set up to all sorts of vendors and none are familiar with any such service..

The best idea we've encountered is using failover DNS. We currently use dnsmadeeasy.com, and if their monitors detect that site A has gone dark, they'll change DNS to resolve to the IP of site B. We've done some tests and even with our TTL at 1 minute, it took around 15 minutes on average for DNS servers to get the change, while some DNS providers we queried overseas (like Australia, which is important to us) took almost an hour to make that change. That's not good enough.

So what am I missing?

  • Does a hosted failover solution exist? Why do all these big name companies act like they've never heard of such a concept before when it seems like it should be fairly common?
  • Is mod_proxy something that will work for us? If so, can you set up a cluster of apache server's doing mod_proxy so IT doesn't become a single point of failure?
  • Is there a better solution for redundant sites that you could suggest?

To answer your questions:

  • Yes, a hosted failover solution exists. It's not commonly provided or advertised because it's not a common requirement. The cost of a redundant site, plus the cost to do any sort of GLB properly is really expensive. Normally when we tell a customer how much it's going to be, they go slightly pale and suddenly they can live with a little more downtime than they were previously anticipating.
  • I wouldn't use Apache, but you can do it with a proxy balancer of some kind. The problem is that you need to make that geographically-distributed, too -- and since your proxy will add latency, you need to make sure you've got them available quite close to your customers to minimise that latency penalty. There's a reason why Google and Akamai try to make sure they've got a few racks of kit very close to (preferably colocated in) large ISPs.
  • Rather than proxies, I'd just use BGP failover to provide active-passive, with GLB-DNS to provide multi-active for optimum performance (no point having a second facility sitting around doing nothing for 1576785 minutes out of every 1576800 -- you may as well get some on-going benefit out of it).

Personally, I'd recommend you take a deep breath and really think about whether near-zero-downtime geo-redundancy is really necessary. Really, what are you doing here that is going to cost you $2M+ for a 15 minute outage? Because that's the sort of cost-per-outage-event you're looking at to provide a basic form of this sort of infrastructure (assuming you're using good facilities that don't go down regularly). Between the engineering effort required to set something like this up and keep it running, and the cost involved in re-engineering your application to work properly in this sort of distributed environment, you're going to be dropping a lot of coin on this, and it'll be a constant, on-going cost to keep it up (think of all the features you won't be able to implement because it won't work right across multiple sites, let alone the monthly costs on running all the extra infrastructure).

  • Thank you for the reply. I'm now looking at Akamai's Global Traffic Management, which has a failover option. On paper it looks like what we're looking for. I'm awaiting a call from their tech specialists to call and go over some of the finer details with. We've done some more tests with DNSME's dns failover and using whatsmydns.net to test and it looks like we get complete failover within 3 minutes, which isn't all that bad. I'm wondering if my first test were invalid because some sites still had the old 60 minute ttl. – Safado Jul 9 '12 at 17:12
  • At any rate, this does put it into perspective a little bit. If akamai doesn't work out, a 5 minute downtime window will probably be OK. Now to figure out how to make mysql master-master replication stable between our two sites! – Safado Jul 9 '12 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.