0

We have some servers which run a web-app, all the servers run in a single datacenter and we've never had any issues. However as we start to get bigger I've had to think about what we're going to do if our datacenter goes down. It's not cost-effective for us to have servers in two different datacenters running 24/7 so my current plan is we have our main servers running like normal, but a 'hot' database server running on a 'cloud'/VPS server provider constantly keeping in sync with the main database servers but with no application servers connecting directly to it. Then when our datacenter goes down we clone the database server to give us enough capacity and spin up some new application servers minimizing downtime to a few minutes.

The issue I have is figuring out how to failover to the cloud servers. I don't want to use DNS round-robin because under normal conditions no requests should be going to our VPS', I also want to avoid using DNS failover (I.E. when our DC goes down we update the DNS to point to the new servers) because, in my experiance, qutie a few ISPs don't honor DNS TTLs and will cache records for days.

I'm not looking for an exact guide on how to do this, just some topics I should look into. I've looked at IP {any,multi,broad}cast but they don't seem relevant to what we're trying to do (And I'm not sure it's possible to have an IP point to multiple servers from multiple providers on different networks, but I might be wrong). I also don't want to put a proxy/load-balancer infront of all requests, again this would require equiptment in a seperate datacenter and probably reduce performance.

3 Answers 3

0

So, if your site is reachable at www.example.com and you want www.example.com to go 'somewhere else' at certain times, there aren't a ton of choices. As you said, dns failover won't work for you, because you want the cold site carrying 0 traffic and instant fail back and forth. So, we're looking at some IP based failover for 1.2.3.4. You could very quickly make another announcement for 1.2.3.4 in another DC, but that requires routers and BGP and is most likely not something you have expertise in. So, the only alternative I can see is making 1.2.3.4 be a "front end" device that redirects queries to the appropriate location. Something from maybe this product family: http://www.cisco.com/en/US/products/hw/contnetw/index.html (but I'm a cisco person, so I might be biased).

2
  • If we were to go with a device in front of our main application, we'd need to have it in at least two locations/datacenters else when our primary DC went down we'd loose our frontend anyway. Also those Cisco boxes look pretty expensive
    – user80776
    Aug 15, 2011 at 13:19
  • Generally speaking, you can either do it right, or do it cheaply. Your statement that you want to avoid 'adding complexity' and at the same time this will be more complex, period, makes it nontrivial to come up with the right solution. Or you could just use a cloud load balancer: rackspace.com/cloud/cloud_hosting_products/loadbalancers
    – Aaron
    Aug 15, 2011 at 14:21
0

There are several companies that offer DNS that will do failover to a second machine if the 1st is not available.
But before you go out and look at all this, what is the reliability of your current datacenter?

Does it have redundant Internet connections from different providers?

Does it have redundant power from different power feeds?

Does it have backup UPS & Generator of at least N +1?

What class facility is it? Is it a Facility like Terremark, Level 3, etc or a local mom & pop ISP?

What does downtime cost you per Day/Hour/Minute? Is the cost of mitigating this justified?

4
  • As mentioned in the question, I don't want to use DNS failover because some ISPs don't honor TTLs. Our current datacenter is very stable (Telecity, UK), we've been with them for 18 months and never had any problems. However we need to have a plan for if something does go wrong. If we were running a backup DB server in 'the cloud' our normal running cost of our backup system will be <$100/month. Although we don't know our exact cost of downtime, it's certainly more than the cost of running the cloud servers, and the $100/month is an acceptable cost to get a fairly reliable backup
    – user80776
    Aug 15, 2011 at 13:16
  • Why not just host another server in a different datacenter with Telecity. And have them worry about failover, or for that matter use their cloud as failover. Then they can handle all the issues for you.
    – tkrabec
    Aug 15, 2011 at 15:43
  • Because that requires the upfront cost of having spare hardware in a second DC, with using a public cloud like EC2/Rackspace we just need one tiny server keeping in sync with the database until we need to failover, then we just scale up. We don't want to split our servers over multiple DCs as low latency is critical to our application so we need to reduce it wherever possible.
    – user80776
    Aug 15, 2011 at 18:31
  • Have you checked with Telecity to see if they can provide the routing/DNS/IP changes for you to go with an EC2/Rackspace backup? Also does Telecity provide a cloud offering that would meet your needs?
    – tkrabec
    Aug 17, 2011 at 11:28
0

What about a cloud based load balancing solution, effectively configuring the load balancing between your two sites?

You should be able to pull the second location out of the configuration and have the service forward only to your main data center and in the event of a failure perform a fast cutover.

That also removes the need for having multiple physical devices since you've outsourced that part of the infrastructure.

1
  • Ideally we'd like to avoid this as it adds latency and complexity to the system. It would also be more expensive as all our bandwidth would have to run through the cloud system. We also handle a lot of requests (Current system is running on just under 20 dedicated servers) so the cloud system would need to be fairly scalable/powerful
    – user80776
    Aug 15, 2011 at 13:44

You must log in to answer this question.