Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am building a web application where uptime is key. I understand that 100% uptime is not realistic but I would like to achieve five nines. I'm unsure as to the most prudent way to accomplish this.

My preliminary plan was to have the web app running in two geographically separate data centres. The "main" data centre would contain master server and this would replicate to the unused "slave" server elsewhere. If downtime occurred at the main data centre, DNS failover would move traffic to the "slave" server. There are challenges with this technique, including some users being unable to access the site for a while due to odd DNS record caching, etc.

However, I have read a lot of opinion stating that DNS failover is not a great solution and that you should keep everything in a single data centre and focus on redundancy there. The issue I see with that is that even the good data centres seem to have the odd network issue that can cause enough downtime to blow apart the expectation of five nines.

Should I go with the DNS failover option? Are there better options?

share|improve this question
    
That's 5 minutes of downtime per year. DNS records are commonly cached for a week, and sometimes inconsiderate DNS cache operators keep them for a month or more. You can see the problem. –  Chris S Jan 11 '13 at 19:52
3  
If you had a valid business case for five nines, you probably wouldn't be asking it here. Your boss might say "we need five nines" but when he/she understands the cost of five nines, that requirement will likely be loosened. –  northben Jan 11 '13 at 21:00

3 Answers 3

My rule-of-thumb for clients is: two nines you get for free. Every extra nine increases cost by up to an order of magnitude.

That is to say, you can have 99% uptime by just putting your application on a half-decent server on your corporate internet connection. To improve on that, you can colocate. You can colocate with load-balancing and fast failover. You can colocate with load-balancing, fast failover, and a cold spare DR site. You can colocate with load-balancing, a hot spare site, PI address space, run your own ASN and have BGP peering arrangements in place to ensure that your address space is always globally-routable. You can investigate high-availability hardware, where everything including memory and CPUs can be quiesced and hot-swapped. If your application supports it, you can run fully-distributed hosting, or outsource to the highly-available content provision networks. You can, and will, need five times as many staff to manage all this 24*365, including holidays and sickness cover, and the frequent live DR tests you will need to do to have confidence in all of this.

You can do lots of clever stuff. But it all costs, and most of it costs a very large amount of money.

So my sincere advice is: work out what it'd cost you to host your app on a single server in the corporate office. If your employer isn't willing to spend up to a thousand times as much as that, forget five-nines; it's not realistic.

share|improve this answer

For five nines, you're looking at a lot more involvement than just one failover solution. You need HA within one datacenter plus a hot (or at least warm) standby datacenter that's geographically far but topologically near your primary data center. And that's just the start...

share|improve this answer

If five nines was easy, Twitter, Facebook, Gmail, Azure and Amazon would probably already be there. They definitely have the money and most valid business cases for it. Instead, I would recommend you aim for hosting with a cloud provider who has the expertise in providing reliable infrastructure so that they can worry about this while you develop your product.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.