4

I have an Idea for fail-over high availability for my website but i am not sure if this is good, bad or a disaster in itself.

My main server hosts an ASP.net website that uses a SQL server database on another server.

Both servers are running mirrored raid drives, two network cards, 2 switches etc. The provider is guaranteeing 99.999% up time but something did go wrong and they took almost a a day before it resolved.

I am more concerned about issues like domain name / dns issues that are out of our direct control and may take 6-24hours to propagate.

Or for that matter wide spread disasters that might take out our main data center, power lines, network connectivity infrastructure, domain hijacking, and the rise of man eating undead ;) etc.

So My Idea is as follow: Host a second domain at another provider in another country. Call the domain something similar to the main site name.

Have a server for the site and a server for the SQL db hosted at this secondary provider. The webserver is setup and configured with the website exactly as the main site.

My main SQL server mirrors (using High Performance mirroring) to the secondary server at the secondary provider every 5 minutes.

Assume for some reason the main site is unreachable due to something big and nasty going on.

Change the DNS to point to the backup domain and get the word out on twitter, facebook etc that anybody who needs my site can use www.backupdomain.com until the dns updates propagate across the web.

Would this work and is there a better option that would handle issues like this?

Most of the research I have done pointed to fail over clusters, load balances,duplicate hardware, mirroring and the like, which I do realize will make the local hosting redundant, but how do i handle wider spread interruptions.

The budget is also limited so we can't spend millions on a super duper Google never die system. But something that can handle a really bad disruption and only take 30mins to 1 hour downtime would be perfect.

Tip, suggestions, links are all welcome.

Improve this question
3
  • Find a reliable DNS provider that allows low TTL, set the TTL for the web site to 300sec or so, and you won't need to worry about propagation delays. All properly configured DNS servers will follow TTL, and most (if not all) web browsers expire their DNS cache after 15 minutes. So the worst case delay will be 20 minutes, well within your downtime window.
    – Max Alginin
    Jun 24, 2011 at 16:32
  • @ynguldyn, I was under the impression that even with a low ttl, the time a dns entry takes to propagate globally through the dns servers was up to 24 hrs. Am i wrong about that?
    – SetiSeeker
    Jun 25, 2011 at 8:47
  • Yes. You're thinking about registrars and their records. Those do take time, because their databases may not be updated immediately, and then the TTLs of those records are usually set very high. But when you have a DNS server that you can reload immediately and a low TTL, those issues don't exist. Moreover, DNS is the foundation of all load balancing solutions that provide multi-site capabilities (google "GSLB").
    – Max Alginin
    Jun 27, 2011 at 15:07

2 Answers 2

Reset to default
2

The options you describe are not bad ones -- in fact they're good ones, and the fact that you're considering this speaks well of you.

You can certainly implement what you described above, or use a cloud provider as a (much less expensive) backup site like ksm suggests below, but I'd address some more fundamental items first.

Here's the rough order I would work in:

  1. Make sure your hosting provider is decent
    Redundant Power, Pipes & Cooling at a minimum.

  2. Make sure your environment is well-designed.

  3. Make sure your environment has redundancy (local mirrors of everything critical, HA/Failover).
    If your provider is good, your design is good, and everything is redundant to handle at least one component failure you've taken care of the biggest portion of your outages. You've also probably given yourself the ability to perform concurrent maintenance if your design from #2 was good.

  4. Make sure you have backups. Make sure you can restore them and get back a working system.

  5. Test the hell out of numbers 3 & 4 (Think like the Chaos Monkey & simulate failures)

  6. With 1-4 done and solid, now consider how you would replicate this to a remote location in case a meteor hits your provider's building.
    If 2-4 above were done well this part should have obvious, relatively easy implementation paths.

  7. Test the hell out of failing over/back with what you implement in #6.
    A VMWare lab is VERY helpful here.

Note that I didn't get down into specifics -- Your environment will dictate how you proceed with each of the steps above.

Improve this answer
3
  • Thanks for the great response. Would you be able to suggest some good places to read up on specific implementation details about robust HA systems?
    – SetiSeeker
    Jun 25, 2011 at 8:48
  • A good start would be right here on Server Fault -- barring that the best advice I have is to understand the principles of redundancy and robustness (any good engineering textbook helps here), and think critically about applying them to IT & information systems. I can give you lots of advice on specific technologies, but in 3-5 years the field will have changed: People who know the principles will still be able to find work, people who know specific technologies will be out of a job once the transition is done.
    – voretaq7
    Jun 27, 2011 at 14:29
  • 1
    It is possible to have an AS and doing BGP with 2 ISPs for network redundancy. You can have also do BGP based load balancing. See: stack.nil.com/ipcorner/LoadBalancingBGP
    – Mircea Vutcovici
    Nov 7, 2012 at 16:11
1

Why don’t you just get an instance on AWS? Get an instance on the E2C, host your app there, and let them worry about the uptime.

To be doubly sure, you could have two instances (the second as a hot backup, perhaps) in different regions: one on their American DC and the other on their Asian DC.

Improve this answer
1
  • 2
    Recent events have demonstrated that Amazon (AWS/EC2) instances are subject to the same kind of failures as any other (poorly designed, non-redundant) system -- all of @SetiSeeker's points would still need to be addressed. See also Jeff's article - codinghorror.com/blog/2011/04/…
    – voretaq7
    Jun 24, 2011 at 14:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .