Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When is the right time to introduce high availability for web site?

There are many articles on High Availability options. It’s not that obvious however WHEN is the right time to switch from single server to high availability configuration.

Please consider my situation:
http://www.postjobfree.com is 24/7 web site with significant traffic:
http://www.quantcast.com/postjobfree.com

Currently I run it on a single server: both IIS 7.0 web server and SQL Server 2008 run on the same hardware box.

There is occasional (~one per month) ~5 minutes downtime usually caused by reboot required by some Windows Server update. Usually downtime is scheduled and happens at night. Still it’s unpleasant, because Google Bot and some users are still active at night.

Current web site revenue is at ~$8K/month.

I consider switching to two-servers configuration (web farm of 2 web servers and cluster of 2 SQL Servers hosted on two hardware servers).

Pros:
1) High Availability (theoretically no downtime). Even if one of servers goes down – another server would take over.
2) No data loss: without SQL cluster, up to one day of data can be lost in case of hardware failure (we do daily backup).

Cons:
1) More effort to setup and maintain such configuration.
2) Higher hosting cost. Instead of ~$600/month it would be about $1200/month.

What would be your recommendation?

share|improve this question

migrated from stackoverflow.com Jun 14 '11 at 6:38

This question came from our site for professional and enthusiast programmers.

    
The answer to my question might affect development. For example, I may consider splitting database in parts and keep data that requires high reliability (user input) separately from data that requires high performance (calculations). –  Dennis Gorelik Jun 14 '11 at 4:26
2  
Hi Dennis, this isn't really a recommendation so I've stuck it as a comment, but your hosting costs seem pretty high for a single windows server? I assume it's a fully dedicated server (not a VM), but even then you should be looking at perhaps half that cost for a decent specification server with 8GB of RAM, a good amount of disk space, etc. It might be worth speaking with your hosting company about getting a better price. –  Ewan Leith Jun 14 '11 at 8:12
6  
I think High Availability should be planned for from the first moment of the project's conception. –  Tom O'Connor Jun 14 '11 at 9:57
    
Ewan, I want my web site to work fast, so I have Quad processor with 8 GB memory and SDD drive. Factor in cost of software licenses (Windows, SQL Server), SSL and tech support. Do you have a good solution with low price for that? I currently use Server Intellect (backed by SoftLayer) for hosting. Would you recommend something better? –  Dennis Gorelik Jun 14 '11 at 13:18
1  
Windows update are coming with security updates. If I don't patch my server, it might be vulnerable to attacks. What update frequency would you recommend for Windows production server? –  Dennis Gorelik Jun 16 '11 at 5:42

7 Answers 7

Short answer: When down time or the risk of it costs you more than it would cost you to have high availability.

It is fundamentally an economic decision. As an example. $8k/month implies that an outage of 2 hours will cost you $22. If you can configure your system such that you can go from scratch to a fully functional site in 2 hours, then high availability would only gain you $22 of functionality above that.

Put another way, you can save money unless / until you have 54 hours of unpreventable down-time in a given month.

share|improve this answer
15  
You have to consider risk to reputation too –  gbn Jun 14 '11 at 4:57
6  
The cost per hour of downtime will almost certainly depend on just when the server goes down. The transactions are very unlikely to be evenly spread over a 24 hour period. It is more normal to occur during just a few peak hours, at which time the loss would be much greater. –  John Gardeniers Jun 14 '11 at 8:44
    
Slartibartfast, I understand your answer that way: make sure that recovery time after catastrophic failure is reasonable (few hours), data loss is reasonable (few hours), and allow myself to have short scheduled downtimes time to time (at least for now). That would mean having daily backups, incremental partial backups, and a server available to restore all that configuration to. Does it sound right? –  Dennis Gorelik Jun 14 '11 at 13:41
    
Responses: gbn: Agreed; I was going for a simple explanation, but reputation could easily be a significant factor. John Gardeniers: Sure, but if the site is only used on Sundays between 11AM and 1PM then scheduled down time isn't really a problem, while the $2k price tag for an unplanned 2 hour outage right_then is. At that point you have to figure out how likely that untimely outage is (at $2k revenue cost) against the certain $600/month charge for the addnl server. Hint: unless random failures during the critical period happen more often than 4/year, It's not worth it. –  Slartibartfast Jun 20 '11 at 3:02
    
Dennis Gorelik: Decide on the risks you want to protect against, (e.g. loss of business during maintenance, loss of server, loss of datacenter, account / security / database breech) and act to protect against them. In this case you're protecting against down time due to maintenance and unpredictable failure (as far as I can tell). What you describe should do the trick, but keep in mind that you don't have to own the server as long as you can be confident that you can procure it and get it set up in the restore period. –  Slartibartfast Jun 20 '11 at 3:04

Your stakeholders/business folk (which could be you!) have to decide

Loss of revenue is easy to quantify: the rest can't be answered here sorry...

share|improve this answer

You've already mentioned Google as a factor in terms of indexing, but it may also be worth considering the impact that latency/site responsiveness may have on SEO. It's a black box and all that, so difficult to quantify - though for what it's worth, Matt Cutts reckons it's a one-percenter. I'd be more concerned about reputation, as others have stated.

share|improve this answer

I think most users can handle a bit of scheduled downtime. Consider that ebay has weekly updates on friday nights, and bids around then sometimes don't work. My (major australian) bank's online banking has scheduled outages for hours every week. Twitter goes offline all the time. Heroku / EC2 was down for days recently.

I'd keep it in that perspective, if you're really only talking 5 mins a month, you're doing quite a good job as a sysadmin.

share|improve this answer

Keep in mind that HA, like security, isn't a product, but rather a process.

For example, database replication will only get you to the point where each mirror of the database will be able to continue on its own, but you will also need a strategy for resynchronization after failed components have been replaced.

Consider an ordering system as an example: the customer submits an order, and during processing, the physical system he was talking to fails after storing the order information in its local copy of the database. Impatient, the customer presses "submit" again, and is directed to another server, which accepts the order. If your databases resynchronize by simply replaying the missing INSERT statements on the other side, then the order will be duplicated, which may not be what you want.

As @Slartibartfast suggested, it all boils down to an economic decision, however I'd recommend that you also plan a few years in the future here. If you expect to need a proper HA setup then, then now would be a good time to set aside resources for the preparatory work.

share|improve this answer

To avoid data loss, you should look into Raid configurations before clusters. You should also configure a Failover IP that you can switch from one server to another in case of a disaster without having to wait for DNS propagation.

share|improve this answer
    
where does this come from? what makes you think the poster isn't using RAID already? –  Chopper3 Jun 14 '11 at 14:13
    
Chopper3. All I said is that Raid would solve his data loss problem. –  yqt Jun 14 '11 at 14:21
1  
How? if one disk died sure but what about if his controller went bad –  Chopper3 Jun 14 '11 at 14:56

Have you considered using something like EC2 that will let you scale flexibly and also negate your cons ? It is ultimately an economic decision if using EC2 is worth it or not, but it is at the least, an option to consider.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.