Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

For a project I have the task of planning a high availability setup for a web shop and CMS system. However, of course the project is on a tight budget. So a high end solution might not be in the budget.

There will be two machines running web server (CMS, shop), one machine running the database, and one machine for running a fax server needed for delivering orders to partners. All systems run Linux. All of these components need to be highly available and should support transparent fail over.

To reduce hardware costs I think about a virtualized environment. There is a lot of information out there, but I do not know exactly were to start. It seems obvious that at least to servers are needed as host for the virtual machines, so that there is no single point of failure.

Which is the best way to go to support high availability?

The first question is which virtualization solution is the best in this situation. There needs to be some kind of management interface. There need to be a way to move a running virtual machine from one host to another, so maintenance of the host can be done. There need to be some kind of mechanism, so that virtual machines are still available if one host fails. Could you advice on a valid solution here?

A shared file storage seems to be prerequisite of high availability in most cases (expect VMware vSphere which is rather expensive). However, would rather put more money in the virtual machine hosts than to add another two servers to the setup to provide a redundant NFS file store. Is there a possibility to get along with only the two virtual machine hosts? A solution might be two use these two as NFS hosts also. Is there much of a performance penalty to do this?

EDIT: I aim at a 99,9% availability. However, no 24/7 availability is required as there are regular business hours, which gives some space to maneuver. The period of availability which has to be in some way guranteed is between 10am to midnight.

share|improve this question
2  
How 'high' is 'high availability'? Are you shooting for 1-nine or 6-nine availability, or somewhere in between? Until you have concrete requirements in place, it's impossible to say whether or not what you want to do is achievable on a given budget. –  growse Mar 4 '12 at 21:09
    
Yes you are right. I aim for a 99,9% availability. –  spa Mar 4 '12 at 22:19
    
"99.9%" isn't just a phrase we throw about. It equates to about 8.8 hours downtime a year. That takes you out of the range of systems that are just thrown together on a tight budget. If your budget is limited, can you afford to support that level of availability? –  RobM Mar 5 '12 at 8:39
1  
@RobMoir - I'd argue that if you meet the criteria that I outlined in my answer, there's not many problems you couldn't fix in those 8 hours (and the budget could still be smallish). If you make sure that advanced-warning, out-of-hours, scheduled downtime doesn't count towards your SLA (for non-24/7 software). –  Mark Henderson Mar 6 '12 at 4:34
    
@MarkHenderson I know you're right, I'm just saying that the process requires some thought and planning and won't "just happen" (you need to make sure you can get spare parts on site well within that 8 hours, for example, so you don't want to lose 7 hours of 'window' to the post office, or find your favourite supplier chose that day to be out of stock on some trivial cable that they'd normally have in stock by the thousand). –  RobM Mar 7 '12 at 9:14

5 Answers 5

up vote 12 down vote accepted

As a general overview, to achieve High Availability you need:

  1. Multiple servers
  2. Multiple consistant copies of the data
  3. Consistant data that can be accessed between multiple servers
  4. A way of automatically booting a 2nd instance on the standby server

Number 1 is as simple as it sounds - buy two identical servers.

Number 2 can be achieved by a replicating SAN (expensive, very fast, very reliable), or a replicated filesystem on each of the servers (cheap, speed and reliability can depend on your knowledge of the chosen technology).

Number 3 can be achieved by a SAN (one storage LUN, accessed by two servers), or a replicated file system (two seperate storage areas, each server can only see its own).

Number 4 can be achieved by a heartbeat application.

To do this with a small budget, let's say VMWare vSphere, you can use either a SAN or VMWare now offer a self-replicating storage appliance that offers two distinct data stores on two servers that can be used for high availability. vSphere also offers built-in heartbeats and high availability configurations.

To do this with no budget, you could go down the Xen path, and use DRDB to replicate the storage between the two nodes. Then you set up heartbeat to switch the active DRDB storage node and Xen instance to boot up the VM's on the 2nd host when the first goes down.

You won't get 5-nine's (99.999%) uptime using these basic recommendations, but you could easilly get 3-nines (99.9%) by using the cheapest methods if you know what you're doing.

share|improve this answer

You talk about "expense" in terms of "how much cash will this cost to buy" when discussing shared storage. That's a totally valid point of course, money's tight everywhere.

But if you're talking about High Availability then you you also need to ask "why do we want high availability?" and if the answer is, for example, "because the business turns over $2000 per hour in online sales, so if we're off for an hour then we've lost $2000" then the question of expense and affordability can become "Can we afford not to buy something that enables or greatly improves our high availability deployment?"

This is an important detail and it plays to your comment about budget - the IT 'tail' must not wag the business 'dog' by insisting on an overly complex and expensive solution to a small problem, but at the same time if the business has certain requirements of its IT infrastructure then it has to be prepared to either budget for them properly or to adjust its requirements.

I think virtualisation has a lot of potential in improving the availability of systems, but its not a magic wand. The hardware side of things, while important, is very much secondary to the software requirements - its no good having a SQL database cluster that falls over with no trouble in the event of one of the SQL servers crashing if the front-end application that talks to the database chokes because it can't handle the failover.

And two "highly available" servers sitting next to one another in a datacentre are still vulnerable to power failures, theft, etc. Again, depending on the answer to "why are we doing this?", you might need to consider this aspect quite carefully as it can add expense and complexity to quite a few parts of your project.

share|improve this answer
3  
...no good having a SQL database cluster that falls over with no trouble in the event of one of the SQL servers crashing if the front-end application that talks to the database chokes because it can't handle the failover. - I could not emphasise this enough. We had a client who had us implement a HA SQL Server cluster on a big SAN and at the end of the day their software had to be restarted in the case of a failover because it couldn't handle a break in communications. It was an expensive exercise that was futile when an SQL Mirror and NLB would have sufficed. –  Mark Henderson Mar 4 '12 at 21:50
    
Sounds like we've both got similar scars from old projects –  RobM Mar 4 '12 at 21:55
    
@MarkHenderson why did the communication break (btw which one - SAN or network)? –  Nils Mar 5 '12 at 21:07

Without knowing which DB and application server you use I would recommend:

  • Use XEN >3.2 in PV mode for the VMs (just my personal favorite) - compartments or other lightwight virutalization solutions might fit as well (OpenVZ to name one).
  • Build four VM machines on each physical node
  • Use a local RAID 5 with SAS 3,5" disks - as many disks as locally possible (5 is good)
  • Use 15k RPM disks (your DBs will need it)
  • Use DRBD and OCFS2 to provide cheap "shared" storage, use a fast, secure, reliable local network for this connection (bonding direct interconnects is pretty fast and good).
  • Do the HA on application level
  • Use load-balancing between the pairs of machines, so you get 8 machines doing concurrent tasks

HA-Examples:

  • Application-Server: Use Tomcat in clustered active/active-mode
  • LVS: Use concurrent slave and master replication of lvs
  • Oracle-DB: Use RAC (I don`t know if there are equivalent solution for OpenSource DBs)

If you do HA on application layer that layer knows best how to replicate sessions. If one node goes down (planned or unplanned) the surviving node will take over - including sessions.

share|improve this answer
    
"Oracle-DB: Use RAC" - Standard Edition is not licensed or supported with OCFS2. Other than that, a very informative answer. –  kubanczyk Oct 27 '13 at 12:02
    
@kubanczyk Oracle-RAC is more than ocfs2. But ocfs2 is free. So you can use it whenever you want. –  Nils Oct 29 '13 at 19:36

Why do you want to buy your own hosts? Why don't you find an Enterprise Cloud / IaaS provider like BlueLock or Terremark that will provide the infrastructure you need. They will provide services like vSphere HA (more like reduced downtime than service HA but it's a cost-effective solution), Firewall, LTM / SSL Offloader, SAN (with redundant shelves), Monitoring/ Alerting, etc. Note that we're not talking about consumer cloud solutions here so be prepared to pay for value.

share|improve this answer
    
Yes you are right. However the setup includes like custom hardware for the fax delivery. So a cloud solution won't do sadly. –  spa Mar 4 '12 at 22:31
    
@spa, you could still provision the custom hardware on their physical environment, the rest on virtual and bridge the VLANs. –  HTTP500 Mar 4 '12 at 22:51

You could look at an all-in-one virtualized/storage replication solution.

The ZFS filesystem makes this possible, as outlined in this blog post.

Another option would be following the tutorial detailing a solution with Red Hat KVM.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.