Setup for high availability virtualized environment

Question

For a project I have the task of planning a high availability setup for a web shop and CMS system. However, of course the project is on a tight budget. So a high end solution might not be in the budget.

There will be two machines running web server (CMS, shop), one machine running the database, and one machine for running a fax server needed for delivering orders to partners. All systems run Linux. All of these components need to be highly available and should support transparent fail over.

To reduce hardware costs I think about a virtualized environment. There is a lot of information out there, but I do not know exactly were to start. It seems obvious that at least to servers are needed as host for the virtual machines, so that there is no single point of failure.

Which is the best way to go to support high availability?

The first question is which virtualization solution is the best in this situation. There needs to be some kind of management interface. There need to be a way to move a running virtual machine from one host to another, so maintenance of the host can be done. There need to be some kind of mechanism, so that virtual machines are still available if one host fails. Could you advice on a valid solution here?

A shared file storage seems to be prerequisite of high availability in most cases (expect VMware vSphere which is rather expensive). However, would rather put more money in the virtual machine hosts than to add another two servers to the setup to provide a redundant NFS file store. Is there a possibility to get along with only the two virtual machine hosts? A solution might be two use these two as NFS hosts also. Is there much of a performance penalty to do this?

EDIT: I aim at a 99,9% availability. However, no 24/7 availability is required as there are regular business hours, which gives some space to maneuver. The period of availability which has to be in some way guranteed is between 10am to midnight.

Answer 1

As a general overview, to achieve High Availability you need:

1. Multiple servers
2. Multiple consistant copies of the data
3. Consistant data that can be accessed between multiple servers
4. A way of automatically booting a 2nd instance on the standby server

Number 1 is as simple as it sounds - buy two identical servers.

Number 2 can be achieved by a replicating SAN (expensive, very fast, very reliable), or a replicated filesystem on each of the servers (cheap, speed and reliability can depend on your knowledge of the chosen technology).

Number 3 can be achieved by a SAN (one storage LUN, accessed by two servers), or a replicated file system (two seperate storage areas, each server can only see its own).

Number 4 can be achieved by a heartbeat application.

To do this with a small budget, let's say VMWare vSphere, you can use either a SAN or VMWare now offer a self-replicating storage appliance that offers two distinct data stores on two servers that can be used for high availability. vSphere also offers built-in heartbeats and high availability configurations.

To do this with no budget, you could go down the Xen path, and use DRBD to replicate the storage between the two nodes. Then you set up heartbeat to switch the active DRBD storage node and Xen instance to boot up the VM's on the 2nd host when the first goes down.

You won't get 5-nine's (99.999%) uptime using these basic recommendations, but you could easilly get 3-nines (99.9%) by using the cheapest methods if you know what you're doing.

Answer 2

You talk about "expense" in terms of "how much cash will this cost to buy" when discussing shared storage. That's a totally valid point of course, money's tight everywhere.

But if you're talking about High Availability then you you also need to ask "why do we want high availability?" and if the answer is, for example, "because the business turns over $2000 per hour in online sales, so if we're off for an hour then we've lost $2000" then the question of expense and affordability can become "Can we afford not to buy something that enables or greatly improves our high availability deployment?"

This is an important detail and it plays to your comment about budget - the IT 'tail' must not wag the business 'dog' by insisting on an overly complex and expensive solution to a small problem, but at the same time if the business has certain requirements of its IT infrastructure then it has to be prepared to either budget for them properly or to adjust its requirements.

I think virtualisation has a lot of potential in improving the availability of systems, but its not a magic wand. The hardware side of things, while important, is very much secondary to the software requirements - its no good having a SQL database cluster that falls over with no trouble in the event of one of the SQL servers crashing if the front-end application that talks to the database chokes because it can't handle the failover.

And two "highly available" servers sitting next to one another in a datacentre are still vulnerable to power failures, theft, etc. Again, depending on the answer to "why are we doing this?", you might need to consider this aspect quite carefully as it can add expense and complexity to quite a few parts of your project.

Answer 3

Without knowing which DB and application server you use I would recommend:

• Use XEN >3.2 in PV mode for the VMs (just my personal favorite) - compartments or other lightwight virutalization solutions might fit as well (OpenVZ to name one).
• Build four VM machines on each physical node
• Use a local RAID 5 with SAS 3,5" disks - as many disks as locally possible (5 is good)
• Use 15k RPM disks (your DBs will need it)
• Use DRBD and OCFS2 to provide cheap "shared" storage, use a fast, secure, reliable local network for this connection (bonding direct interconnects is pretty fast and good).
• Do the HA on application level
• Use load-balancing between the pairs of machines, so you get 8 machines doing concurrent tasks

HA-Examples:

• Application-Server: Use Tomcat in clustered active/active-mode
• LVS: Use concurrent slave and master replication of lvs
• Oracle-DB: Use RAC (I don`t know if there are equivalent solution for OpenSource DBs)

If you do HA on application layer that layer knows best how to replicate sessions. If one node goes down (planned or unplanned) the surviving node will take over - including sessions.

Answer 4

Why do you want to buy your own hosts? Why don't you find an Enterprise Cloud / IaaS provider like BlueLock or Terremark that will provide the infrastructure you need. They will provide services like vSphere HA (more like reduced downtime than service HA but it's a cost-effective solution), Firewall, LTM / SSL Offloader, SAN (with redundant shelves), Monitoring/ Alerting, etc. Note that we're not talking about consumer cloud solutions here so be prepared to pay for value.

Answer 5

You could look at an all-in-one virtualized/storage replication solution.

The ZFS filesystem makes this possible, as outlined in this blog post.

Another option would be following the tutorial detailing a solution with Red Hat KVM.