1

We have the following servers:

A. 1 x web server running MySQL (a forum).
B. 1 x web server running RT with Postgres.
C. 1 x web server running a home grown app with MonetDB as the backend

Now of course we want to add a bit of reliability here by adding an extra server to each web server in case 1 server dies, etc. But what is the best and relatively cheaper way to do it?

  1. Get an extra server each, and replicate the DBs using the appropriate technologies in each case? MySQL replication for A, Slony, etc for B, and I don't really know how to replicate MonetDB, but I guess it's possible.

Why I don't like this: possible data corruption due to synchronisation issues, i.e. a temporary power blip could cause data to be written to the slave, then the master comes back, and then the replication gets broken. In the case of Slony, you can't even do that, you have to promote the slave to a master first, etc, AFAIK.

The other thing I don't like about this is having to manually trudge through all this and figure out who is doing what now and syncing things across, etc.

2.Get 2 servers with a decent amount of storage and setup DRBD across them.

Since they will have a single shared IP, put all the DBs, e.g. MySQL, Postgres, and Monet on the DRBD storage. The good thing about DRBD is that we won't have a single point of failure since even if we lose a part of the cluster, the other server can take over so it's much more resilient. And my understanding it that the web servers above this level can simply fail over and back without any need for worrying about replication, and syncing things, etc.

3.VMs?
What is best practice when using VMs to setup something like this?

Improve this question

4 Answers 4

Reset to default
3

VMs won't help you that much here as far as high availability is concerned, although they can be useful in simplifying the consolidation.

Your question cannot be answered without two important data points: your budget, and the load. If you have a limited budget but your load is low enough, you can easily move all this onto two servers in an active/passive setup.

If your budget allows it, you might want to consider acquiring a SAN (or preferably two for HA). In that case, you can still have the DBs, possibly virtualized, on the same server as the apps, but with the data written to the SAN. Databases are mostly IO-bound.

Another route is indeed to have a two node fail-over cluster for the DBs, and have the DB partitions replicated through DRBD, it works quite well. However you want to make sure that each have their own disk; for that I would recommend using something like a 2U machine with 6 disks with battery-backed hardware raid.

If you have even more traffic but still wants to keep a low budget, you can try the following:

  • 2x DB server in active/passive setup replicated with DRBD
  • 2x front-end servers running the apps, both active
  • Run a software load-balancer on a VIP on the front-end, only one of them will receive incoming connections, but it will redispatch them either locally or to the other node. I recommend Haproxy

Beware of the load though; if one of the nodes fails when you're at peak usage you're in for a lot of trouble, but it might be a good way to handle the occasional slashdotting.

Improve this answer
4
  • Thanks for the answer, I have a few questions: If we virtualise the DB servers on the apps servers themselves, we need to setup some sort of heartbeat between them? In the DRBD paragraph above, what did you mean by "you want to make sure that each have their own disk"? Instead of using HAProxy, we could simply just use a normal loadbalancer in front, couldn't we? We don't have an overly restrictive budget, but we still need to keep it reasonable, not something top of the line, and very expensive, etc. Our load at the moment is quite low.
    – afa
    Jun 2, 2009 at 11:43
  • Whether you virtualize or not, if you want an active/passive server, you need some sort of heartbeat obviously. Personally I use RedHat Cluster. By "own disk" I mean real disk, you don't want to have two DBs competing for access. You can indeed use a "normal" loadbalancer that will do the exact same thing as haproxy but cost more money. Here you can run haproxy on one of the two machines, and have it target itself and the other node. I was just pointing out the cheapest solution, one that is also likely to be just as good.
    – niXar
    Jun 2, 2009 at 13:02
  • any reason why the 2U machine needs battery backed hardware raid? If we already have 2 servers setup with DRBD in an active/passive state, then why do we need it?
    – afa
    Jun 2, 2009 at 17:17
  • All mid-end hardware raid cards (such as those found on systems with hot-swap disks) have battery backed cache nowadays; it's a must for database workloads, as it makes a fsync instantaneous instead of having to wait for the disk to seek, which takes at least a few ms.
    – niXar
    Jun 2, 2009 at 21:55
2

Don't use drbd for mysql failover. Use master-master replication and put haproxy in front of them to load balance the mysql connection and provide failover. Then put heartbeat on the haproxy servers. Complete failover with load balancing.

drbd is good for things like web files with nfs running on top of it.

Improve this answer
1

For a solution that is there "in case" your primaries fail and not for balancing / frequent fail-over, a combination of #2 & #3 would work well. Get a box with reasonable resource to store and run VM's for each of primary systems. Use DRBD to simplify the replication of the data. For added bells and whistles you could throw in heartbeat for automated failover- or not. And of course replication != backup.

Improve this answer
0

Why not use VMWare/virtualisation to create a two-server+ DB cluster, each running all of your various DBs and a second two-server+ web server cluster, each serving all of your various web apps.

That way you need as few servers as possible, have a built-in scaling solution that will allow you to grow/move your hot-spots as they occur and will also make best use of your assets.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .