5

The situation: setting up a load balancer

Currently we have all of our servers (running CentOS Linux) in pairs in our data center: each server has a mirroring server. We don't employ any load balancing at the moment, so serverA gets all traffic and when it fails (hardware or software) we can quickly switch to serverB by configuring the serverA IP address on serverB. We are using MySQL master/master replication (although we could simply use master/slave replication for the current set-up) and rsync to keep the vhost files in sync (serverA syncs to serverB).

This is working well for us, but it is quite inefficient as we have 50% of your hardware doing nothing until a machine fails. We are thinking about putting up load balancers in front of the server-pairs so we can divide the load to both machines and also add extra servers per cluster.

The problem: sharing file storage

Setting this up will probably not take much more than putting a load balancer in front of each server pair and then have it divide traffic to each server of the pair. Except for one thing: file storage. Currently rsync 'pushes' changes from serverA to serverB, but not the other way around. We can set it up so that rsync also runs from serverB to serverA, but the problem is that rsync never knows whether to create or delete a file that exists on serverA but not on serverB. I looked at Unison, but that project seems to be discontinued.

The question: what's the best solution for software-based shared file storage?

So, I'm looking for a different solution. Please mind that I don't want to add more hardware (so no NAS/SAN solution). Also mind that we need a low amount of storage (below 500GB) per cluster and that all servers are on the same local network. We have a decent back-up solution in place (back-ups run every 3 hours).

I've been looking at DRBD and that seems to fit our situation well, but I have no experience with that. Is DRBD the way to go for us? Please share you experience with this and other similar solutions. Any pitfalls to think about? Am I on the right track? Please enlighten me :)

Improve this question

3 Answers 3

Reset to default
1

DRBD is great.

The good things:

  • It does a magnificent job at replicating data
  • DRBD has in a couple of cases prevented disaster, where it has discovered that the volume was already mounted on the other node, which the raw volumes we get from a SAN are unable to tell us.
  • Heartbeat already has great support for DRBD.

The challenges:

  • Remember to monitor it properly, so you discover split brains when they happen - and can deal with it.
  • DRBD can't be mounted on both servers without a cluster-enabled filesystem on top - I don't have any experience with that part.
  • It is easy to "DOS" the servers by configuring DRBD to use all the available bandwidth for syncing disks. Just configure for lower throughput, and you're OK.

For mounting "the same" filesystem on several nodes, we keep going back to NFS, even though we keep testing various solutions for it. A setup I have no problem with having in production is NFS on top of EXT4 on top of DRBD. I wouldn't dare to do this with the database filesystems, but it's OK for the wwwroot.

Improve this answer
4
  • Thank you for your answer. Do I understand correctly that I make one server the 'NFS host', and that the other server(s) read from that NFS share? So each server has the DRBD block-device, each formatted with the EXT4 filesystem and one server shares a NFS volume, and the other server(s) connect to this volume?
    – user99559
    Nov 1, 2011 at 13:34
  • You only format the DRBD-block device once - as the filesystem will be replicated (on a block level). The NFS-server should be the only one actually mounting this block device - and you can use heartbeat (or similar) to move this server. Then both servers again mount the nfs-share.
    – Kvisle
    Nov 1, 2011 at 13:51
  • Excuse my ignorance, but I'm afraid I don't fully understand this. In a 2-server set-up I would have 2 NFS-servers both mounting the NFS-share? Or just 1 NFS-server and 2 mounting the NFS-share? What if 1 server goes down, I need to set-up the NFS-server on the other (remaining) server?
    – user99559
    Nov 1, 2011 at 15:48
  • Both servers mount the NFS-share at let's say /srv/mirror. The NFS-server exports /srv/failover/mirror, which also is the mount point of the DRBD-volume. Then you set up heartbeat to control the nfs-server, with the IP-address associated with this service, and the DRBD-block-device. In this case - you have automatic failover, and always access to the files on both servers.
    – Kvisle
    Nov 1, 2011 at 16:05
0

DRBD mirrors data in real time, transparently. Please be noted some points as under:

  • You need a cluster file system for the dual-primary mode. OCFS2 is only supported CentOS 5. If you're using CentOS 6, you should use GFS2 instead.
  • Although all the server are on the same local network, I still recommend connecting with a crossover cable.
  • You should have a monitoring mechanism to detect split brain, for e.g: Nagios's check_drbd plugin.
  • Don't forget to measure and optimize DRBD performance

If you want to setup 3 or 4 nodes, take a look at:

Improve this answer
2
  • Hi, thank you for your answer. Concerning your comment "If you have only two backend servers, your data is updated quite often": would this stay valid with more servers? So can we safely add 2 more servers (creating a cluster of 4)?
    – user99559
    Nov 1, 2011 at 13:36
  • I haven't tried with more than 2 nodes.
    – quanta
    Nov 1, 2011 at 14:03
0

Why don`t you set up two (or more) services per cluster and let run one on each side by default?

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.