Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The situation: setting up a load balancer

Currently we have all of our servers (running CentOS Linux) in pairs in our data center: each server has a mirroring server. We don't employ any load balancing at the moment, so serverA gets all traffic and when it fails (hardware or software) we can quickly switch to serverB by configuring the serverA IP address on serverB. We are using MySQL master/master replication (although we could simply use master/slave replication for the current set-up) and rsync to keep the vhost files in sync (serverA syncs to serverB).

This is working well for us, but it is quite inefficient as we have 50% of your hardware doing nothing until a machine fails. We are thinking about putting up load balancers in front of the server-pairs so we can divide the load to both machines and also add extra servers per cluster.

The problem: sharing file storage

Setting this up will probably not take much more than putting a load balancer in front of each server pair and then have it divide traffic to each server of the pair. Except for one thing: file storage. Currently rsync 'pushes' changes from serverA to serverB, but not the other way around. We can set it up so that rsync also runs from serverB to serverA, but the problem is that rsync never knows whether to create or delete a file that exists on serverA but not on serverB. I looked at Unison, but that project seems to be discontinued.

The question: what's the best solution for software-based shared file storage?

So, I'm looking for a different solution. Please mind that I don't want to add more hardware (so no NAS/SAN solution). Also mind that we need a low amount of storage (below 500GB) per cluster and that all servers are on the same local network. We have a decent back-up solution in place (back-ups run every 3 hours).

I've been looking at DRBD and that seems to fit our situation well, but I have no experience with that. Is DRBD the way to go for us? Please share you experience with this and other similar solutions. Any pitfalls to think about? Am I on the right track? Please enlighten me :)

share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

DRBD is great.

The good things:

  • It does a magnificent job at replicating data
  • DRBD has in a couple of cases prevented disaster, where it has discovered that the volume was already mounted on the other node, which the raw volumes we get from a SAN are unable to tell us.
  • Heartbeat already has great support for DRBD.

The challenges:

  • Remember to monitor it properly, so you discover split brains when they happen - and can deal with it.
  • DRBD can't be mounted on both servers without a cluster-enabled filesystem on top - I don't have any experience with that part.
  • It is easy to "DOS" the servers by configuring DRBD to use all the available bandwidth for syncing disks. Just configure for lower throughput, and you're OK.

For mounting "the same" filesystem on several nodes, we keep going back to NFS, even though we keep testing various solutions for it. A setup I have no problem with having in production is NFS on top of EXT4 on top of DRBD. I wouldn't dare to do this with the database filesystems, but it's OK for the wwwroot.

share|improve this answer
    
Thank you for your answer. Do I understand correctly that I make one server the 'NFS host', and that the other server(s) read from that NFS share? So each server has the DRBD block-device, each formatted with the EXT4 filesystem and one server shares a NFS volume, and the other server(s) connect to this volume? –  vvanscherpenseel Nov 1 '11 at 13:34
    
You only format the DRBD-block device once - as the filesystem will be replicated (on a block level). The NFS-server should be the only one actually mounting this block device - and you can use heartbeat (or similar) to move this server. Then both servers again mount the nfs-share. –  Kvisle Nov 1 '11 at 13:51
    
Excuse my ignorance, but I'm afraid I don't fully understand this. In a 2-server set-up I would have 2 NFS-servers both mounting the NFS-share? Or just 1 NFS-server and 2 mounting the NFS-share? What if 1 server goes down, I need to set-up the NFS-server on the other (remaining) server? –  vvanscherpenseel Nov 1 '11 at 15:48
    
Both servers mount the NFS-share at let's say /srv/mirror. The NFS-server exports /srv/failover/mirror, which also is the mount point of the DRBD-volume. Then you set up heartbeat to control the nfs-server, with the IP-address associated with this service, and the DRBD-block-device. In this case - you have automatic failover, and always access to the files on both servers. –  Kvisle Nov 1 '11 at 16:05
    
Ah yes, I understand. Great, thank you! –  vvanscherpenseel Nov 1 '11 at 16:10
add comment

DRBD mirrors data in real time, transparently. Please be noted some points as under:

  • You need a cluster file system for the dual-primary mode. OCFS2 is only supported CentOS 5. If you're using CentOS 6, you should use GFS2 instead.
  • Although all the server are on the same local network, I still recommend connecting with a crossover cable.
  • You should have a monitoring mechanism to detect split brain, for e.g: Nagios's check_drbd plugin.
  • Don't forget to measure and optimize DRBD performance

If you want to setup 3 or 4 nodes, take a look at:

share|improve this answer
    
Hi, thank you for your answer. Concerning your comment "If you have only two backend servers, your data is updated quite often": would this stay valid with more servers? So can we safely add 2 more servers (creating a cluster of 4)? –  vvanscherpenseel Nov 1 '11 at 13:36
    
I haven't tried with more than 2 nodes. –  quanta Nov 1 '11 at 14:03
add comment

Why don`t you set up two (or more) services per cluster and let run one on each side by default?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.