15

I'm looking for a solution to mirror or replicate one directory (or one filesystem) across a few Linux servers. Ideal solution would be one, which permits all servers read-write access. I also want it to me resilient, if one of the servers goes down, rest should still work w/o loosing any data at all.

I've been looking at some solutions:

  • DRBD: block level replications, seems a bit overkill;
  • lsyncd: looks very simple, but I have doubts about performance;
  • GlusterFS: seems like that would be a good match, haven't figured out yet how exactly the replication mode works. Will it have characteristics I require?

Any other suggestions are welcome.

Improve this question
2
  • Are you looking at a shared-nothing setup or will you connect all servers to the same backend SAN?
    – HampusLi
    May 6, 2011 at 12:06
  • logically it shared-nothing (physically it's EC2 with EBS).
    – vartec
    May 6, 2011 at 12:29

5 Answers 5

Reset to default
6

The first question I would ask is do you want this replicated to two servers or more then two servers? For two servers I would go with DRDB, for three or more I would go with gluster.

If I/O latency is not a critical concern I would go with gluster. It is pretty easy to setup and could clearly do what you needed. All you need to do is make a gluster server serving the files up on all three boxes and then also make each box act as a gluster client mounting the files.

DRDB is going to to be complicated to get working in a master<->master mode with 3 or more servers. You have to configure a ring based setup and I would not recommend it. However for two servers DRDB is fantastic. Master<->Master mode is not complicated to setup and you don't have to learn any of the file system stuff.

lsycd is great for a master / slave setup, but you don't appear to want to that.

Ceph is still pretty new, last time I checked it did not even have fsck support yet. I would rather base my infrastructure on something more stable.

Lustre is a fantastic product for large scale deployments, but you need to setup heartbeating and failover for the mds server or its going to have a single point of failure. Given the limited number of servers he is talking about I suspect its overkill in this case.

Improve this answer
1
  • Initially I'll start with just 2 servers per cluster, but I want have option to have more than 2 servers for scale-out; The gluster setup you're describing, would it handle one of the servers crashing? would it be easy to add extra server?
    – vartec
    May 6, 2011 at 12:36
2

How about Ceph or Lustre?

Improve this answer
2
  • Does Lustre support Ubuntu Server? I'm checking out Ceph now, thanks for suggestion.
    – vartec
    May 6, 2011 at 10:06
  • From the ceph wiki: "Ceph is under heavy development, and is not yet suitable for any uses other than benchmarking and review. " ceph.newdream.net/wiki
    – Jeff Busby
    May 6, 2011 at 17:10
2

You should look into OpenAFS -- it's a mostly distributed filesystem that allows for multiple copies of data to exist distributed across the cluster and everyone can read / write to the FS at the same time.

It's got a bunch of other useful features as well (good authentication, encryption on the wire, built-in local caching on clients, native windows client, portable across lots of versions of unix, etc)

It's a bit of a heavy lift to setup, though.

Improve this answer
3
  • same question as with NFS: "allows multiple copies". OK, but does it take care of actually synchronizing these copies?
    – vartec
    May 9, 2011 at 8:13
  • Yep. And it is possible, though annoying, to recover from losing the primary master. But it is trivial to have multiple writers and the resulting blocks stored on multiple hosts.
    – chris
    May 9, 2011 at 12:12
  • The key to understanding OpenAFS is that it is a cache management system -- there is one namespace (IE only one "file" exists) but there are cached copies of the file everywhere and a protocol to make sure that all the cached copies are consistent. If you lose the master you can turn one of the cached copies into the master, but it isn't ideal to be in that situation.
    – chris
    May 9, 2011 at 12:23
1

Getting this to work with DRBD is going to be really difficult - the problem is not as n8whnp seems to think an issue regarding multi-way replication (you just make all the nodes stripes in a mirror set) but is of concurrency control - you'd need to run a cluster filesystem on top of the mirroring on top of DRBD.

lsyncd is even worse as there's no practical solution for concurrency control.

I'd recommend an AFS type solution (AFS, OpenAFS) as a mature, stable, open solution. I'd stay clear of lustre since Oracle shut it down. Not overly familiar with glusterfs but as it relies on distributed rather than replicated storage I'd recommend you have a long hard look at how it will behave in split-brain operation (AFS OTOH is designed to work in disconnected mode).

Improve this answer
0

NFS might also work fine too, depending on your needs.

Improve this answer
3
  • AFAIK, NFS doesn't provide way to mount replicated servers, but not the replication itself. But I haven't been using NFS for long time, so maybe that has changed. Could you link to NFS docs describing such setup?
    – vartec
    May 6, 2011 at 12:02
  • A way off the top of my head would be to replicate the dirs to several nfs servers, one of which has a primary vip, and migrate the vip across servers if one goes down. Or round robin dns maybe. NFS itself may not do everything you need, but in conjunction with heartbeat or red hat cluster services, it may be what you need. I'm not sure the original question had all the requirements in it. You could even do rsync across a bunch of servers, say every hour or so, for a really quick and easy solution.
    – lsd
    May 6, 2011 at 19:15
  • Skip the part about rsync. It would do the replication, but it is primarily for read-only setups, which doesn't fit your requirements. I meant to edit my comment above, but it wouldnt let me.
    – lsd
    May 6, 2011 at 19:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .