I'm trying to set up a redundant setup consisting of two servers that have everything redundant:

  • the database (MySQL master-master in active/passive mode)
  • the file system (distributed/replicated)
  • our application software (kept in sync using the distributed file system)

Mostly one of the two servers will be the "main" server and the other will replicate all it's data and will also be used to distribute workload (Gearman). In case the main server fails, everything is switched to the "standby" server which will become the "active" server and continue it's work.

To reduce the risk of complete fail of both servers, they are geographically separated in two distant data centers (same country / direct connections).

I read a lot about distributed file systems, but still have no clue which solution is suitable for just two nodes...

Some more requirements to the distributed file system:

  • must be POSIX compliant
  • must replicate everything (all data must be available on both servers all the time) in both directions (all data can be changed anywhere)
  • current stats relating to the already existing data that should be replicated in future:
    • about 30 GB of data, constantly growing since 3 years
    • about 3 million files in 7,500 directories
    • average file size approx. 5-10 kb; there are a few big files around 10-50 MB
    • files are mostly added periodically through the day and moved to another directory once processed (similar to file based mail server)
    • once a day a few thousand files (received the day before) are archived to a number of TAR archives and left there "forever"
    • when adding files, the data is first written to a temporary file starting with a dot "." and then renamed when complete. Only rarely an existing file is being changed.
  • the system should deal well with unexpected connection losses, reboots of a server, etc.
  • no problem if the replication lags 1-2 seconds, but it should be always in a consistent state
  • as said, the distr. filesys. will consist of only two nodes, but it would be a big bonus if I could add additional nodes/servers, should I need more computing power in the future

Update/more details:

  • I just need redundancy in the sense of "files stored at both servers, synched immediately". When accessing files, I do not need the file system to read data from the other server just because the local hard disks fail. When the local HDD failed, the whole server machine is considered as "broken" and thus shall stop it's work.

Which file system would be suitable in this scenario?

  • You might want to look at this post. stackoverflow.com/questions/269179/… – joeg1ff Jul 9 '14 at 17:11
  • Are there lots of changes to your files or are those 3 million files mostly read only? – Janne Pikkarainen Jul 10 '14 at 6:45
  • mostly read-only. Most of the files are written once (as decribed via a temporary file), read, moved to an archive directory and at midnight moved to a daily compressed TAR archive. The rest is application source code. – Udo G Jul 10 '14 at 12:53
  • More precisely: all files must be readable and writable by both servers, but generally content of existing files is very rarely being modified. – Udo G Jul 10 '14 at 12:55
  • 1
    Seriously, if a question like this is considered off topic then I'm not sure what the purpose of Serverfault is anymore. – Stefan Lasiewski Jul 10 '14 at 18:03

XtreemFS seems to be the thing you want to achieve. You can probably do pretty much the same things with CephFS.


Try DRBD. This is not filesystem, but blockdevice.

From http://lwn.net/Articles/329543/

Protocol A: Writes are considered to complete as soon as the local disk writes have completed, and the data packet has been placed in the send queue for the peers. In case of a node failure, data loss may occur because the data to be written to remote node disk may still be in the send queue. However, the data on the failover node is consistent, but not up-to-date. This is usually used for geographically separated nodes.


Single Primary: The primary designation is given to one cluster member. Since only one cluster member manipulates the data, this mode is useful with conventional filesystems such as ext3 or XFS.

See also http://www.drbd.org/home/feature-list/ for more details.

  • 1
    DRBD alone does not meet the requirements as it will also require a distributed filesystem on top of it so it can handle all the features required. – Florin Asăvoaie Jul 10 '14 at 6:38
  • 1
    If I understand DRBD correctly, only one server can mount the filesystem at a time, but I need both servers to have r/w access. – Udo G Jul 10 '14 at 12:54
  • You can use DRBD in multi-master mode and it will do the replication correctly BUT you also need to run a filesystem with distributed locking mechanism on top of it, otherwise normal filesystems will not work on it. – Florin Asăvoaie Jul 10 '14 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.