I need to sync two huge file systems regularly in one direction. Both sides run linux with full root access.

My preferred solution: I can read the list of changed files and directories and sync only the changed files. But how can I get the list of changes? Inotify needs a handler for every directory, but there are too many. Maybe from the journal of the file system?

Here are some solutions and why they don't fit:

  • rsync: Needs to check recursively all files. There are some million files and only little changes. The check takes too long.
  • inotify: I need a handler for every directory and there too many. Inotify was not build for "watch all files" scenarios.
  • DRDB: Both sides should run independent. It can happen that the hosts can't connect for some days.

Both machines need to synced about every 15 minutes. The initial sync is no problem, this question is only about syncing the changes.

  • 1
    Sync them once, regularly, or continuously? – nickgrim Jun 21 '12 at 9:07
  • Do you know previously the list of changed files? In other words, do you keep track of all changed files somehow? Otherwise, you need to compare the whole directory tree to know what has changed since last sync. – Khaled Jun 21 '12 at 9:21
  • How far out of sync can the filesystems be, i.e. do they need to be within 1 minute, 10 minutes, 1 hour, 1 day of each other? – EightBitTony Jun 21 '12 at 9:23
  • I updated the question: I only care about syncing the changes about every 15 minutes. Initial sync is not problem. – guettli Jun 21 '12 at 10:12
  • 1
    You say "rsync... takes too long" - how long does it take, and how long would be acceptable? You say DRBD is unacceptable since both sides should be independent - could you expand on why this is? It seems like this question is moving away from "sync two filesystems" and towards "how do I quickly find all changes that happened within the last some-period-of-time" – nickgrim Jun 21 '12 at 11:25

How about GlusterFS? I have found that the traffic it develops is considerable less than DRBD.

| improve this answer | |
  • 5
    Sadly enough, websites go stale, even personal ones. As such, it may be a VERY good idea to summarise the key points of the how to so people in future could work out what is going on, especially if the link goes down. – Journeyman Geek Jun 21 '12 at 9:31
  • while I do agree with @JourneymanGeek, I still would say +1 to Gluster – dyasny Jun 21 '12 at 9:56
  • 1
    Welcome to Server Fault! We really do prefer that answers have content, not pointers to content. This ensures that the answer will remain available even if the link goes dead. Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – user9517 Jun 21 '12 at 10:43
  • And there it happened. New URL is apparently chriscowley.me.uk/blog/2012/02/08/… – Henrik Mühe May 10 '13 at 16:18

+1 for GlusterFS, it's a userspace Storage Clustering platform, it's pretty easy to setup, and works at file level.

Once installed all you need to do is create a new replicated Gluster Volume on the main server, it will replicate the contents realtime over TCP/IP. Then setup the client (which is basically mounting a filesystem), you can even do that on one of the servers and then write data always to that new mounted filesystem.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.