2

I need to sync two huge file systems regularly in one direction. Both sides run linux with full root access.

My preferred solution: I can read the list of changed files and directories and sync only the changed files. But how can I get the list of changes? Inotify needs a handler for every directory, but there are too many. Maybe from the journal of the file system?

Here are some solutions and why they don't fit:

  • rsync: Needs to check recursively all files. There are some million files and only little changes. The check takes too long.
  • inotify: I need a handler for every directory and there too many. Inotify was not build for "watch all files" scenarios.
  • DRDB: Both sides should run independent. It can happen that the hosts can't connect for some days.

Both machines need to synced about every 15 minutes. The initial sync is no problem, this question is only about syncing the changes.

7
  • 1
    Sync them once, regularly, or continuously?
    – nickgrim
    Jun 21, 2012 at 9:07
  • Do you know previously the list of changed files? In other words, do you keep track of all changed files somehow? Otherwise, you need to compare the whole directory tree to know what has changed since last sync.
    – Khaled
    Jun 21, 2012 at 9:21
  • How far out of sync can the filesystems be, i.e. do they need to be within 1 minute, 10 minutes, 1 hour, 1 day of each other? Jun 21, 2012 at 9:23
  • I updated the question: I only care about syncing the changes about every 15 minutes. Initial sync is not problem.
    – guettli
    Jun 21, 2012 at 10:12
  • 1
    You say "rsync... takes too long" - how long does it take, and how long would be acceptable? You say DRBD is unacceptable since both sides should be independent - could you expand on why this is? It seems like this question is moving away from "sync two filesystems" and towards "how do I quickly find all changes that happened within the last some-period-of-time"
    – nickgrim
    Jun 21, 2012 at 11:25

2 Answers 2

4

How about GlusterFS? I have found that the traffic it develops is considerable less than DRBD.

4
  • 5
    Sadly enough, websites go stale, even personal ones. As such, it may be a VERY good idea to summarise the key points of the how to so people in future could work out what is going on, especially if the link goes down. Jun 21, 2012 at 9:31
  • while I do agree with @JourneymanGeek, I still would say +1 to Gluster
    – dyasny
    Jun 21, 2012 at 9:56
  • 1
    Welcome to Server Fault! We really do prefer that answers have content, not pointers to content. This ensures that the answer will remain available even if the link goes dead. Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.
    – user9517
    Jun 21, 2012 at 10:43
  • And there it happened. New URL is apparently chriscowley.me.uk/blog/2012/02/08/… May 10, 2013 at 16:18
0

+1 for GlusterFS, it's a userspace Storage Clustering platform, it's pretty easy to setup, and works at file level.

Once installed all you need to do is create a new replicated Gluster Volume on the main server, it will replicate the contents realtime over TCP/IP. Then setup the client (which is basically mounting a filesystem), you can even do that on one of the servers and then write data always to that new mounted filesystem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.