1

What is the best way to sync large sets of files between to geographically separate locations so that both locations have the same file?

We have a constantly changing 500gig file share in Location1 with an avg rate of change of (+/- 5 gigs an hour consisting of adds, modifications, deletes).

Location2 will be coming up soon and they too have local storage space available. I would like them both to be working in their local shares respective of their locations but have everything synced/replicated at both locations.

A few quick notes: - Both sides have NetAPP filers - Users are connecting to CIFS shares directly to the Netapp filers - We can not use the netapp sync mirror (1. no license, 2. it is a one directional read only transfer) - Location1 has a private 500mbit fiber ethernet connection - Location2 has a private 100mbit fiber ethernet connection - Because the LUNS are not mapped to a server (FC,iscsi), we can not use DFSR or GlobeSCAPE for bit level replication

I'm looking for a stable solution that will connect over CIFS and sync both locations.

Thanks for your input.

  • l30
Improve this question
1
  • Any particular OS or do you want something that will work for everything?
    – John Gardeniers
    Jun 22, 2010 at 23:03

3 Answers 3

Reset to default
4

Use rsync.

Improve this answer
4
  • Given that these are CIFS shares, rsync may not be a viable option. Is it available for a cygwin or similar environment? Native Windows?
    – Tekhne
    Jun 22, 2010 at 18:37
  • 1
    You should be able to install rsync and run it as a service (at one end) and from the command line or scheduled task (at the other end) using cygwin. You can definitely install ssh (client at both ends, server at at least one) & rsync with cygwin and use rsync-over-ssh, I find it works well on my Windows systems, syncing to/from both other Windows machines and those running Linux.
    – David Spillett
    Jun 22, 2010 at 18:45
  • I'm currently using rsync but it just dosn't perform well. Everytime it runs, it has to build a list of all the file changes. And when it starts to sync, if something was moved that doesn't correspond to its list, it throws errors. I'm looking for a real time replication solution that will let me use CIFS shares. THanks
    – l30
    Jun 22, 2010 at 19:09
  • rsync is the right answer; the problem is that netapp doesn't support it. to use it correctly, it has to run on both sides of the slow link, ideally it would be on the servers themselves; if not possible, it might still work ok between two clients, each one mounting locally each server.
    – Javier
    Jun 22, 2010 at 19:48
1

to expand on my comment to mcandre's answer:

  • on each location, get a fast workstation and mount the local server.
  • run rsync (or better, Unison) between those workstations.

since the copy takes some time, and things change during the copy, you have two different options:

  • if the server supports it, use snapshots and copy between those. the semantics get confusing (can you write to a snapshot? if so, should you propagate that to the 'live' volume?) so i don't think it would be best option

  • do it incrementally:

    1. run a full copy. it will take hours, and there will be errors.
    2. run it again, it will correct those errors, and will take a lot less time, so there might be new errors, but should be far less.
    3. run it again. now it should take only a few minutes.
    4. suspend any other activity, run it one last time, and unsuspend everything.

because of the differential copy, each time you run it, it gets faster, up to a limit. when you get to that limit, it's not so hard to suspend other activities for a short time. if you can't get that time, then you should bite the bullet and get a server system that supports real sync natively.

Improve this answer
0

You're going to be pushed finding a solution that deals with any conflicts that can occur. What kind of changes are happening (regarding file changes... is it lots of small files?).

It may be something where you could throw a small application together using the Microsoft Sync Framework that keeps the 2 directories in sync. This would offer you a user interface for resolving conflicts (if you wrote one in).

Improve this answer
2
  • Files are graphic images. 99% of the the time it is just file additions. Also both sites work independently on files stored on their desktop and only when they are done they copy it to the share. The share is mostly an archive. Sometimes they place a file in the wrong directory which has to be moved.
    – l30
    Jun 22, 2010 at 19:12
  • You could write an application that has FileSystemWatchers in it that runs at both ends and copies the files to the other side when they're changed, moved, deleted or created. It could run as a system service and would be fairly quick with low overhead.
    – Matthew Steeples
    Jun 22, 2010 at 19:26

You must log in to answer this question.