I have the requirement to sync files (documents mostly, sound bites, high quality images) between multiple PCs, Macs and Linux clients and servers.

The number of files is large and delta syncs would be required. I would like a file changed on anywhere to be replicated. Additionally offline access is required. Version Control would be exceptionally useful and graceful recovery from duplicates etc.

In short I'm looking for dropbox but not hosted, SaaS or any other 3rd party involvement. Live mesh from Microsoft (or what it's called this week) would be ideal but for again 3rd party.

I figure this must be out there with a webdav backend or such and I just haven't found the cleanest solution.

I'm looking for the best way to implement the above. Open Source tools preferred but not explicitly required.

Thanks all.

  • Does the syncing need to be automatic? Will it be performed by clueful users? – Clinton Blackmore Dec 4 '09 at 20:28
  • @Clinton Blackmore; in a wide enough install base assuming clueful and diligent users is dangerous. So no probably automated safest – Antitribu Dec 10 '09 at 17:33

What you describe sounds like a very general solution, which might be impossible (e.g. you cannot have both offline access and completely avoid manual merging, e.g. for multimedia files).

In addition to the other solutions, you might want to look at Unison. It will sync multiple directory trees to keep them identical after changes (particularly offline changes). Unlike rsync, it will actually cope with changes in multiple copies. It will intelligently merge them if possible, or prompt if it detects conflicts.

I regularly use it to sync my desktop to mobile devices, and it works really well.

It does need to be run periodically however, it is not a continuous service.

  • 1
    Unison has a 'repeat' option where it repeatedly re-syncs e.g. every 30 seconds. – Michael Ekstrand Mar 9 '11 at 15:47

the environment you are trying to create could fast become very messy. what if multiple copies of a binary file changed at the same time in different places? How are you going to merge those changes?

You probably want to implement a (distributed) network file system of some sort. Versioning could be done with an underlying file system snapshots.

I am not sure if WebDAV will give you easiness and I/O you are looking for, but it could be so. Other alternatives are NFS, OpenAFS and CIFS.

  • Messy yes I know, but it can be done, just look at drop box/live mesh for evidence. NFS, CIFS don't give local copies as such. I wasn't aware that OpenAFS had builds for Windows and Mac but will definitely be looking down that path. The Distrib file system is the idea; i was just hoping someone had a private backend for dropbox. – Antitribu Dec 4 '09 at 17:59
  • now, said all of this, I think maybe what you really want is a digital asset management solution (en.wikipedia.org/wiki/Digital_asset_management). and I don't know any open source solution there, it doesn't mean they don't exist. – monomyth Dec 4 '09 at 18:03

Use the popular rsync tool. It is multiplaform, mature and you'll get a lot of support.

  • Sorry should have stated it in the question but I'm also looking for something that requires minimal interaction from users and isn't so much of a "cron job thing that gets run every minute". Similar to dropbox where files are detected as changed. – Antitribu Dec 4 '09 at 17:56

Sounds like a job for Subversion to me, what w/ the version control requirement. Had you not said that, I'd have probably suggested either rsync or Unison, but neither of those is going to help you with version control.

You can get nice graphical clients for Windows and MacOS (see http://en.wikipedia.org/wiki/Comparison_of_Subversion_clients for a long list of clients), and access via WebDAV is available.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.