Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am searching for a very fast file syncing software to synchronize data across multiple machines, across multiple sites. Something like Dropbox or Spideroak.

The software should fulfill the following two criteria:

1) and most important. It should be very fast. If I change a file, the upload to the server should start within 10s. The other connected machines should start downloading asap as soon as the upload completes. If the software would support locking the file in question it would be great.

2) it should support client side encryption.

Dropbox is not suitable because of the second point, Spideroak is not suitable because of the first (sometimes I got very disappointing results).

What file syncing tools are you using? What can you recommend?

EDIT: The computers are all running linux systems.

EDIT2: I have 4-5 sites in mind. Unfortunatelly they have not very huge uploads. The worst (my home office;-)) has only a DSL connection with 80 kb/s upload. Putting something together myself is not an option, since I am searching for a solution with moderate costs and reliability ;-). I have root access at all the clients in question.

share|improve this question

5 Answers 5

up vote 3 down vote accepted

I would recommend GlusterFS. It's a NAS filesystem that connects several servers into one filesystem. You can then mount it via fuse. You could also connect the servers via an IPsec solution, see Openswan for example. To add security.

From wikipedia about GlusterFS:
GlusterFS has a client and server component. Servers are typically deployed as storage bricks, with each server running a glusterfsd daemon to export a local file system as a volume. The glusterfs client process, which connects to servers with a custom protocol over TCP/IP, InfiniBand or SDP, composes composite virtual volumes from multiple remote servers using stackable translators. By default, files are stored whole, but striping of files across multiple remote volumes is also supported. The final volume may then be mounted by the client host through the FUSE mechanism or accessed via libglusterfs client library without incurring FUSE filesystem overhead. Most of the functionality of GlusterFS is implemented as translators, including:

  • File-based mirroring and replication
  • File-based striping
  • File-based load balancing
  • Volume failover
  • scheduling and disk caching
  • Storage quotas
share|improve this answer
    
Thanks for your answer. This sounds very interesting. I will take a look at it. –  ftiaronsem Aug 21 '11 at 5:44

How many sites are you talking about? Personally I'd look at setting something up with my own servers using something like the DRBD filesystem or DFS (Windows uses DFS as a synching filesystem over the network, DRBD is a Linux RAID 1 over IP solution), then have the clients connect to a share on the servers (or a mapped drive) and everything would be synced automatically.

As a second research topic, rsync directories among servers over SSH.

Otherwise you might be looking at rolling your own application and set of scripts to do what you're looking for which would probably not be simple, cheap or easy.

Without knowing specifics (number of sites, control at the client sites you have, bandwidth, etc.) it's hard to make other suggestions.

EDIT - DRDB seems optimized for 2 servers; I don't know what it would take to "chain" data. Plus are you going to sync data from one server out to the outliers? Have you planned the priority of sync paths (as in do you have a central repo that everything is syncing from, or are you decentralized in where and how the data will be coming from and going to? Or are these outlying offices syncing things to a central server?) You could be looking at quite a complicated setup when you factor these things in. You would have to look at either having a way to run a sync utility as suggested by http://billboebel.typepad.com/blog/2006/11/data_mirroring_.html or rsync at particular times or finding a cluster-based filesystem that handles multiple active "primary" peers without enough overhead that it slams your bandwidth.

You didn't mention the size of the data being edited or reliability of your connection; if you're dealing with the average document you'll have different potential corruption and issues than if you're editing large graphics.

Given the complication of this kind of setup you're looking at, I would also suggest considering a remote access solution as a potential fix. If you're running Linux it's not altogether impossible to have a central server in the "primary" office and have people log in with remote SSH terminals and run sessions directly off the server, much like a Windows Terminal Services solution. This gives you more control over how the data is backed up, securely accessed and audited. But you should have a decent connection to do this. Very fast connections would allow SSH connections with X-forwarding, mediocre to fast connections you could set up a remote terminal solution more akin to VNC (encrypted tunnel/VPN) for the client<->server connections.

Another consideration is VPN to a central site and mount directories via NFS or a FUSE module like SSHFS. Again, depends on your bandwidth and connection stability.

If you want to keep syncing data as a solution you'll still have potential lock problems and data being updated with race conditions to consider so you'll need to research the best filesystems that can automatically handle that situation.

share|improve this answer
    
thanks for your answer, I have added the requested information to my question. Thanks, ftiaronsem –  ftiaronsem Aug 20 '11 at 15:08
    
Depending on the overall infrastructure, DRDB is a way to sync up servers without thinking more about it to each site and from there you can mount a directory off the server without syncing to the clients. But any solution involving something like what you're describing will take some management and monitoring for reliability. –  Bart Silverstrim Aug 20 '11 at 17:32
    
Added a little more to think about, too. –  Bart Silverstrim Aug 20 '11 at 20:21
    
Thanks Bart for this very detailed answer and your hints to think about. I will try glusterfs for starters and go through the other suggestions here, if I am not satisfied. Your answer definitly was very handy, thanks. –  ftiaronsem Aug 21 '11 at 5:45

If you're using Windows for your clients and servers, I suggest you investigate Distributed File System. Also take a look at offline caching with EFS. If you're not using Windows, please let us know what you're using.

Edit: Take a look at tsync (beta) for Linux.

share|improve this answer
    
Thanks for your answer. Unfortunatelly all the computers are running linux OSes. Sry for forgetting to mention that in the first place. –  ftiaronsem Aug 20 '11 at 15:02

ChironFS is a distributed filesystem designed for replication. It doesn't take care of encryption, but you can use it underneath an encrypted filesystem like EncFS if you want each client to manage encryption, or ue SSHFS to protect the data over the wire. I don't know if it'll be suitable performance-wise.

share|improve this answer

You may want to look at Unison http://www.cis.upenn.edu/~bcpierce/unison/ however a filesystem solution may fit your needs better.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.