3

Does anyone know of a solution that lets you sync files bidirectionally over a WAN, but also syncs immediately/incrementally?

Basically Unison is perfect for the file syncing, but it doesn't sync incrementally - it will wait for a file to finish changing before syncing, and unfortunately we need the sync process to start immediately.

Essentially the workflow is:

  1. User uploads file to Server A
  2. Server A starts syncing the file to Server B as it is being uploaded
  3. Upload finishes
  4. Server B does some processing, and writes some results to a directory
  5. Server B syncs the results back to Server A
  6. User downloads the results from Server A

Server A also does processing, but since it writes back to Server A it is not important for syncing purposes.

My first thought was to perhaps use DRBD in dual-primary with a clustered filesystem, but a) the WAN latency is a good 300ms and bandwidth can sometimes be poor (500K/s) and b) we can't change the current filesystems, and Server B is using a NAS (NFS) as its storage - I don't believe that combination (and using loopback devices) would be practical (correct me if I'm wrong).

1 Answer 1

2

I think GlusterFS can solve your problem.

It is a distributed filesystem that operates in userspace, so you don't have to change your current filesystem. With a gluster replicated volume you have a single virtual FS, which you can mount on your servers using FUSE module or NFS. This virtual FS is immediately synced between the nodes on your cluster.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .