Does anyone know of a solution that lets you sync files bidirectionally over a WAN, but also syncs immediately/incrementally?
Basically Unison is perfect for the file syncing, but it doesn't sync incrementally - it will wait for a file to finish changing before syncing, and unfortunately we need the sync process to start immediately.
Essentially the workflow is:
- User uploads file to Server A
- Server A starts syncing the file to Server B as it is being uploaded
- Upload finishes
- Server B does some processing, and writes some results to a directory
- Server B syncs the results back to Server A
- User downloads the results from Server A
Server A also does processing, but since it writes back to Server A it is not important for syncing purposes.
My first thought was to perhaps use DRBD in dual-primary with a clustered filesystem, but a) the WAN latency is a good 300ms and bandwidth can sometimes be poor (500K/s) and b) we can't change the current filesystems, and Server B is using a NAS (NFS) as its storage - I don't believe that combination (and using loopback devices) would be practical (correct me if I'm wrong).