We have 1TB of files that we wish to keep in sync between three locations. This data consists of around 1.5 Million files which grows/changes at an average of 100MB/day.
The first location is a Queensland office which has a dedicated ADSL2+ syncing at 20000/1024kbps connected to a co-located server in a data centre on a 10MB link. The remote site is an office in Argentina which is on a measly SDSL 512/512 link. This could possibly be upgraded to a maximum of 2Mb SHDSL.
The structure we want is: QLD Office <--> QLD Datacentre (For Offsite Backup Only) <--> Argentina Office
I ran some speed tests from speedtest.net from the QLD office and the Argentina Office:
Upload speed from QLD office: 0.95Mb/s
Upload speed from Argentina office: 0.51Mb/s to local server, 0.38Mb/s to QLD Server :(
We can’t use standard sync software (rsync, viceversa etc) to keep this data in sync because of the time zone differences make it really complex so we are considering a distributed file system solution.
We are trying to work out what is more viable – using Microsoft DFS-R with Windows 2008 servers or a third party WAFS provider such as GlobalScape WAFS.
I’m trying to find a solution that will do the following:
- The ability to initially populate the files by posting an external hdd to the remote sites etc.
- The ability to manually populate after it is set up – i.e. if we need to add 200GB of files we can post the drive again with this data and add it at all sites manually
- The ability to account properly for the large time zone difference of the remote sites
- Offline availability of files – i.e. if internet connections are down we can still view/edit files
- Either Delta Sync compression or file compression – a lot of files we work on are large ascii files which are easily compressed with a large ratio so compression would be nice.
- File locking would be nice but is not essential since the files are in different time zones
- The ability to do all of the above with the slow internet links as described.
I have looked at GlobalScape WAFS which looks promising because unlike most DFS you can point it at where the files are located instead of copying them into the share, but with a quote of $10kUSD for 3 agents for the software alone I was wondering if there are any better solutions.
Microsoft DFS-R looks good also but I can’t find much information as to if it would handle this many files over a very slow link.
Any suggestions/directions to point me in would be much appreciated