IT Infrastructure for Small Business - Data Synchronisation

Question

A friend of mine has asked for some advice about the I.T. infrastructure he should put in place for his small business.

His business is in the science field and is very data-centric (i.e., they create and consume a lot of data, to a point that it is one of their major assets).

They would like infrastructure that which would automatically synchronise data for each of their projects with a server in the office. However, they also need to be able to work on local copies of their data whilst they're out doing field work - synchronisation should restart automatically when they return to the office.

As a software engineer, I know about SCM tools such as SVN, Git, etc. However these tools don't seem to be that well suited to the job - commits and updates need to be performed manually. Additionally, these tools are targeted towards plain text data.

So my question is this: What Software/Infrastructure should I recommend?

Some considerations:

• It must be able to handle large (100s of MB to several GB) files
• Field work is in remote locations and the data is large, making mobile internet impractical
• Primarily data is proprietary scientific formats (i.e., binary data)
• MS Windows Environment

Answer 1

There's likely a better way, but what comes to mind at first would be something to the effect of a sheepdog cluster.

If the Windows machines are in QEMU/KVMs they can be moved in & out of the cluster that lives at the office- to & from a laptop for example.

If each project has it's own fileserver in QEMU/KVM, a copy could be migrated to the laptop to provide a snapshot of the project's datastore running alongside and providing data to the Windows instance on the laptop node.

When returning to the office, the project's fileserver would be returned to the main cluster, and data epochs, or versions would be automatically updated across the cluster.

http://www.osrg.net/sheepdog/index.html

Answer 2

If the "local" copy is always Authoritative, something like CrashPlan Pro might work. It is a backup software, so it doesn't "sync" so much as just always copy the changed data from the client to the server.

We use it for a few of our clients and it works very well. I haven't used it for Binary Data, but it does support "unlimited size files".

Answer 3

It sounds like you're talking about file replication rather than database replication - there are different approaches to each.

I'd recommend having a long hard look at afs and it's derivatives (although afs is the only one I'm aware of which runs on MSWindows).

However it does sound very peculiar that your friend works in the "science field" with very large volumes of valuable data which are not maintained in a structured database.

Answer 4

What about Windows offline files? AFAIK, there's no limit as far as offline cache is concerned and with Windows 7's new Transparent Cache feature, files are cached more aggressively so there's less trips back to the network (server) copy if the file hasn't changed, which can dramatically reduce load on your network (speaking of which, I'd strongly recommend Gigabit ethernet and quality switches, possibly NIC bonding on your server with fast SAS 15k/RAID 10 storage or if no server required, a high-performing NAS).

I know I've had some clients with fairly large offline caches (10s of GBs) on laptops and offline folders worked flawlessly.

For multiple users accessing and modifying the same data offline, well, it's not the most elegant and doesn't compare to a true version control system, but it doesn't sound like that would be an issue here.