Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

A friend of mine has asked for some advice about the I.T. infrastructure he should put in place for his small business.

His business is in the science field and is very data-centric (i.e., they create and consume a lot of data, to a point that it is one of their major assets).

They would like infrastructure that which would automatically synchronise data for each of their projects with a server in the office. However, they also need to be able to work on local copies of their data whilst they're out doing field work - synchronisation should restart automatically when they return to the office.

As a software engineer, I know about SCM tools such as SVN, Git, etc. However these tools don't seem to be that well suited to the job - commits and updates need to be performed manually. Additionally, these tools are targeted towards plain text data.

So my question is this: What Software/Infrastructure should I recommend?

Some considerations:

  • It must be able to handle large (100s of MB to several GB) files
  • Field work is in remote locations and the data is large, making mobile internet impractical
  • Primarily data is proprietary scientific formats (i.e., binary data)
  • MS Windows Environment
share|improve this question
    
Roughly how many users require this type of service? –  CurtM Nov 23 '10 at 2:03

4 Answers 4

There's likely a better way, but what comes to mind at first would be something to the effect of a sheepdog cluster.

If the Windows machines are in QEMU/KVMs they can be moved in & out of the cluster that lives at the office- to & from a laptop for example.

If each project has it's own fileserver in QEMU/KVM, a copy could be migrated to the laptop to provide a snapshot of the project's datastore running alongside and providing data to the Windows instance on the laptop node.

When returning to the office, the project's fileserver would be returned to the main cluster, and data epochs, or versions would be automatically updated across the cluster.

http://www.osrg.net/sheepdog/index.html

share|improve this answer

If the "local" copy is always Authoritative, something like CrashPlan Pro might work. It is a backup software, so it doesn't "sync" so much as just always copy the changed data from the client to the server.

We use it for a few of our clients and it works very well. I haven't used it for Binary Data, but it does support "unlimited size files".

share|improve this answer
    
Reading your question again, it seems like the users need to work against the server version of the data when they are in the office. If that's the case, you can ignore this, as CrashPlan stores all backups in its own format, so you can't just use it as a separate synced location. –  minamhere Nov 23 '10 at 3:18

It sounds like you're talking about file replication rather than database replication - there are different approaches to each.

I'd recommend having a long hard look at afs and it's derivatives (although afs is the only one I'm aware of which runs on MSWindows).

However it does sound very peculiar that your friend works in the "science field" with very large volumes of valuable data which are not maintained in a structured database.

share|improve this answer

What about Windows offline files? AFAIK, there's no limit as far as offline cache is concerned and with Windows 7's new Transparent Cache feature, files are cached more aggressively so there's less trips back to the network (server) copy if the file hasn't changed, which can dramatically reduce load on your network (speaking of which, I'd strongly recommend Gigabit ethernet and quality switches, possibly NIC bonding on your server with fast SAS 15k/RAID 10 storage or if no server required, a high-performing NAS).

I know I've had some clients with fairly large offline caches (10s of GBs) on laptops and offline folders worked flawlessly.

For multiple users accessing and modifying the same data offline, well, it's not the most elegant and doesn't compare to a true version control system, but it doesn't sound like that would be an issue here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.