Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm looking for a solution to the following:

Server A (Site A) - Win 2008 R2 - approx 10TB (15TB max) of data - well over 8 million files

Server B (Site B) - Win 2008 R2

I want to assynchronously replicate Server A's volume to a volume on Server B for data redundancy. Something that I can say to my users, "go here for data" when/if Server A goes belly up due to machine problems, disaster, etc.

Windows 2008 R2 does have DFS, but microsoft does not apparently support this large of a dataset (or more accurately, more than 8 million files - according to the docs I could find).

I also looked at Veritas Volume Replication, but this seems almost too much as I would also require Veritas Volume Manager.

There are numerous "back-up" software which makes a 1-1 backup, which would be ok, but since it will be transfering over internet, I'd like something that has compression during transfer like DFS has.

Does anyone have any suggestions regarding this?

share|improve this question
add comment

5 Answers

I've used Doubletake Move to move large datasets across the Internet. They use byte level replication, and keep track of changes to files. There is also a nice bandwidth throttling scheduler to use less during the day, and crank it up at night and on the weekend. It also recovers pretty well in the event of a connection break of some kind.

Now, I am assuming this is some sort of MSA attached to a physical machine, but in the event you are using a SAN, Check with your SAN vendor for async replication options.

Whatever replication you use, there are a couple things you want to think about:

  1. Bandwidth at the source and target side
  2. File rate of change

If your rate of change is too high on the source side, and you don't have enough bandwidth to overcome it, you will never get a good replication.

Re-indexing databases, defrags, and bulk file moves/adds/deletes have all caused me headaches in the past.

Hopefully my past pain will help someone that reads this! :D

share|improve this answer
add comment

DFSR in 2003 R2 (2008 and 2008 R2 are likely much more scalable, exp from being x64) worked so wonderfully for us with 75 servers with all on different WAN links (1-10MB) syncing files totaling at least 500GB (per server times 75) on bad, slow, or saturated WAN links that I would just recommend doing what you can to get DFSR to work. We found it easier to manage org wide then Veritas, but this was 2006-7 after 2003 R2 came out. GPO’s controlling BITS bandwidth is your friend.

One of the things with DFSR is it keeps a folder with copies of all the blocks that were changed that need to be sent out, so storage is something you want to be liberal with. I am curious about the limits you quote, as I would think that 8M files would be fine in context of plenty of resources (RAM, CPU, Disk). I don't know our file count.

Also, in the past DFSR has been incompatible with backup, A/V, and other software. In 2008-2009 we found that Netbackup didn’t like DFSR and was reporting success with our file server backups but was actually backing up NOTHING. Only on testing restores did we discover this horrible, horrible fault in Netbackup. If there’s one thing you never want backup systems to do, is report a successful backup but truly have an empty tape.

Anyway I give a vote of confidence to DFSR, especially in its 3rd version with 2008 R2 as something you should give a shot if you can't find a vendor's product that specifically says they've tested your scenario. Often what Microsoft officially supports is much more conservative to what they know will work. Obviously your mileage may vary, and you have to determine your own level of risk you're willing to take.

share|improve this answer
add comment

The key for looking at replication is to look for a "consistent set" of data that gets replicated. For example: Db & corresponding Log files should be replicated in a "consistent" manner such that the data is use able on the other replicated site.

The second important feature to look at is - the time required to recovery in case there is a drop in connection. Does the replication start from where it left? Does it re-start the replication?

The third - how well (or worse) do they perform on varying network condition and what is the bandwidth utilization.

The other important things to look at are - are the file permissions maintained? Are the other file attributes maintained eg. what happens to compressed folders? What happens to encrypted files? Are open files replicated? etc etc.

Give all the above, block-based replication solutions are much better than file based replication. Host block-based replication would be cheaper than "off-host" block-based replication.

share|improve this answer
add comment

I have a Customer using the SureSync product to replicate data to a "hot standby" file server. They're replicating just under 2TB and it's worked very well. It uses the NTFS change journal to track file updates and performs delta replication.

I'm not finding published product limitations, so you'd probably better check in with the "manufacturer" to see.

share|improve this answer
add comment

While Veritas Volume Manager with the Veritas Volume Replicator option isn't cheap, it may offer some advantages, such as:

  • It works at the block level, so it doesn't need to replicate entire files, or wait for files to be closed before saves can be replicated.
  • Write-order fidelity is maintained, so the order that data are changed on the primary is the order data are changed on the secondary - good for databases and the like.
  • If you have to fail over to the secondary, then your modifications to the secondary are easily synced back to the primary.
  • Compression and throttling of the transfer.

There is a tool (VRAdvisor) available which you can use to monitor your rate of data change, which will then tell you how much bandwidth you need to keep the secondary within a given data-change amount of the primary.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.