5

PLEASE READ UPDATE AT THE BOTTOM. THANKS! ;)

Environment Info (all Windows):

  • 2 sites
  • 30 servers site #1 (3TB of backup data)
  • 5 servers site #2 (1TB of backup data)
  • MPLS backbone tunnel connecting site #1 and site #2

Current Backup Process:

Online Backup (disk-to-disk)

Site #1 has a server running Symantec Backup Exec 12.5 with four 1TB USB 2.0 disks. BE jobs for full backups run nightly on all servers in site #1 to these disks. Site #2 backs up to a central file server there using software they already had when we purchased them. A BE job pulls that data nightly to site #1 and stores them on said disks.

Off-site Backup (tape)

Connected to our backup server is a tape drive. BE backs up the external disks to tape once a week which gets picked up by our off-site storage company. Obviously we rotate two tape libraries, one is always here and one is always there.

Requirements:

  • Eliminate the need for tape and off-site storage service by doing disk-to-disk at each site and replicating site #1 to site #2 and vice versa.
  • Software based solution as hardware options have been too pricey (ie, SonicWall, Arkeia).
  • Agents for Exchange, SharePoint, and SQL.

Some Ideas So Far:

Storage

DroboPro at each site with an initial 8TB of storage (these are expandable up to 16TB at present). I like these because they are rackmountable, allow disparate drives, and have iSCSI interfaces. They are relatively cheap too.

Software

Symantec Backup Exec 12.5 already has all the agents and licenses we need. I'd like to keep using it unless there is a better solution, similarly priced, that does everything BE does plus deduplication and replication.

Server

Because there is no more need for a SCSI adapter (for tape drive) we are going to virtualize our backup server as it is currently the only physical machine save for SQL boxes.

Problems:

  • When replicating between sites we want as little data as possible to go across the pipe. There is no deduplication or compression in what I have laid out here so far.
  • The files being replicated are BE's virtual tape libraries from our disk-to-disk backup. Because of this each of those huge files will go across the wire every week because they change every day.

And Finally, the Question:

Is there any software out there that does deduplication, or at least compression, to handle just our site-to-site replication? Or, looking at our setup, is there any other solution that I am missing that might be cheaper, faster, better?

Thanks. Sorry so long.

UPDATE 2:

I've set a bounty on this question to get it more attention. I'm looking for software that will handle replication of data between two sites using the least amount of data possible (either compression, deduplication, or some other method). Something similar to rsync would work but it needs to be native to Windows and not a port involving shenanigans to get up and running. Prefer a GUI based product and I don't mind shelling out a few bones if it works.

Please, answers that meet the above criteria only. If you don't think one exists or if you think I'm being to restrictive keep it to yourself. If after seven days there is no answer at all, so be it. Thanks again everyone.

UPDATE 2:

I really appreciate everyone coming forward with suggestions. There is no way for me to try all of these before the bounty expires. For now I'm going to let this bounty run out and whoever has the most votes will get the 100 rep points. Thanks again!

14
  • 5
    Are you absolutely sure that you want to eliminate your tape solution entirely? Jun 16, 2009 at 21:36
  • 8
    God, I wish I had off-site and tape backup. My boss won't let me invest what's necessary. Do not throw that away, please! Don't! Jun 16, 2009 at 21:55
  • 3
    Eliminating tape is sexy to people, FWIW. I don't agree with it, but people seem to get really excited about it. Jun 16, 2009 at 23:12
  • @Matt: Sure. @Server Horror: We will have off-site backup in this scenario. @Evan: Sexy? Really? Jun 18, 2009 at 12:01
  • @Evan: Thanks for your comments and answer below but we have decided to eliminate tape for several reasons. Just looking for software now that might help us with dedupe/compression for our site-to-site replication. Again, we never used tape for retention. Jun 18, 2009 at 13:46

18 Answers 18

6
+50

Windows Server 2003 R2 and later has support for DFSR, which I used extensively to sync and backup large amounts of data over a rather small pipe across three sites (80GB+ over a T1<-->T1<-->T1 topology).

msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx

Replicating data to multiple servers increases data availability and gives users in remote sites fast, reliable access to files. DFSR uses a new compression algorithm called Remote Differential Compression (RDC). RDC is a "diff over the wire" protocol that can be used to efficiently update files over a limited-bandwidth network. RDC detects insertions, removals, and rearrangements of data in files, enabling DFSR to replicate only the deltas (changes) when files are updated.

DFSR is fully multimaster and can be configured however you want. That will keep your data in sync on the "backup" location, for a very small amount of bandwidth and CPU. From here, you can use the Volume Shadow Copy Service.

technet.microsoft.com/en-us/library/cc785914.aspx

The Volume Shadow Copy Service can produce consistent shadow copies by coordinating with business applications, file-system services, backup applications, fast-recovery solutions, and storage hardware. Several features in the Windows Server 2003 operating systems use the Volume Shadow Copy Service, including Shadow Copies for Shared Folders and Backup.

The shadow copies reside on disk, and take "no space" aside from the changed files from snapshot to snapshot. This is a process that can run on a live dataset with no ill effects, aside from slightly increased disk I/O as the snapshot is being created.

I used this solution for quite some time with great success. Changes to files were written out to the other sites within seconds (even over the low bandwidth links), even in cases where just a few bytes out of a very large file changes. The snapshots can be accessed independently from any other snapshot taken at any point in time, which provides both backups in case of emergency and very very little overhead. I set the snapshots to fire at 5 hour intervals, in addition to once before the workday started, once during the lunch hour and once after the day was over.

With this, you could store all data in parallel at both locations, kept relatively up to date and "backed up" (which amounts to versioned, really) as often as you want it to.

The Shadow Copy Client can be installed on the client computers to give them access to the versioned files, too.

www.microsoft.com/downloads/details.aspx?FamilyId=E382358F-33C3-4DE7-ACD8-A33AC92D295E&displaylang=en

If a user accidentally deletes a file, they can right-click the folder, properties, Shadow Copies, select the latest snapshot and copy it out of the snapshot and into the live copy, right where it belongs.

MSSQL backups can be written out to a specific folder (or network share) which would then automatically be synched between sites and versioned on a schedule you define.

I've found that data redundancy and versioning with these can act as an awesome backup system. It also gives you the option to copy a specific snapshot offsite without interfering with the workflow, as the files it reads from aren't in use...

This should work with your setup, as the second backup site can be configured as a read-only sync/mirror.

0
4

Windows isn't my area of expertise but rsync may help get the backups from one site to the other. Rsync works by breaking down files into smaller blocks and then only transfers the blocks that changed across the network. It can also compress the data as it sends it.

There are some versions of it for Windows out there but I've never used them so I can't comment on how well they work. With cygwin you can get rsync on windows but that may make things a bit messy. But ideally you should find a rsync client for windows that will allow you to use the scheduled tasks to automate it's execution.

6
  • Here's a free Windows version of rsync that a quick search turned up. Again I have no experience with this one. aboutmyip.com/AboutMyXApp/DeltaCopy.jsp Jun 16, 2009 at 21:33
  • I would definitely look into rsync, but I would hope a company like Symantec would have a similar solution.
    – Harley
    Jun 16, 2009 at 21:38
  • 1
    Harley, Symantec does have a replication/dedupe solution called PureDisk but it is an add-on for NetBackup. Would be way too pricey to switch. 3dinfluence, thanks for the link. I'll check it out. Jun 16, 2009 at 21:48
  • 2
    DeltaCopy is way outdated. If you want to use rsync on Windows use cwrsync (itefix.no/i2/node/10650).
    – Jonas K
    Jun 16, 2009 at 22:37
  • 1
    We use rsync as well, and have found it far better than any commercial off-site transfer mechanism. It allowed us to eliminate tapes and move entirely to disk based storage and archival.
    – Beep beep
    Jun 20, 2009 at 17:29
2

Edit:

We are using "SureSync" from Software Pursuits (see http://www.softwarepursuits.com/SureSync/SureSync.asp) to replicate data between a production and standby server in a Customer's remote site with great success. It is a native Windows application, runs as a service on the publisher and subscriber machines, copies deltas, retains security, follows the NTFS change journal, and in general has rocked for our needs.

(Our particular Customer who is doing this still has traditional off-site tape backup rotation, too. I think you still need offline backups, and I can't answer your question in good conscience without leaving that section of my answer intact, but I can tell you that SureSync has been great. Read some of the changelogs on the product-- it's clear to me that the manufacturer is really, really attentive to detail.)


One observation: If you eliminate tape, you're eliminating offline storage. Off-site is one thing, but offline is a different thing. When a remote attacker destroys your production system it's really, really nice to have an air gap between the tapes and the tape drive to stop them from being able to destroy the backups too.

You need off-site storage of backups, and you need offline backups, too.

It's also very hard to have an independent third-party do a test restore and data verification without something like tape. Perhaps in your industry that's not a concern, but I've worked for financial institutions that sent their data, via tape, off-site to a third-party to independently verify the integrity of their data, both from a "restorability" perspective, and from a "let's see if your totals compare properly with the totals that we compute on a trusted installation of your application using only your data as input".

5
  • In my experience this security risk is only present if you're pushing your backups offsite. Which is a really bad idea. You should be pulling your backups from your offsite location. Jun 17, 2009 at 2:36
  • I don't buy the push / pull argument. Unless you're saintly about not using the same credentials to administer your backup infrastructure as your production infrastructure, somebody who wants to screw up your data could attack the a "pull" backup infrastructure. It's harder, in my mind, to attack tapes stored in a physically secure facility than it is to access computers via a network. Jun 17, 2009 at 13:00
  • @Evan: That is a good point. If we decide that offline backup is necessary then we will probably have two sets of disks. Tape is not necessary to accomplish this. Jun 18, 2009 at 12:08
  • Heh heh... I didn't know you were going to "go bounty" on it. Sometimes I get so wrapped up in my deeply-held opinions that I forget to speak to what the post is asking about. Just do me a favor and don't end up w/ both your production and backup data trashed-- I'll feel like it's partially my fault. SureSync, though, has been very helpful. We've done cutovers to the standby server, both testing and in real failure situations (thanks Dell MD1000), and we've always had consistent data. I'd use rsync in some low-end situations, but SureSync does a great job when it needs to be done right. Jun 20, 2009 at 1:33
  • I know I said I was willing to pay but their quote was well over what we can afford. Thank you for your suggestion though. Jun 26, 2009 at 18:38
1

If you want to increase the speed of your site to site replication, you might look into a WAN accelerator. There are several on the market. Another admin just recommended the ones from Riverbed to me: http://www.riverbed.com/index.php?cnt=1

Essentially they compress the data before sending it and decompress the data after receipt. It's seamless to the user.

1
  • Thanks, but looking for a software solution. Jun 26, 2009 at 18:39
1

BackupExec doesn't make this easy. Ideally, you should have the option to 'copy' a backup to somewhere, and I'm not sure BE has that.

Here is what I'd build if I were using HP Data Protector in this environment. It does have a 'copy' for jobs. It also has a de-duplication option, but I'd be deeply careful of that on file-systems with more than about 500K files on it.

  • Disk array in both locations You'll need it for DR replication
  • Continue backups in both locations
  • Use a copy job on Monday to copy weekend jobs to the remote site Once a week, copy your full backup from tape to the remote data-center's disk-array. The DP database will keep track of both backup jobs.
  • Set expiration of the B2D job appropriately Decide how many remote copies you need in the remote location, and set the data-retention period correctly on the copy job. This can be a separate interval than your tape rotation retention.

I believe you can set disk-backups to use compression, but this'll really slow down your backup speeds as that'll be done in software.

The DataProtector dedupe works only for file-servers on Windows and Linux. It won't de-dupe Exchange/MS-SQL/Sharepoint.

1
  • Backup Exec can do copy jobs. Jun 16, 2009 at 22:35
1

IBM aquired a company previously called "Softek" that has a software solution called Replicator. It's block level replication that runs over TCP/IP. After the initial synchronization is complete only the changed blocks are copied over to the remote site. So just because one of your huge BE files gets changed somewhat it would not be necessary to copy the entire file. This is a native windows application, has an easy to use console, and is a really good way to manage disk synchronization over a network.

IBM/Softek Replicator

1
  • I looked at this at your suggestion but can't seem to get a hold of them about it. I wonder if they are going to keep developing this product. Thanks. Jun 26, 2009 at 18:36
1

I had a similar issue about a year ago and looked at everything from robocopy and rsync to Cisco WAAS and WAN accelerators. Eventually I stumbled upon a stupid cheap solution that works great for securely and quickly delta syncing files between sites. Delta syncing is the key. Most, if not all, P2P clients do full file syncs only.

Powerfolder
It has a LAN-Only mode, allowing you to specify which sets of IP's you'll allow clients to connect with. It also has a pretty good mix of Transfer Modes.

There was a little bit of digging to get things set up exactly how I wanted. The Powerfolder guys are definitely not UI developers, but support was extremely helpful and their wiki documentation is great even if the search function on the wiki is not. :-)

We haven't gone to an inline solution for Exchange, SQL, and Sharepoint yet, but saving a backup of the databases to disk and having Powerfolder sync them is enough peace of for us.

This solution works well and the company loves it as it cost less than $100 (excluding man hours for research and setup) to implement.

It's surprising Powerfolder isn't more well known.

P.S. - sorry for the lack of links (LAN-Only Mode, Transfer Modes, etc). "new users can only post a maximum of one hyperlink"

0
1

As much as I hate to say it, the easiest, and fastest way to perform multi-site backups is with a good storage array. Both Dell/EqualLogic and HP/LeftHand have software built into their SAN products that will allow constant incremental backups across multiple SANs. They are quick and easy to set up, but may not be the cheapest solution.

1

Adding a new option to this running thread.

The software we started using is made by AppAssure, (Now acquired by Dell) Product name is called Replay.

it's designed for Windows servers doing disk to disk backup and there is a replication option that allows you to automatically copy the snapshots over to a remote site.

It includes automatic deduplication, automatic version rollup, and the replication is pretty efficient and can be scheduled to happen on off hours even if the backup snapshots are happening all day on 15 minute or hourly basis.

Only the changes get sent over the WAN not a full copy of the data and if you need to do a brand new full copy of data you can offload the initial backup to an external disk and ship that out to the remote site to be imported to save you from having to send a full backup over the WAN for the initial sync.

For backup disks a perfect companion is the Drobo B800i Iscsi san system. It's relatively cheap, takes commodity off the shelf SATA drives and has reasonable performance for doing backups (but not good enough for doing anything too heavy like VMWare ESX hosts or SQL data hosting) There is a reason that Drobo does not publish performance data on the B800i, it's pretty low end compared to something like an Equallogic PS san, or anything from the big SAN vendors like EMC, or HP. But it's great as the storage for a disk to disk system.

0

One thing to consider. How long are you going to keep snapshots/backups on disk? Tape is generally used for long term archival. If I need to restore an email that was deleted a year ago, I go to tape. Unless you're going to keep your disk around for that long without deleting, you're eliminating a large portion of your ability to restore data.

5
  • I agree. You need to consider the logevity of what you are backing up. Also if the backup is a drive attached to the system then it is available for attack. A tape - not so much unless you leave it in your car. See my post at serverfault.com/questions/22064/…
    – AudioDan
    Jun 17, 2009 at 1:26
  • @Kevin and @danbays: Our users work under the impression that if they delete an email and don't put in a help desk ticket right away, then its gone. That goes for files too. As far as retention of application data, we keep all data live that requires retention, then delete it forever. In other words, we don't have a need to archive data. Jun 18, 2009 at 12:06
  • I find that rather incredible that you have no need to restore a file from a version that could be a year old. It may not be a deleted item but could be for a variety of reasons such as legal discovery or simple auditing. Jun 18, 2009 at 12:38
  • @Kevin All legal requirements for data retention are handled via our main application which is developed in-house. All data that requires retention remains live and not archived. Retention of files and emails are not part of our legal retention requirements. Thanks for finding it incredible though. ;) Jun 18, 2009 at 13:50
  • We are considering keeping old hard drives ... access speed is MUCH faster, and 1TB drives now cost <$75 apiece.
    – Beep beep
    Jun 20, 2009 at 17:35
0

For speed, ease of use, and price you should really look at a target dedupe appliance like ExaGrid. Up and running very fast, with replication to a DR site. Completely eliminate tape. www.exagrid.com

1
  • Thanks but looking for a software solution. Jun 26, 2009 at 18:35
0

We have two servers A and B.

Server A Runs a primary node for SQL Server (Primary) IIS 6.0 Network Load Balancing (NLB) Replicated file service

Server B Secondary (Failover for Server A) node for SQL Server IIS 6.0 NLB Replicated file service

If A goes down then all of the data is on B and vice versa. The files are replicates via the replicaiton agent which takes care of our websites and this is updated whenever files chang ein either server. The Data is mirrored to the failover machine so if A goes down B will automatically assume the role of primary DB. With NLB the end user doesn't need to know which server they are hitting so this minimizes both downtime and the need to worry about a lot of configuration issues.

In addition to this we still take a backup of the DB regularily and the files are backuped offsite. If you're data is chanign a lot then you will probably have to consider some hardware medium for your backup but if the changes are failry small each day then you could consider an online backup service in that case.

Your data is your reputation and your business Whatever you do spend the money to ensure that your data is secure befause on the day when you loose it all or it's corrupted the cost of a robust backup solution now will seem like pennies then.

1
  • I don't need load balancing or database replication. Thanks. Jun 26, 2009 at 18:34
0

The first thing I would attempt to do in this environment is eliminate the servers at the smaller site. The feasibility of doing that depends on a lot of factors such as number of users, availability of bandwidth, type of data that would be going across the link, and so forth. I assume that with 5 servers we're talking a relatively small number of users, so investigating a Riverbed or equivalent solution (possibly backed up with a hardware print server device and using the router to handle DHCP and local DNS caching) may be economically more viable than maintaining servers/etc, would greatly simplify your setup, and would overall lead to a reduction in admin overhead, better service levels, and so forth.

1
  • Thanks for the suggestion. This smaller office of ours is eventually going to be a backup site for most of our operations. If our main office goes down due to natural disaster (we are in FL) then the other will carry on our business. For this reason, we are actually beefing up our infrastructure there. Jun 22, 2009 at 12:58
0

You should give a look at robocopy, or, if you need a gui, richcopy. Both tools are multithreaded, fast, efficient, and have lots of options for merging and syncing. You can use in conjunction with the osql backup database command (for DB backup) and exmerge (for brick-level exchange backup), and can create a simple scheduled task to automate.

1
  • Robocopy isn't going to do delta transfers. If one bit changes in a 2GB file then 2GB gets to cross the wire. Jun 24, 2009 at 20:50
0

This may seem slightly insane but I have resorted to using a version control system to push backup files to a single repository that I backup religiously. It is a spoke and hub model (not distributed like git) and I have found a multitude of clients. The really good thing about it is the audit trail and easy of getting the backups using the nearly ubiquitous clients for the version control system. De-duplication is something that has been a mainstay goal for version control systems and some of them do various checks and shallow copies when the same data arrives more than once.

0

I'd personnally use Unison to sync sites. It's well supported on windows, it synchronize only changed data blocks like rsync, it has an easy to use GUI, and is free both as speech and beer. I set it up for 16TB backups and it just works fine.

You could then simply replicate your VTL backups from each site to the other one, and that's about it.

2
  • How does Unison do for replicating NTFS ACL's? I'm not finding any documentation that indicates that it can replicate NTFS ACLs at all. Jun 24, 2009 at 18:26
  • Actually it doesn't (talking of similar tools, rdiff-backup does copy NTFS ACLs). It shouldn't be a problem in the OP case (replicationg VTL files), but you could also use robocopy to synchronise ACLs after the file replication itself.
    – wazoox
    Jun 24, 2009 at 19:56
0

If you haven't looked already, checkout Data Protection Manager(DPM).

http://www.microsoft.com/systemcenter/dataprotectionmanager/en/us/default.aspx

I am in the process of setting this up for internal use. I don't know all the ins/outs yet. I was unaware that Microsoft offered such a product until just recently.

I know it will do the delta diffs, but not sure about your replication needs.

2
  • CDP always comes at a high premium and we don't really need it. Thank you for posting. Jun 26, 2009 at 18:32
  • If you are going to virtualize as your question stated DPM is ridiculously inexpensive when purched with the rest of the management suite, as it's license is per host not per OSE.
    – Jim B
    Jun 26, 2009 at 18:57
0

Check out EVault Express Recovery Appliance

http://www.i365.com/disaster-recovery/index.html

It is a disk based system that does offsite as a service or you can use your own evault system. It's easy on the bandwidth and fast backups/restores.

I've used a few popular backup solutions and this one was by far the easiest to learn/use. The service was 10 times better as well.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .