Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a server with a LOT of small files -- many millions files, and over 1.5 TB of data. I need a decent backup strategy.

  • Any filesystem-based backup takes too long -- just enumerating which files need to be copied takes a day.

  • Acronis can do a disk image in 24 hours, but fails when it tries to do a differential backup the next day.

  • DFS-R won't replicate a volume with this many files.

I'm starting to look at Double Take, which seems to be able to do continuous replication. Are there other solutions that can do continuous replication at a block or sector level -- not file-by-file over a WAN?

Some details:

  • The files are split up into about 75,000 directories.

  • 99% of the daily change comes from adding new directories; existing files are rarely changed.

There's some other relevant discussion here.

share|improve this question
    
Is there any possibility of splitting the files into organized subdirectories or are they already split up? –  Bart Silverstrim Jan 3 '11 at 15:35
    
What is the change rate of the files, and what process does the changing? Any chance of obtaining a list only of changed files, and syncing those (with a monthly full sync during a maintenance window)? –  Matt Simmons Jan 3 '11 at 15:39
    
Bart, Matt -- answers added to the question above. –  Jesse Jan 3 '11 at 16:45
    
@Matt: just getting a list of edited files is extremely time-consuming. I was hoping some software that could keep track of recent modifications would be faster. –  Jesse Jan 3 '11 at 16:46
1  
Just to be clear, you want to do the replication to increase availability, not to implement the backup, right? –  hmallett Jan 3 '11 at 17:03

3 Answers 3

1.5 TB would take 30 minutes with http://www.exdupe.com/ :p

... given that your disks are fast enough (exdupe is so fast that it's IO bound, not CPU bound).

And I havn't reached any limits on file count yet. Had millions too.

Edit: Ah, you need a partition/sector based backup and not file system level? It can't do that... Maybe http:// www . drivesnapshot.de/en/ is worth a shot (had to add spaces because of spam protection). Does diff backups and shadow volume copy too (no reboot).

share|improve this answer
    
I've never seen exdupe before, but am testing it now. Very interesting. Good suggestion. –  ewwhite Jan 4 '11 at 6:17
    
The only reason I was avoiding filesystem backup was that so far, any attempt to do anything at the filesystem level with 10M+ files takes forever. If exdupe does millions of files in 30 minutes, filesystem is OK with me. Sadly, I'm writing to a USB drive, though. –  Jesse Jan 4 '11 at 13:44

Check out Shadowprotect. It's not continuous, but it can be set to do an incremental every 15 minutes. It's pretty awesome software. Add on the ImageManager enterprise portion and it also gives you some great replication abilities for offsite backups.

share|improve this answer
    
Thanks, just talked to them. They can't actually mirror the volume, but it does sound like they could do good things with disk images. –  Jesse Jan 3 '11 at 17:02

I've had great luck with Doubletake, despite the price. Their "move" product might fit the budget though...

See my answer to a similar question here.

share|improve this answer
1  
Looks like DT RecoverNow (formerly "Backup") is also a possibility, and much cheaper than DT Availability. –  Jesse Jan 10 '11 at 14:40
    
Are the target servers vmware or Hyper-V virtual machines? If so, it looks like the right answer for you! –  JakeRobinson Jan 12 '11 at 1:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.