Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are currently doing a complete remote backup of our VM's once a week. This takes the whole weekend, and I'd like to speed up the process some if possible.

We are backing up the VM's using Dedup, and the backup size is 300GB using a 128KB block size. Our small office connection can do 10mbit upstream at its best, and this would roughly translate to 3 days for 300GB.

Do you think 128kb is too large a block size for this application? Should it be lowered at the cost of system overhead and the benefit of smaller backup volume?

Can we do differential backups via rsync with Dedup, eliminating the need for complete backups?

Any and all advise/suggestions are welcome.

Thanks

share|improve this question
    
VMWare is a company not a product, could you provide details and change your tag please. Also provide details of what you're using to back them up and what dedupe software you're using. –  Chopper3 Feb 28 '11 at 16:32

2 Answers 2

up vote 1 down vote accepted

I assume you expect biggest gain from dedup to come from the OS part of VM images. In that case if your VMs aren't clones of each other, then I'd say that 128 KiB is too big block for optimal dedup. If your bottleneck is the network and more efficient deduplication would be helpful, I'd go way down. If you are deduping disk images, then optimum size would be the minimum VM OS-level unit of allocation. In Linux that's a 4KB block (by default) in ext3 and ext4 filesystems. With bigger block sizes pay attention to partitioning, you may have identical systems shifted by half the dedup-block-size because of different virtual disk layouts.

It's hard to give better answer with a rather vague question.

share|improve this answer
    
I appreciate the answer. With a smaller block size, there are more files to backup. If rsync is getting a hash from all the files, will this create significant overhead 4x as many files albeit smaller, to process? –  David Houde Mar 1 '11 at 16:41
1  
There are two scenarios: either you are CPU bound or I/O bound. If you are CPU bound, then the number of files will not make difference, and if you end up with less data due to better dedup, then the total time should drop. If you are I/O bound, then increasing the number of files can mean more seeks and lower performance. You will still have less data to hash, so the net result may be improvement or loss of performance. Only testing will tell. –  Paweł Brodacki Mar 2 '11 at 10:38

VMware ESX4+ has changed block tracking, which allows backup software to determine which blocks to back up, rather than the entire disk. The backup software can then roll the changes in to the first full backup, essentially eliminating the need for doing a full every week.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.