1

We are currently doing a complete remote backup of our VM's once a week. This takes the whole weekend, and I'd like to speed up the process some if possible.

We are backing up the VM's using Dedup, and the backup size is 300GB using a 128KB block size. Our small office connection can do 10mbit upstream at its best, and this would roughly translate to 3 days for 300GB.

Do you think 128kb is too large a block size for this application? Should it be lowered at the cost of system overhead and the benefit of smaller backup volume?

Can we do differential backups via rsync with Dedup, eliminating the need for complete backups?

Any and all advise/suggestions are welcome.

Thanks

1
  • VMWare is a company not a product, could you provide details and change your tag please. Also provide details of what you're using to back them up and what dedupe software you're using.
    – Chopper3
    Feb 28, 2011 at 16:32

2 Answers 2

1

I assume you expect biggest gain from dedup to come from the OS part of VM images. In that case if your VMs aren't clones of each other, then I'd say that 128 KiB is too big block for optimal dedup. If your bottleneck is the network and more efficient deduplication would be helpful, I'd go way down. If you are deduping disk images, then optimum size would be the minimum VM OS-level unit of allocation. In Linux that's a 4KB block (by default) in ext3 and ext4 filesystems. With bigger block sizes pay attention to partitioning, you may have identical systems shifted by half the dedup-block-size because of different virtual disk layouts.

It's hard to give better answer with a rather vague question.

2
  • I appreciate the answer. With a smaller block size, there are more files to backup. If rsync is getting a hash from all the files, will this create significant overhead 4x as many files albeit smaller, to process? Mar 1, 2011 at 16:41
  • 1
    There are two scenarios: either you are CPU bound or I/O bound. If you are CPU bound, then the number of files will not make difference, and if you end up with less data due to better dedup, then the total time should drop. If you are I/O bound, then increasing the number of files can mean more seeks and lower performance. You will still have less data to hash, so the net result may be improvement or loss of performance. Only testing will tell. Mar 2, 2011 at 10:38
1

VMware ESX4+ has changed block tracking, which allows backup software to determine which blocks to back up, rather than the entire disk. The backup software can then roll the changes in to the first full backup, essentially eliminating the need for doing a full every week.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .