How to determine the best byte size for the dd command

Question

I know that doing a dd if=/dev/hda of=/dev/hdb does a deep hard drive copy. I've heard that people have been able to speed up the process by increasing the number of bytes that are read and written at a time (512) with the "bs" option. People have suggested that the optimal byte size is due to sector size. I personally think it would have something to do with the amount of cache that the hard drive has.

My question is:

What determines the ideal byte size for copying from a hard drive?

and

Why does that determine the ideal byte size?

Edit:

ok, so I read the man page for dd again and it says

"bs=BYTES Block size, both read and write BYTES bytes at a time. This overrides ibs and obs."

So the dd command is asking for the block size in bytes that is used on the hard drive? Is this the same as the "Allocation unit size" under the windows format options? Indeed these are a power of 2 and the smallest option happens to be 512. So in order to optimize the speed... you can either remember what block size you used when you formatted the drive or find a tool that tells you. I still don't fully understand what formatting a drive with different block sizes does and why dd is able to copy faster if it knows the block size.

Thanks to Chris S for mentioning that block size is 2 to the power of some number. That is what reminded me of the "Allocation unit size" thing.

Answer 1

As Chris S says the optimum block size is hardware dependent. In my experience it is always greater than the default 512 bytes. If your working with raw devices then the overlying file system geometry will have no effect. I've used the script below to help 'optimise' the block size of dd.

#!/bin/bash
#
#create a file to work with
#
echo "creating a file to work with"
dd if=/dev/zero of=/var/tmp/infile count=1175000

for bs in  1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1M 2M 4M 8M 

do
        echo "Testing block size  = $bs"
        dd if=/var/tmp/infile of=/var/tmp/outfile bs=$bs
        echo ""
done
rm /var/tmp/infile /var/tmp/outfile

Answer 2

Unfortunately the perfect size will depend on your system bus, hard drive controller, the particular drive itself, and the drivers for each of those. The only way to find the perfect size is to keep trying different sizes. Fair warning that some devices only support one block size, though this is rare, and usually drivers make up the difference anyway.

I find that block size of 2^15 or 2^16 work best for my WDC (8mb cache) SATA drives connected to a Adaptec SAS RAID controller, 4x PCIe, 64-bit FreeBSD 8.0-STABLE. But for my cheap old thumb drive, sizes of 2^10 seem to be fastest.

The "perfect size" is almost always a power of two.