Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We backup a set of virtual machines to an external USB drive using rsync -a. The source directory is 145G as reported by du -sh, but the target is reporting 181G.

The two file systems are ext3 and the block size is the same, so can someone explain what the discrepancy is?

share|improve this question

4 Answers 4

up vote 1 down vote accepted

As Dennis mentioned, it seems to be a sparse file issue. An example of that can be:

$ dd if=/dev/zero of=sparse.txt count=0 seek=1000
0+0 records in

$ du sparse.txt
0   sparse.txt

$ ls -l sparse.txt
-rw-r--r-- 1 user user 512000 2010-03-22 11:54 sparse.txt

As you can see du reports how many blocks are actually used, while ls shows how big the file is supposed to be.

share|improve this answer
    
OK, so if the source files are created as 'thin' backups using ghettoVCB then they'll be sparse? I understand. Thanks. –  nickd Mar 22 '10 at 12:31
    
I think VMWare images are sparse, so you are seeing this issue. The original files are sparse, but the backup is not. –  Dan Andreatta Mar 22 '10 at 13:55
-S, --sparse
              Try to handle sparse files efficiently  so  they  take  up  less
              space on the destination.  Conflicts with --inplace because it's
              not possible to overwrite data in a sparse fashion.

              NOTE: Don't use this option when the destination  is  a  Solaris
              “tmpfs”  filesystem.  It  doesn't seem to handle seeks over null
              regions correctly and ends up corrupting the files.
share|improve this answer
    
I'm sorry, but I don't understand. Are you suggesting that the source files are sparse? The destination copy is bigger. –  nickd Mar 22 '10 at 11:51
    
Sparse files are smaller than they claim to be. It's copying the sparse files as if they were actually sized as allocated. It's including the empty space in the destination, thus it does get bigger. –  Dennis Williamson Mar 22 '10 at 14:33

Others already told about sparse files, but there is another thing: hard links. Hard links – multiple names for the file (and space on the disk) are often used on system partitions (e.g. for multiple shell commands implemented in the same binary) and they are not handled specially by rsync with the '-a' option only. So, e.g. a file with four hard links will be stored as four separate files.

Try using rsync -aH.

share|improve this answer
    
This is the case in BusyBox or ESXi, correct? –  ewwhite Mar 22 '10 at 12:42

With the rsync script, are you deleting the items that exist on the destination and are no longer on the target? If you're looking to have an identical copy on both sides, you would need a "--delete" flag in your rsync routine.

"rsync -a --delete /source/ /destination/"

You can also inject "-n and -P" into the string to provide a dry-run and progress indicator, respectively, to show you what would happen with the "--delete" option.

share|improve this answer
    
Ah, I didn't notice that these were VM images. If they are thin-provisioned, then you're dealing with sparse files. Use the -S rsync flag. –  ewwhite Mar 22 '10 at 12:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.