0

I ran an rsync to backup one of our production servers. I put the production server in readonly mode so that no additional data could be added or modified. I then did a recursive rsync with archive (-a) to backup the production servers data directory to a remote backup, which is configured identical to the production server.

After days passed, what I found was that the backup (destination) server ended up having about 100MB more of data. How could this be -- Is that normal? Any idea how to track this down? Right now I'm doing a ls -laR out to a file on both the production and backup server. I'll then try to diff the files to see if there is any differences. Any other tips?

2
  • 1
    Did you also ask to have files which were deleted on the production server, deleted on the remote backup server too? Dec 30, 2014 at 23:26
  • Show the full rsync command line.
    – ewwhite
    Dec 31, 2014 at 1:49

5 Answers 5

2

I would not be overly concerned. Afterall, there might be sparsely allocated files. When these are copied via logical file access that rsync uses, the unallocated space is expanded read as a zero filled area and thus you get more space. The file sizes would still be the same on both the source and destination.

BTW, instead of a diff I might do compare file checksums via md5sum or sha1sum.

2

There are a few possible reasons for a copy to take up a different amount of space than the original:

  • Sparse files. If the copying does not take advantage of sparse files, the copy may take up more space than the original. If the copying does take advantage of sparse files, the copy may take up less space than the original. In case of rsync there are two possible options (controlled with the --sparse option), either the destination files will be sparse or they will not. A normal cp command has three options: make all the copies sparse, make none of the copies sparse, make the copy sparse if the source was.
  • File system slack. If the source and destination are on different file systems (even if they are using the same driver but different block sizes), then the storage requirements may differ.
  • Meta data. Over time developers come up with more and more kind of metadata that can be stored along with files. Not all copying tools can keep up with the introduction of new kinds of metadata, and not copying all the metadata can cause the copy to take up less space.
  • Directory overhead. Size of directories can depend on the order in which files are added and removed. For example ext2,3,4 file systems do not release directory space when files are deleted. This could cause the copy to take up less space than the original.
1

One other possibility is that some files have been deleted from the production server, and rsync hasn't been told to delete the files from the backup (--delete -option).

0

By default rsync will not remove files from the destination when it finds that the file has been removed from the source, so that's the likely source of the size difference. You can define this behaviour with the --delete flag, and also specify how to back up deleted/changed files on the destination with the --backup and --backup-dir flags.

Here is an excerpt from an old nightly backup script that used this:

rootdir='/usr/local/backup/'
cmd_frame='rsync -ave ssh --delete --backup --backup-dir=%s %s %s'
logfile=${rootdir}logs/`date +%s.log`
backup_root=${rootdir}copy/
diff_root=${rootdir}diffs/`date '+%Y/%m/%d/'`

sources=''
for domain in `cat ${rootdir}backup_list.txt`; do
    sources=`printf '%s user@host:/home/user/%s ' "$sources" "$domain"`
done

`printf "$cmd_frame\n" "$diff_root" "$sources" "$backup_root"` > $logfile

The most recent backup lives in copy/ with deleted/modified files being backed up under the respective diffs/year/month/day/ folder, plus the file's full path.

-1

If you are using different OSs on the backup/target machines yes there might be a difference. Same file is larger on linux than windows because of line endings and that would make a lot of sense if you have a lot of text files.

Another scenario might be that some OS might be using power of 10 instead of power of 2 when listing filesize e.g. 2^10=1024 which is definitely not 10^3=1000

This one is less likely but here goes...make sure you are not looking at size on disk if you have different OSes e.g. FAT, NTFS, exFAT are using clusters as a block unit which is completely different from ext(2,3,4)

1
  • rsync is not going to change the line-endings of your text files, regardless of what each operating system prefers.
    – fukawi2
    Dec 30, 2014 at 23:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .