sporadically, rsync takes exceptionally long

Question

We do backups with rsync like this:

rsync -axH --inplace --delete --delete-excluded \
--exclude-from=excludes --stats \
--link-dest="${previous?}" "${source?}"/ "${dest?}"/"${stamp?}"

$previous points to the previous backup, so that unchanged files will be created using hardlinks. The destination filesystem $dest is on an external USB hard drive with nothing else on it than the collection of backups.

This method is amazingly fast in most of the cases. On the test system, each backup is about 200 GB and contains some big maildirs - still the whole rsync (provided that not much was changed since the last run) takes only around a minute.

However, in rare cases, maybe every 100 runs on average, it takes very long, around 20 minutes or longer. The rsync statistics show nothing unusual. The host system shows no unusual activity during such runs. Nothing exciting in syslog.

It appears to be worse on some filesystems (for $dest) than on others. The above figures are for EXT4. On JFS for example, the normal runs take about 3 minutes and the exceptional runs are less severe, but still a problem for us.

A look at rsync's debug output reveals that during the long runs, certain (large) files are found to be not uptodate, although they have not been changed on the sender. No hardlinks are created for those files, as a look at their inode reveals. But rsync's statistics do not show more transferred bytes than usual, and from observing the hard disk activity LEDs, only the destination drive is working in those cases. Are those files copied on the destination from one directory to the other? This is turning out not only to be a perfomance problem, but could lead to unnecessary space consumption, too.

In case it matters: directly before the backup, the oldest of the existing backups is deleted using:

rsync -a --delete empty/ "${dest?}"/"${old?}"

where 'empty' is an empty directory. This is much faster than 'rm -fr'.

Could anyone please offer possible explanations for this and perhaps a cure?

Using rsync version 3.1.0 protocol version 31.

Answer 1

Short answer: the culprit was the way we deleted old backup directories, namely rsyncing an empty directory. Now we use:

find "${old?}" -delete

This is also fast and avoids the problem.

Longer answer: in fact, the runs which took exceptionally long occurred absolutely deterministic. We always keep a number of, say n, backups and delete the oldest one before performing a new. Every (n+1)th backup took a long time. It appears that by deleting an old backup with rsync, part of it is somehow invalidated for the --link-dest operation, so that some files are not hardlinked but copied (apparently copied from the destination filesystem itself). This copy procedure starts a new "period", which comes to an end when the first backup of it is deleted, which happens after n runs. This is most probably due to a bug in rsync or the kernel, but I will not investigate further.