Rotating Backups with RSync and CP - Excludes not working perfectly

Question

My script looks essentially like this:

cp -al $yesterdays_snapshot $todays_snapshot
rsync -vadlH --exclude-from excludes.rsyc $source $todays_snapshot

This works great except for one problem:

If yesterday I had no excludes, but then today I realize /directory/cache should be excluded, rsync stops looking at the cache, but the cp continues to copy it forward each day. After a while this means that the snapshots will have the contentsof /directory/cache from a year ago, which isn't the desired behavior. We don't want /directory/cache in the snapshot at all.

It would be better if rsync deleted /directory/cache from the newest snapshot.

Alternatively, it would be good if cp knew not to copy any files listed in excludes.rsync

I can write a function that manually deletes any excludes files from todays_snapshot, but it'll take a little hacking, and this feels like the sort of problem that has a ready-made answer out there.

Any suggestions?

Answer 1

What you are trying to do with cp and rsync is create sets of backups with identical files hard-linked to each other.

This can be very useful, which is why the rsync developers already though of this and included native functionality directly within rsync to accomplish this.

This functionality is provided by the --link-dest option.

Basically your rsync command will look like this rsync -va sourcepath newsnapshotpath --link-dest previoussnapshot. Rsync will hardlink any identical files to the previous snapshot, and copy down new files as needed. Files no longer present on the source will not be linked/copied.

In my answer to another question I showed a series of commands you would could issue to see exactly how this all works.

For the record, if you want to avoid re-inventing the wheel you might also want to take a strong look at dirvish. Which is a backup specifically designed to take advantage of the --link-dest feature.