Handling renamed files or directories in rsync

Question

I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but am not liking the fact that its copying renamed files or directories as a new file or directory, keeping the files not in sync. I have also set a bandwidth limit of 1MB since this will run during business work hours. Here’s my script:

rsync -zvru --bwlimit=1024  /mymounts/test1/ /mymounts/test2

How can I keep files and directories in sync if someone renames something, with still only copying new or modified files? Here are the files in question

ls "/mymounts/test1/some stuff"
new directory  newfile1.txt  newfile3.txt  renamedFile.txt

ls "/mymounts/test2/some stuff"
new directory  newfile1.txt  newfile2.txt  newfile3.txt  renamedFile.txt

Or would there be a way to even move the renamed files to another directory say: /mymounts/VerControl?

Answer 1

You may want to look at -y | --fuzzy rsync option. Other than that, rsync has no way of tracking renames, so you'll end up transferring renamed file.

From rsync manpage:

   -y, --fuzzy
          This option tells rsync that it should look for a basis file for
          any  destination  file  that  is missing.  The current algorithm
          looks in the same directory as the destination file for either a
          file  that  has  an identical size and modified-time, or a simi-
          larly-named file.  If found, rsync uses the fuzzy basis file  to
          try to speed up the transfer.

Answer 2

You can handle moved and renamed files with rsync if the filesystems on the source and target directory have support for hard links. The idea is to let rsync reconstruct hard links before real transfer. You can find a brilliant explanation here.

We ended up with a simple solution that create an hidden tree of hard links inside the source/target directory, the basic script could be like this:

# Name of hidden directory
Shadow=".rsync_shadow"

# do real sync
rsync -ahHv --stats --no-inc-recursive --delete --delete-after "$Source"/ "$Target"

# update/create hidden dir of hard links in source
rsync -a --delete --link-dest="$Source" --exclude="/$Shadow" "$Source"/ "$Source/$Shadow"

# update/create hidden dir of hard links in target
rsync -a --delete --link-dest="$Target" --exclude="/$Shadow" "$Target"/ "$Target/$Shadow"

I have an example script on GitHub. But I advise you to do a large amount of testing before use this method on production.

Answer 3

Here is a tool that is designed to work before rsync is run: rsync-sidekick

This propagates following changes from source directory to destination directory (or any combination of below):

1. Change in file modification timestamp
2. Rename of file/directory
3. Moving a file from one directory to another

Using this tool saves you from horrible situations of renaming a directory leading to transferring GBs of files.

Disclaimer: I'm author of above tool

Answer 4

While this is an old post, these were the first discussion I found when searching, so I thought I'd share a solution. Since about 2010, there's been patches to rsync that detect renames. One patch that worked for me is here.

So, if you can patch your rsync in your environment, you can use rsync --detect-renamed or --detect-renamed-lax.

Answer 5

You can't.

Rsync has three modes,

• Copy all files
• Copy existing (modified/non-modified) files -- on this you have many options
• Copy non-existing files.

However these three modes have two subcategories,

• Exclude on
• Include on

Rsync doesn't track which files are renamed, it has no state. Consider instead copying all files, and excluding the ones you don't want. You can't have a shifting-whitelist, you can have a blacklist.

rsync [..stuff..] --exclude 'lib/'

Answer 6

As far as I know, rsync cannot recognize renaming of files The new files will have to transferred again. See artyom's answer.

To delete the gone files, make sure to use --delete option.

Also, for mirroring, I would recommend you to use -a (archive), which aliases some nice options.

Have a look at the man 1 rsync for details.