Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but am not liking the fact that its copying renamed files or directories as a new file or directory, keeping the files not in sync. I have also set a bandwidth limit of 1MB since this will run during business work hours. Here’s my script:

rsync -zvru --bwlimit=1024  /mymounts/test1/ /mymounts/test2

How can I keep files and directories in sync if someone renames something, with still only copying new or modified files? Here are the files in question

ls "/mymounts/test1/some stuff"
new directory  newfile1.txt  newfile3.txt  renamedFile.txt

ls "/mymounts/test2/some stuff"
new directory  newfile1.txt  newfile2.txt  newfile3.txt  renamedFile.txt

Or would there be a way to even move the renamed files to another directory say: /mymounts/VerControl?

share|improve this question

4 Answers 4

You may want to look at -y | --fuzzy rsync option. Other than that, rsync has no way of tracking renames, so you'll end up transferring renamed file.

From rsync manpage:

   -y, --fuzzy
          This option tells rsync that it should look for a basis file for
          any  destination  file  that  is missing.  The current algorithm
          looks in the same directory as the destination file for either a
          file  that  has  an identical size and modified-time, or a simi-
          larly-named file.  If found, rsync uses the fuzzy basis file  to
          try to speed up the transfer.
share|improve this answer

You can handle moved and renamed files with rsync if the filesystems on the source and target directory have support for hard links. The idea is to let rsync reconstruct hard links before real transfer. You can find a brilliant explanation here.

We ended up with a simple solution that create an hidden tree of hard links inside the source/target directory, the basic script could be like this:

# Name of hidden directory
Shadow=".rsync_shadow"

# do real sync
rsync -ahHv --stats --no-inc-recursive --delete --delete-after "$Source"/ "$Target"

# update/create hidden dir of hard links in source
rsync -a --delete --link-dest="$Source" --exclude="/$Shadow" "$Source"/ "$Source/$Shadow"

# update/create hidden dir of hard links in target
rsync -a --delete --link-dest="$Target" --exclude="/$Shadow" "$Target"/ "$Target/$Shadow"

I have an example script on GitHub. But I advise you to do a large amount of testing before use this method on production.

share|improve this answer
    
This is a fantastic answer! Welcome to Server Fault, dparoli! –  JakeGould May 17 at 15:11
    
Also the “brilliant explanation” you link to also refers to this great basic rsync article, FWIW. everythinglinux.org/rsync –  JakeGould May 17 at 15:17

As far as I know, rsync cannot recognize renaming of files The new files will have to transferred again. See artyom's answer.

To delete the gone files, make sure to use --delete option.

Also, for mirroring, I would recommend you to use -a (archive), which aliases some nice options.

Have a look at the man 1 rsync for details.

share|improve this answer
    
thanks that will work --delete /mymounts/test1/ /mymounts/test2/ –  jmituzas Mar 19 '13 at 15:46

You can't.

Rsync has three modes,

  • Copy all files
  • Copy existing (modified/non-modified) files -- on this you have many options
  • Copy non-existing files.

However these three modes have two subcategories,

  • Exclude on
  • Include on

Rsync doesn't track which files are renamed, it has no state. Consider instead copying all files, and excluding the ones you don't want. You can't have a shifting-whitelist, you can have a blacklist.

rsync [..stuff..] --exclude 'lib/'
share|improve this answer
    
gotcha thanks for the input will have to live with the renamed files then. –  jmituzas Mar 19 '13 at 15:31
    
You can have it delete the files that don't exist, that's perfectly fine. But as a function of copy, if you rename on source, it will copy the renamed file. –  Evan Carroll Mar 19 '13 at 15:34
    
I can keep the versions with -b and set suffix=date +%Y%m%d%k%M%S anyway I can get suffix=date +%Y%m%d%k%M%S.ORIGINALSUFFIX? –  jmituzas Mar 19 '13 at 20:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.