6

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync calculates checksums of files. Surely if it sees 2 files with the same checksum it can just copy the first file, then do a local copy on the remote host for the 2nd file? That would make it faster, no?

On a similar note, doesn't rsync hash all the remote files before copying? If it saw a different file with the same hash as a file that was to transfered, it could do a local copy on the remote host.

Does rsync support this sort of thing? Is there some way to turn it on? Is there a tool similar to rsync that will do this sort of 'hash based' local copies?

Improve this question

6 Answers 6

Reset to default
3

On a similar note, doesn't rsync hash all the remote files before copying? If it saw a different file with the same hash as a file that was to transfered, it could do a local copy on the remote host.

no, but Unison does.

Improve this answer
1

I don't believe that there is any product ready to do that today as a single software package. There are WAN optimizers that will do this for you and then you still run RSYNC.

What you are looking for is a deduplication / reduplication process. Ideally you would not actually want to redupe but to dedupe locally and then do the RSYNC. That way you are only copying links rather than expanding the files on the other side.

Improve this answer
1

If both your files are truly identical, then you would have to rsync one of the files, and then make a local copy separately on the remote side, outside of the rsync process.

If you have two identical files, then rsync will try to copy them both - the filename, rather than the checksum, is the most important identifier. (I've just tested this on my laptop!)

It would be interesting to wrap rsync to do some of the stuff you mention though...

These are worth a read:

http://en.wikipedia.org/wiki/Rsync

http://samba.anu.edu.au/rsync/tech_report/

Improve this answer
1

git would do what you describe

Improve this answer
1

The --fuzzy option might do what you are looking for:

From the man page:

-y, --fuzzy
This option tells rsync that it should look for a basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.

If it doesn't someone with relevant programming skill should be able to tweak the option to behave as you describe quite easily - though that is probably not something you would want to be getting involved in!

Improve this answer
0

probably not what you want, but the EMC avamar can do that... it builds a checksum list and compares with the local one, and transfers only files not already in the local backup (even if the file comes from other computer)... it does deduplication before the copy and its very useful on similar machines

but this is for backups only, not a plain copies... and avamar is not cheap :)

the closest open source alternative to avamar that i know is backuppc... it still uses rsync and do only the deduplication on already transferred files

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .