0

I am using rsync to transfer/update 100s of TB from one fileserver to another, millions of files. Many of those files are quite similar (but not identical).

I was wondering if there's a way to benefit from these similarities by not transferring chunks of data that already have been transferred identically for other files. Kind of a deduplication, but not on filesystem level but during data transfer.

As far as I can see rsync is looking at each file separately to transfer only changed chunks of data but never takes into account data from other files.

By caching and reusing transferred chunks rsync could safe quite a lot of network bandwidth.

So is there any way to achieve that?

2
  • welcome, Questions seeking installation, configuration or diagnostic help must include the desired end state, the specific problem or error, sufficient information about the configuration and environment to reproduce it, and attempted solutions. Questions without a clear problem statement are not useful to other readers and are unlikely to get good answers. Furthermore, Product suggestions are off topic
    – djdomi
    Oct 17, 2023 at 14:40
  • With that kind of volume, why is filesystem level replication/shipping not possible? Oct 18, 2023 at 5:35

1 Answer 1

1

Answering myself: It looks like --fuzzy is just something similar:

This option tells rsync that it should look for a basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .