1

in our small sized company we use Vmware. We backup the VM with Veeam. Resulting files are in order of 200-300 gigabyte per VM, for a total of 1/1.2 terabyte of data. I use robocopy to copy VM files backuped to external hard drive. I would like to know if Robocopy use something like rsync to copy only changed 'blocks' of data instead the entirely file . Thanks

1

Robocopy doesn't use any method to transmit only changed blocks of datas.

If you have to transmit your files thru a slow network, just use rsync.
If you have to transmit your files thru a "reasonable speed" network, no algorithm can speedup the process, because rsync (for example) have to read the entire file on client and on server to find duplicate contents. So if your network is Gigabit, or if you do a local copy, you just copy the files and voila. You can't get something better, execpted if supported by the filesystem (ala ZFS, but only works on the same partition).

4

@Gregory MOUSSAT:

no algorithm can speedup the process, because rsync (for example) have to read the entire file on client and on server to find duplicate contents.

... I think that affects only transfers where just one single rsync works, eg. copying between a share and a local disk. Here, that one-and-only rsync of course needs to "fetch" the whole file from the share to the client to do its comparing/copying.

However, if you use two different rsyncs on both the source and the target machine, you will in fact get a huge speed up! Both separated rsyncs build their checksums of the file in question locally (without transferring any bit of it!), compare these checksums over the network and then decide which (changed) parts of the huge file to transfer.

That can be achieved by using a rsync daemon on the fileserver (remember to open its rsyncd port in the firewall), or by using ssh as a transport medium between the two machines. The latter of course requires you to have local login rights to the remote machine.

Cheers

  • If your network is able to transfert files at the same speed than the disks can be read, then you can use whatever amount of rsync, they can't be faster than disks. So they can't be faster than the network. Rsync is only handy when transmit speed is slower than read/write speed. – Gregory MOUSSAT Mar 21 '12 at 8:02
  • 3
    ... that was not the question and it isn't relevant. In the case of two separated rsyncs, these communicate just the changes over the network, not the whole file. Imagine a 500 GByte file which has 2 bytes changed at the end. Two communicating rsync processes would only transfer one single 512 byte block with the changed 2 bytes over the wire, and the receiving rsync would just write this changed block to disk. This smart behavior would even speed up local disk2disk copies and does not depend on disk speed versus network speed ;-) – Christian Mar 22 '12 at 8:51
  • 5 years later - yes but how is rsync going to know those two bytes have changed without reading the whole file? – tomfanning Dec 4 '17 at 16:51
  • I admit that the transfer speed of course can never exceed the speed of the slowest of both disks. Even with two rsync's at both ends, both have to locally read in the whole file to build their respective checksums for comparison. That's right! Perhaps I should have said the "transfer volume" over the wire will be much smaller, and in the case of fast disks, this results in less transfer time. In any way, the network is less loaded with rsync in case of only a few bytes changed... – Christian Mar 20 '18 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.