I have several large directories containing hundreds of GB of files that I'd like to keep synchronized between home at work. Rsync is the obvious choice for transferring new files (both machines are Linux) but the bandwidth usage is a problem. Is there a way to use rsync to store the deltas in a directory (on a portable drive) and then use those deltas on the other side for synchronization?
You can rsync from one disk location to another, but as far as I know there's no way to store the deltas rsync would use (because there's no way to guarantee that the files are still identical from one box to the next).
Alternatively you can use bsdiff & bspatch to create binary diff files, but this requires you to have the "original" files laying around (doubles your storage requirement for each file - albeit briefly & without needing an external HD).
How fast is the data changing? Do you need bi-directional sync? Rsync is quite good at optimizing the search for differences, so the amount of network I/O is usually much smaller than the amount of disk I/O needed to reconcile the system - i.e. the disk I/O is more commonly the limiting factor on the speed at which datasets can be synchronized.
What is it you are synchronizing? If it's just smallish files, and you know when the systems were last synced, you could just pick up on the files modified since that time and rsync (or copy) those files:
If its large database files - that's a different story - it might be better to use application level processing (e.g. mysql's replication).
Another approach would be to use a network file system (NFS, Samba etc) on top of a VPN and use an overlay filesystem to batch up the changes.
Or you could use something like AFS.
I still think that standard rsync might be the best solution though.
If you change a particular file on both ends simultaneously, you'll still have to figure out what to do. But if changes are one-way each time, it should be bulletproof. You wouldn't have to figure out which direction the changes should propagate if things worked as designed.
Then again, I have to wonder if rsyncing entire VMs is the real solution. Can VMs at work and at home be put under a common configuration management system, use the same version control repositories for code checkouts, etc.? Can the VMs just be stored on an external drive and go with you to home and work as needed?