0

I have several large directories containing hundreds of GB of files that I'd like to keep synchronized between home at work. Rsync is the obvious choice for transferring new files (both machines are Linux) but the bandwidth usage is a problem. Is there a way to use rsync to store the deltas in a directory (on a portable drive) and then use those deltas on the other side for synchronization?

3
  • 2
    Does your employer know about this?
    – EEAA
    Apr 7, 2011 at 16:23
  • Is bandwidth still an issue after the initial sync is done? (i.e. are we talking about files that have gigabytes of changes/deltas a day?) -- If not you can do the initial sync using a portable HD, then do maintenance using rsync...
    – voretaq7
    Apr 7, 2011 at 16:48
  • Yes, it's for work and personal purposes. Moving virtual machines, not just tunes. Bandwidth is still an issue. Syncs aren't daily, but when they're done, they're many GBs of deltas. I've already done the initial sync, but I want to avoid chewing up bandwidth at home.
    – Mark Rose
    Apr 7, 2011 at 17:03

3 Answers 3

2

You can rsync from one disk location to another, but as far as I know there's no way to store the deltas rsync would use (because there's no way to guarantee that the files are still identical from one box to the next).
You COULD do what you're asking with a 500GB or 1TB external hard drive (rsync to that, take it home, rsync there, etc.) - and you get the benefit of an additional backup device.

Alternatively you can use bsdiff & bspatch to create binary diff files, but this requires you to have the "original" files laying around (doubles your storage requirement for each file - albeit briefly & without needing an external HD).

2
  • I've already outgrown the capacity of a 250 GB drive. I guess I could go with something bigger. Bsdiff & bspatch look interesting and might be a possible solution.
    – Mark Rose
    Apr 7, 2011 at 17:05
  • This (the external HD) is what I would do -- it gives you an extra backup (so your files are stored in 3 locations instead of just 2, and who doesn't love the added safety of increased redundancy?), and you avoid that nasty headache of how to handle all that bandwidth needed. Also consider that rsync does have the option to compress the diffs it sends, which reduces your bandwidth overhead, but it sounds like that still might not be enough of a reduction for you.
    – Kromey
    Apr 7, 2011 at 17:12
1

How fast is the data changing? Do you need bi-directional sync? Rsync is quite good at optimizing the search for differences, so the amount of network I/O is usually much smaller than the amount of disk I/O needed to reconcile the system - i.e. the disk I/O is more commonly the limiting factor on the speed at which datasets can be synchronized.

What is it you are synchronizing? If it's just smallish files, and you know when the systems were last synced, you could just pick up on the files modified since that time and rsync (or copy) those files:

find /stuff/to/mirror -type f -cnewer /var/sync/lastrun >listofnew
touch /var/sync/lastrun
# then rsync the files using xargs or copy them onto a drive

If its large database files - that's a different story - it might be better to use application level processing (e.g. mysql's replication).

Another approach would be to use a network file system (NFS, Samba etc) on top of a VPN and use an overlay filesystem to batch up the changes.

Or you could use something like AFS.

I still think that standard rsync might be the best solution though.

1
  • Sometimes the backup is the same day to day. Other it's several 10's of GBs of changes. Today it will be 35+ GB. Bi-direction sync is needed (I rsync both ways, depending on where the updates are).
    – Mark Rose
    Apr 7, 2011 at 17:06
0

rsync won't do bidirectional syncing, as far as I know. But unison (Debian package, original homepage) can. Do bidirectional syncing between home and a removable drive (or several drives, if you divide the folders amongst them), and then bidirectional between the removable and work drive.

If you change a particular file on both ends simultaneously, you'll still have to figure out what to do. But if changes are one-way each time, it should be bulletproof. You wouldn't have to figure out which direction the changes should propagate if things worked as designed.

Then again, I have to wonder if rsyncing entire VMs is the real solution. Can VMs at work and at home be put under a common configuration management system, use the same version control repositories for code checkouts, etc.? Can the VMs just be stored on an external drive and go with you to home and work as needed?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .