Simple Distributed Disconnected way to sync a directory

Question

I want to start regularly backup my home directory on my ubuntu laptop, machine X. Suppose I have access to 2 different remote (linux) servers that I can backup to, machines A & B. Machine X will be the master, and should be synced to A and B. I could just regularly run rsync from X to A and then from X to B. That's all I need.

However I'm curious if there's a more bandwidth effecient, and hence faster way to do it. Assuming X is going to be on residential style broadband lines, and since I don't want to soak up the bandwidth, I would limit the transfer from X. A and B will be on all the time, however X, will not be, so I'd also like to reduce the amount of time that X is transfering, potentially allowing A and B to spend more time transfering. Also, X won't be connected all the time.

What's the best way to do this? rsync from X to A, then from A to B? Timing that right could be troublesome. I don't want to keep old files around, so if I was to rsync, then the --del option would be used. Could that mean something might get tranfered from A to B, then deleted from B, then transfered from A to B again? That's suboptimal.

I know there are fancy distributed filesystems like gluster, but I think that's overkill in this case, and might not fit with the disconnected nature.

Answer 1

There are clever ways to do this, and simple ones. I'd opt for simple whenever possible.

Could you do X -> A, and then A -> B?

There are also methods like Unison, but that seems to only go to one replica, and it is bi-directional which seems to not be as useful for a backup. However, if both A and B ran things on Unison, it seems like client X could then back up to A or B, and it would make it to the other nicely.

Answer 2

You want X to initiate the sync with A, which will then pull the data from X and when finished push to B. A script on A could just check for the existence of a file somewhere, that is uploaded to A by X when it is online and ready for a sync. When it is done pulling from X, it should push to B and when finished remove the file. Alternatively you could have almost the same script on both A and B, but when A is done pulling from X, it puts a file on B telling it to sync with A. If you had more than 2 machines to sync, this would be the way to go since then the sync can then just cascade through all of them (possibly in a distributed tree fashion rather than linear)

Answer 3

Would Dropbox work for you?

Answer 4

gibak is right for this, I think.

You can have the laptop order the servers to pull from each other, then push to whichever server; though that means the laptop has to wait and possibly stay online longer. You can add cronjobs so the server pull from each other. Or you could have the laptop push to A, and A push to B in a post-receive hook, which is simpler.

Answer 5

I think Michael Graff's solution is the best idea. rsync X -> A. Then A -> B.

If you want another suggestion that would cut down computationation time, you could look at xdelta (http://xdelta.org/). It essentially the same solution as the multi-hop rsync, with the benefit it could save you the time of having to calculate the delatas at each hop. If the files involved are large, this could become a material savings of time.

Jim C. http://www.kleobackup.net

Answer 6

I agree with naelyn, this sounds like a scenario for dropbox

As you say you are running Ubuntu. I thought Ubuntu One was supposed to provide this functionality too. I don't use it myself but might be worth a look.