Simple, decentralized, automatically synced, local cached file share for linux server

Question

I wonder if someone could recommend me some solution to sharing files between a small amount of linux VPS (virtual private servers).

Basically I have a bunch of linux VPS to manage, and I want to share files between them. The typical use cases are to share data files, precompiled libraries or binaries, etc, therefore the write frequency is very low, most of which are just adding new files instead of modifying existing ones. Usually I use my own desktop machine to generate the data and build the binaries, and publish to the machines.

The requirements are:

1. Simple. It doesn't have to be complicated technology that is designed for large clusters. I only have a small amount of machines, say 10-20.
2. Data are replicated and stored on all machines. The VPS are not close to each other, so the network connection is slow, therefore all data should be replicated and locally stored on all machines.
3. Decentralized. Machines can be down or removed at anytime due to I forgot to pay timely. So machines shouldn't need to keep connection to a centralized server, but talk to each other. Of couse, when a machine is first setup, it knows a list of machines, but after that they should just talk to each other and maintain a list of available hosts. My desktop doesn't have a fixed external IP so the desktop should just be used to push out data, not to be fetched by those VPS servers.
4. When I publish new data from my desktop, it should just send out roughly one copy of data and after that machine should fetch from each other.
5. (Optional feature but strongly prefered to have) the data directory appears as a vfs and mounted to certain directory. This makes it a lot easier to use.

One thing I considered was some fuse-based file system backed by git. There are a bunch of git-fs on github or code.google.com but none of them looks mature and reliable. Also, on each machine it still needs to maintain a list of available machines to talk to, which is not automatic and therefore painful.

Another thing is to use some sort of P2P file sharing software, but is there any that makes files organized? For example, when I publish the files, I should also be able to specify which directories the files should be placed. This should be done automatically, not I login to each machine and manually move them after P2P downloaded the file.

For most distributed files systems, they are designed for clusters where machines are close to each other. For HDFS it requires centralized name node, so I can't use it as well.

Any thoughts? Thanks.

Answer 1

BitTorrent Sync sounds like it will do what you want.

You basically give it a folder to sync, and whatever you put in that folder - structure and all - gets sync'd with any machine you configure with the secret key.

You can even specify a read-only key, so the other machines can't possibly affect your "master".

Machines can be offline for however long, and when they come online they'll connect to the other peers and pull any diffs.

And, of course, it's free.

Answer 2

I'd simply use git clone from DNS name for example git clone ssh://mastergit.example.com/vmshared.git and then git pull periodically with cron. This DNS name would be a alias (CNAME) to one of VMs. When master server changes you just change this DNS alias to another machine.

With authentication with the same SSH keys for every VM you'll have:

• encrypted data transfer as git works over SSH,

• easy switching to different master server,

• easy pushing of changes from your local copy (simply git push),

• version control and easy rollback if something brakes.