Automatic off-site backup

Question

I'm planning to launch a file host fairly soon. I have a 12TB software RAID array in my 1U server unit, and another 12TB software RAID array at a private location. Users will be uploading (and possibly removing) files to and from the space (and modifications may be made to some files as the database will also be stored on the array) and I need everything from machine one to be backed up to machine 2 (and modifications, removals and creations to be submitted).

How can I accomplish this? (I'll want to backup ideally either every day, or just whenever the changes to the 12TB are made) I want the system to be as fast as possible as it'll be sucking up some of the upload bandwidth..

I'm using Ubuntu Server with an mySQL database.

Answer 1

For most of your files, rsync will do the job very nicely, however for the database, you will probably need something more specialised (or create a db dump file which can be rsync'd to the backup and then imported in to the backup db).

Set up password-less ssh between the 2 servers (by exchanging ssh keys) and then you can use a command like this to copy the files (run on the 'master' server):

rsync -az --delete /path/to/backup/ backup-server:/path/to/backup/

This will copy the entire directory tree /path/to/backup to the same path on hostname backup-server. The rsync options are -a (archive mode) preserve user/owner/permissions for each file, and -z to compress files before transferring them and decompress them on the receiving end automatically. The --delete flag deletes files which are on the backup server but no longer on the master server.

Use cron to start this command when required.

All this assumes that the 2 servers are running Unix/Linux.

Answer 2

You don't tell anything about the platform, the database used or whatever.

Nevertheless, at least for the file part, I would have a look into rsync, which works very well for large file trees.

Answer 3

Without getting some expensive, complicated software. You could just set up a cron-job to run an RSYNC task once a day. (That, of course assumes you're running UNIX/Linux. If you're running Windows I'm not sure)

RSYNC has the advantage of only transferring new files or files that have changed. You can even specify RSYNC to delete stuff on the backup server no longer present on the main server.