How do you backup your site? I am especially interested when you have a large site (20GB+) with thousands of files?
Is there anything clever than the usual tar -zcvf backup2010.tar.gz ./public_html/
After answering the above, how do you maintain a consistent backup procedure?
Thanks all for any help
rsync is a quality tool to only copy deltas (changes in files):
rsync --partial --progress --rsh=ssh ./public_html/ [email protected]:/permitted/path
This will backup the minimum amount of data possible to sync your two directories.
In order to then create a backup, I'd recommend running the tar command in your question on the remote machine so you still have an archive per-day, but your bandwidth usage is the lowest possible. You can include SQL database dumps (not the raw tables!) in your rsync backup, but ensure they don't end up in the public facing directories!
To maintain a consistent backup process, cron these tasks.
And don't forget to script and test your restore process! A bad backup is worthless.
There is already a page with more options than I can count, depending upon your needs, at Easy, Automated Snapshot-Style Backups.
The tool I use on a regular basis is rlbackup. rlbackup creates directories like hourly.0, hourly.1, daily.0, daily.1, daily.2, weekly.0, weekly.1, and so on. Old copies are automatically rotated and deleted. The most expensive run will usually be the very first hourly. Every subsequent run will only transfer the changed, new, or deleted files. The directories provide an "snapshot in time" that is easy to browse. Consider this to be a poor-man's version of deduplication.
With all of that said, rsync has it's own performance penalties. Linux Weekly News ran an article that said rsync does have upper limitations about what it can handle well and what it cannot. Regardless of whether or not you have millions of medium files or hundreds of large files, you need to test pure performance. If you have little data changing on your site from day-to-day, but everything is usually new, it may not be too bad.
In the "files are mostly new files" scenario, rsync will probably work well. But if most of the backup is a SQL database, then you need to reconsider the whole approach.
From your suggested use of tar, I presume you're using Unix. That given, bacula is rock-solid. It is somewhat painful to set up, but once you get it configured it does its thing with great reliability. There are a number of features you can use - multiple clients (of course), multiple tape drives, unix/windows clients (and probably MacOS too, though I can't say for sure), encrypted backup tapes, bare-metal restores (and other disaster-recovery features), backups to disc or WORM media (eg, DVD-R). I haven't used all of these features, but I've used a lot of them, and bacula has saved my bacon on a number of occasions - and my users' bacon on many more.
One suggestion I'll pass on from a colleague, with which I'd agree, is that bacula works much better with a tape library than it does with a single tape drive. An old stacker can be had for a song (I use a 6-tape DDS-4 stacker at home, which I picked up for less than £200), and if you're talking about tens of gigabytes and thousands of files, I suggest that such an investment is probably within your grasp.
I also accept that prevailing wisdom is that disc is now so cheap that backup to sequential-storage devices (ie, tape) is an old man's technology. All I can say is that for the price of another box of tapes, I can extend my point-in-time restore capability from four months to eight, and that tapes not in use can be offsited much more easily than an HDD could.
For our Windows server, we use WinSCP, which has a handy feature to synchronize local and remote directories for file transfer. Yes, it must be done manually, but after one backup disaster courtesy of "automatic" backups, I don't care. It only takes clicking one button at the end of the day anyway.
On the remote SSH server, we have a script that makes older copies of those backups, so that if we need a backup less recent than the most recent copy, we can go into the archives.
We use off-site backups, because what happens if the server's drive dies, or worse, the server is destroyed by flood or fire? All local backups would mean we're out of business, if that were the case.
I use a lot rsync. Read carrefully the documentation, there is a lot of interesting possibilities like keeping a backup of all your changed files. Rsync transfers only changed files so it's very efficient if you don't have too many files.
backup-manager is very interesting: easy install on Debian, very quick configuration, differential tarball backup, mysql backup, transfer by samba, ftp ...
There is also this tool who can be interesting: rdiff-backup but I have no experience with it.
Use cron daily to launch these tools, keep the backup on a different host, check often your backups (best practice is to do recovery tests), try to encrypt the backups (in case of someone stole them ...), get some backups off site in case of big disasters ...
rsnapshot.org or writing your own using the techniques from http://blog.interlinked.org/tutorials/rsync_time_machine.html
--link-dest= allows you to hardlink so that your new backup contains hardlinks to the unchanged files, and the actual file for something that has changed since the last backup. Only the changed files are transferred after the initial backup. Deleting a previous generation doesn't affect any files in later backups and only removes the hardlinks. Disk space taken amounts to the full backup + the generational changes. Ideally, you could keep many weeks/months of backups depending on available space and how much of your dataset changes.
We use a combination of rsync and JungleDisk Workgroup to archive / backup our site and its files. It is a pretty simple setup and not too expensive, but the first sync will take a long time.
I use duplicity to backup to S3. I pick my destination dirs such that I do fulls once per quarter and incrementals nightly. (With duplicity, this is as simple as picking a new destination dir once per quarter).
Also, I periodically restore a file or two just to make sure the backups aren't corrupt.
We use BackupPC
Right now my site is small. I make changes locally and copy files to server. I use SVN locally to save most content and copy bin files(as in images) to arch folders. So, this is pretty easy and minor. Also if i lose that data it isnt hard to recover it or simply just update the images.
With data, I have a bash script that backups my sql db on the server and another script locally that logs in with ssh and pulls the backups. Completely automatic (cron + window task scheduler). So all i need to worry is will my site get full (my site will email me before it happens), will i change an image and not back it up and if i forgot to check in my new code to SVN.
I like
s3cmd (http://s3tools.org/s3cmd)
It can be installed quite easily (Debian: apt-get install s3cmd).
You need an Amazon AWS account to store your files on S3. Then a simple command can run your backup, even incremential or as a sync solution, e.g.:
s3cmd sync /srv/backup s3://your-bucket-name-at-amazon/
Make sure you run
s3cms --configure
first to enter your AWS credentials.
I wrote this little script to do exactly what you speak of.
Automatic daily backups of web directory and mysql databases with des3 encryption, and all of these files are displayed elegantly on your web browser.
Screenshot: https://i.stack.imgur.com/CC82a.png
You can customize the web directory and mysql username and how many days to keep old backups.
Download here:
tarcommand on a data-centre or a camp-site. ;)