Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a website which works with lots and lots of text files - now they take up about 40 GB of data and increasing over time. I need to make a full daily backup. My current strategy is to make a password protected archive and store it in the dropbox with this command:

tar cfz - /var/www/mysite | openssl enc -aes-256-cbc -e -k "b@ckupPassword" > /home/user/Dropbox/server_backups/sources/2013_01_04_0500_mysite_source_encrypted.tgz

It works, but making an archive takes about 14 hours and consumes a lot of IO - and it will be worse as the amount of data increases.

What is the proper strategy of backing up such large amount of files?

share|improve this question
    
With that amount of data you should be using a database, not text files. It's a simple enough change and the benefits are great. –  John Gardeniers Jan 4 '13 at 12:48
add comment

3 Answers

up vote 3 down vote accepted

I would use rsync, provided I have space enough.

This example has a full backup and keeps a week of incremental ones.

#!/bin/bash

DAY=$(date +%A)

if [ -e /bkp/incr/$DAY ] ; then
  rm -fr /bkp/incr/$DAY
fi

ionice -c 3 rsync -a --delete --inplace --backup --backup-dir=/bkp/incr/$DAY /var/www/mysite/ /bkp/full/ &> /dev/null

The script can be complicated to store offsite, on dropbox folders and so on.

If you really need to use tar, you can keep track of the modified files

tar zcvf - --listed-incremental=/var/log/mysite.tarlog /var/www/mysite | openssl enc  ...

If you want a full backup, delete /var/log/mysite.tarlog

share|improve this answer
add comment

As suggested above rsync seems to be the best way to backup the whole site. Yet i would suggest you to implement some sort of replicated filesystem, something like a simple GlusterFS volume with replication.

Replication is not a backup, but it can help you to reduce the I/O impact of backups and eventually provide you with a solid base to expand your website later into a cluster.

share|improve this answer
add comment

It is better to use an incremental backup mechanism in this case.

Using rsync you can take incremental backup.

Click on me for more details about using rsync

Click on me to read more about incremental backup

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.