Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is reasonably safe to use rsync (without --delete option) to make my daily backups and keep only 1 copy on the backup server (instead of having multiple copies with rotations) ?

If I don't enable the --delete option, any backup file won't be ever deleted, correct ?

thanks

share|improve this question

4 Answers 4

up vote 1 down vote accepted

(this started as a comment on OnoeOfOne's answer, but grew too long for that...)

I use an rsync+cp -al method similar to that show in http://www.mikerubel.org/computers/rsync_snapshots/ (that page was my original crib sheet some time in 2005, and I've not modified things an awful lot since) both for my personal backups and for managing the online+onsite and online+offite backups at work.

How much space this takes depends on the balance of sizes and how often files are changed and deleted. For us, keeping daily snapshots for over a year this way doesn't take much more space than three full copies of the data, as most of our files on the general network shares are not modified or removed often. In fact for many of the snapshots the directory structure takes more space than the data in changed files.

One thing to be wary of is that because each copy of the same version of a file is in fact the same data, on-disk corruption could kill the file in all your snapshots in one fell swoop, so there is still a case for keeping multiple copies. We mitigate this by having multiple copies on different machines and all the machines using RAID1 to protect against certain possible physical problems. Another way to address this if you only have one online backup location is to actually keep two copies and sync them individually, or just force a full refresh occasionally (say, once a month or once a week) so you end up with groups of snapshots: i.e. for a monthly forced refresh-all all the identical files in snapshots for January are the same data block, as are they in February, but there are at least two copies of the actual data if it existed in both months.

share|improve this answer
    
Wait a minute, so it takes a bit more than 3 full copies of the data ? I thought if you use hard links you just have a copy with only few updates. –  Patrick Jan 9 '11 at 14:27
    
What I mean is that if I'm not going to save too much space, than I could also choose an easier option, such as 2 copies: 1 for daily backups and the other one for monthly backups and that's it. –  Patrick Jan 9 '11 at 14:28
    
@Patrick: "The space of three full copies" covers one snapshot for every day in a year or more, and it is useful reassurance to us for those files that do change often to have the daily snapshots for that long. 3 times as much space as one copy for ~400 snapshots is quite efficient and if we were only keeping daily backups for a week the seven snapshots might be only take a couple of percent more than the space for one single full copy, maybe even less. –  David Spillett Jan 9 '11 at 15:06
    
For instance: ignoring special cases (archive areas that almost never change, our source control which is handled differently, database backups which are again handled differently) the data including filesystem overheads as of 2010-12-19 (going back in time to avoid weeks around the winter break) last night's backup run was 34,643Mb. Seven snapshots (the 13th to the 19th) take just 35,0366Mb. For just the most changed area of the few considered above it is 23,908Mb for one day and 24,110 for 7. –  David Spillett Jan 9 '11 at 17:04

If you don't enable --delete, files are not deleted yes. Although a file could be truncated to 0 bytes if that was what was at the source.

Also check out the --backup option to see if that is suitable for what are you doing.

How are you going to prevent data corruption from simply being copied and destroying your only backup copy, or are you archiving them off somewhere?

Depending on what the source looks like and how it changes, you may over time accumulate lots of cruft and old data this way if you never delete it.

share|improve this answer
    
All right, corrupted data are an issue. But I'm now thinking: (1) if I keep daily backups, do I need 7times the space amount of 1 copy ? (2) if something bad happen, I have 7 days to recover a good copy of the backup ? –  Patrick Jan 9 '11 at 12:18
    
Maybe I could keep 2 copies: a weekly copy and a daily copy and that's it ? (without rotations) –  Patrick Jan 9 '11 at 12:19
    
You can look at keeping incrementals if that is what you are after. Have a look at bacula, it may be a challenge to setup but it is a good product once you have it running. –  giant_squid Jan 10 '11 at 10:02

Take a look at rsnapshot. It does an rsync for modified files, but creates a hardlink for files not modified since last rotation. It's almost as simple to use as raw rsync, but you can save this space you're worried about.

share|improve this answer
    
@funollet I cannot use it because I don't have access to crontab on backup server. –  Patrick Jan 9 '11 at 13:08
    
can't run it as a user? crontab -e? –  OneOfOne Jan 9 '11 at 13:50
    
+1, this is what does the Mac OS Time machine, a marvelous idea... –  ring0 Jan 9 '11 at 14:16
    
@patrick: You only need rsync on the backup server. rsnapshot and cron can run on the backup client. –  funollet Jan 9 '11 at 15:01
1  
@funollet well, snapshot_root parameter in snapshot.conf actually needs a local path and rsnapshot is intended to pull the data from the production server from the backup server. –  Patrick Jan 9 '11 at 15:04

What I personally use is rsync using hard-links and keep 5 copies, the trick is, if the file doesn't change, it won't take extra space that way, and it makes it very easy to recover if something blows up.

#!/bin/sh
BACKUP_DIR=/mnt/data-3/backups/

cd ${BACKUP_DIR}
#remove the oldest backup
rm -rf backup.4 backup.4.log.bz2 &>/dev/null
recycle() {
        i=$1; y=$(($i+1))
        b=${2-backup}
        mv "${b}.$i" "${b}.$y" &>/dev/null
        mv "${b}.$i.log.bz2" "${b}.$y.log.bz2" &>/dev/null
}
recycle 3
recycle 2
recycle 1
recycle 0

OPTS="--numeric-ids --delete --delete-after --delete-excluded"
DIRS_TO_BACKUP="/home /var"
nice -n20 ionice -c2 -n2 rsync -axlHh -v --link-dest=../backup.1 ${OPTS} ${DIRS_TO_BACKUP} backup.0/ --exclude-from=/root/.rsync-exclude 2>&1 | bzip2 -9 > backup.0.log.bz2

my /root/.rsync-exclude :

*~
*.cmd*
*.log
cache4
/tmp/
.ccache
.thumbnails/
lost+found
/var/log/
/var/run/
/var/lock/
/var/tmp/
/usr/src/
share|improve this answer
    
@OneOfOne mhm ok, is there a tutorial there to do what you describe. The code you pasted is a bit dense. How are you exactly creating the links ? –  Patrick Jan 9 '11 at 13:13
    
Most important, how much space does it take to save the 5 copies in this way ? More than twice the original data ? –  Patrick Jan 9 '11 at 13:14
    
@OneOfOneSorry, one more question: what does this imply... I have a full backup copy and all the others copy are just hard links to the first one, with eventually few updated files ? –  Patrick Jan 9 '11 at 13:29
    
And how can I recover a specific version? Should I use rsync using --hard-link option ? –  Patrick Jan 9 '11 at 13:29
    
the --link-dest part links from the most recent backup, and if no files changed at all for all 5 copies it will just use the space for one copy, to recover one copy you can just use cp or rsync, --hard-links doesn't work across filesystems, they have to be on the same filesystem. –  OneOfOne Jan 9 '11 at 13:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.