Is reasonably safe to use rsync (without --delete option) to make my daily backups and keep only 1 copy on the backup server (instead of having multiple copies with rotations) ?

If I don't enable the --delete option, any backup file won't be ever deleted, correct ?



(this started as a comment on OnoeOfOne's answer, but grew too long for that...)

I use an rsync+cp -al method similar to that show in http://www.mikerubel.org/computers/rsync_snapshots/ (that page was my original crib sheet some time in 2005, and I've not modified things an awful lot since) both for my personal backups and for managing the online+onsite and online+offite backups at work.

How much space this takes depends on the balance of sizes and how often files are changed and deleted. For us, keeping daily snapshots for over a year this way doesn't take much more space than three full copies of the data, as most of our files on the general network shares are not modified or removed often. In fact for many of the snapshots the directory structure takes more space than the data in changed files.

One thing to be wary of is that because each copy of the same version of a file is in fact the same data, on-disk corruption could kill the file in all your snapshots in one fell swoop, so there is still a case for keeping multiple copies. We mitigate this by having multiple copies on different machines and all the machines using RAID1 to protect against certain possible physical problems. Another way to address this if you only have one online backup location is to actually keep two copies and sync them individually, or just force a full refresh occasionally (say, once a month or once a week) so you end up with groups of snapshots: i.e. for a monthly forced refresh-all all the identical files in snapshots for January are the same data block, as are they in February, but there are at least two copies of the actual data if it existed in both months.

  • Wait a minute, so it takes a bit more than 3 full copies of the data ? I thought if you use hard links you just have a copy with only few updates.
    – aneuryzm
    Jan 9 '11 at 14:27
  • What I mean is that if I'm not going to save too much space, than I could also choose an easier option, such as 2 copies: 1 for daily backups and the other one for monthly backups and that's it.
    – aneuryzm
    Jan 9 '11 at 14:28
  • @Patrick: "The space of three full copies" covers one snapshot for every day in a year or more, and it is useful reassurance to us for those files that do change often to have the daily snapshots for that long. 3 times as much space as one copy for ~400 snapshots is quite efficient and if we were only keeping daily backups for a week the seven snapshots might be only take a couple of percent more than the space for one single full copy, maybe even less. Jan 9 '11 at 15:06
  • For instance: ignoring special cases (archive areas that almost never change, our source control which is handled differently, database backups which are again handled differently) the data including filesystem overheads as of 2010-12-19 (going back in time to avoid weeks around the winter break) last night's backup run was 34,643Mb. Seven snapshots (the 13th to the 19th) take just 35,0366Mb. For just the most changed area of the few considered above it is 23,908Mb for one day and 24,110 for 7. Jan 9 '11 at 17:04

If you don't enable --delete, files are not deleted yes. Although a file could be truncated to 0 bytes if that was what was at the source.

Also check out the --backup option to see if that is suitable for what are you doing.

How are you going to prevent data corruption from simply being copied and destroying your only backup copy, or are you archiving them off somewhere?

Depending on what the source looks like and how it changes, you may over time accumulate lots of cruft and old data this way if you never delete it.

  • All right, corrupted data are an issue. But I'm now thinking: (1) if I keep daily backups, do I need 7times the space amount of 1 copy ? (2) if something bad happen, I have 7 days to recover a good copy of the backup ?
    – aneuryzm
    Jan 9 '11 at 12:18
  • Maybe I could keep 2 copies: a weekly copy and a daily copy and that's it ? (without rotations)
    – aneuryzm
    Jan 9 '11 at 12:19
  • You can look at keeping incrementals if that is what you are after. Have a look at bacula, it may be a challenge to setup but it is a good product once you have it running. Jan 10 '11 at 10:02

Take a look at rsnapshot. It does an rsync for modified files, but creates a hardlink for files not modified since last rotation. It's almost as simple to use as raw rsync, but you can save this space you're worried about.

  • @funollet I cannot use it because I don't have access to crontab on backup server.
    – aneuryzm
    Jan 9 '11 at 13:08
  • can't run it as a user? crontab -e?
    – OneOfOne
    Jan 9 '11 at 13:50
  • 1
    @patrick: You only need rsync on the backup server. rsnapshot and cron can run on the backup client.
    – funollet
    Jan 9 '11 at 15:01
  • 1
    @funollet well, snapshot_root parameter in snapshot.conf actually needs a local path and rsnapshot is intended to pull the data from the production server from the backup server.
    – aneuryzm
    Jan 9 '11 at 15:04
  • 1
    Woops, you're right, Patrick. Sorry for the misinformation, I was mixing two programs. The one I managed to configure in 'push mode' time ago was Backupninja (labs.riseup.net/code/projects/show/backupninja)
    – funollet
    Jan 10 '11 at 11:54

What I personally use is rsync using hard-links and keep 5 copies, the trick is, if the file doesn't change, it won't take extra space that way, and it makes it very easy to recover if something blows up.


#remove the oldest backup
rm -rf backup.4 backup.4.log.bz2 &>/dev/null
recycle() {
        i=$1; y=$(($i+1))
        mv "${b}.$i" "${b}.$y" &>/dev/null
        mv "${b}.$i.log.bz2" "${b}.$y.log.bz2" &>/dev/null
recycle 3
recycle 2
recycle 1
recycle 0

OPTS="--numeric-ids --delete --delete-after --delete-excluded"
DIRS_TO_BACKUP="/home /var"
nice -n20 ionice -c2 -n2 rsync -axlHh -v --link-dest=../backup.1 ${OPTS} ${DIRS_TO_BACKUP} backup.0/ --exclude-from=/root/.rsync-exclude 2>&1 | bzip2 -9 > backup.0.log.bz2

my /root/.rsync-exclude :

  • @OneOfOne mhm ok, is there a tutorial there to do what you describe. The code you pasted is a bit dense. How are you exactly creating the links ?
    – aneuryzm
    Jan 9 '11 at 13:13
  • Most important, how much space does it take to save the 5 copies in this way ? More than twice the original data ?
    – aneuryzm
    Jan 9 '11 at 13:14
  • @OneOfOneSorry, one more question: what does this imply... I have a full backup copy and all the others copy are just hard links to the first one, with eventually few updated files ?
    – aneuryzm
    Jan 9 '11 at 13:29
  • And how can I recover a specific version? Should I use rsync using --hard-link option ?
    – aneuryzm
    Jan 9 '11 at 13:29
  • the --link-dest part links from the most recent backup, and if no files changed at all for all 5 copies it will just use the space for one copy, to recover one copy you can just use cp or rsync, --hard-links doesn't work across filesystems, they have to be on the same filesystem.
    – OneOfOne
    Jan 9 '11 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.