rsync for backup considered dangerous?

Question

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space or other reasons). We call this our vault.

To store to the vault, a separate machine nfs-mounts the data partition of the scanner and copies data to its own local backup harddisk:

rsync -au /nfsmount/data /pvbackup-vault >> $LOGFILE

My question is: Is this safe? Our data sometimes gets reprocessed, after having been processed once before. So I want the -u flag.

For the actual, raw data (that is sacred) I can foresee one problem: Files on the scanner get overwritten due to some error/mistake/unforeseeable circumstance and then data in the vault will be overwritten. I am not sure how to protect against that. On one side I would like to allow for data to be re-processed maybe even re-acquired on the other side I would like to create a vault that is immune to future changes, at least on the data side. Should I flag those circumstances and deal with them by hand? Tedious.

Note: I have a different incremental strategy (rsnapshot) in place to protect from user error that allows recovery of inadvertently deleted/changed data reaching back a certain number of hours/days/weeks/months.

Note2: Maybe I should mention that we are dealing with ca 250GB currently and ca 10GB per week newly acquired data. So, DVDs are out as alternative...

Answer 1

You are doing the first part right, get the data off of the hardware scanner in the event something happens to it. The second part should be that you take backups of your intermediary backup location. So in other words you either should setup a secondary rsync job to another final resting/backup place or actually have a backup program come in and take a periodic copy for a more permanent/archival purpose.

Often you will do two things to preserve data...

1. Take the immediate disk to disk backup (Scanner using rsync)
2. Take an archival backup disk to tape, or some newer methods just another disk to disk or disk to web mechanism.

This ensures your scanner data is protected and you have long term archives of everything that happens. Treat your disk to disk as temporary backup until the archival run can complete, your archive is sacred.

Answer 2

rsync with --link-dest option might be usefull. It creates a directory for every backup-run. Files that are unchanged since last run are hardlinked to the already present versions. So you have a complete normal folder-structure from every run (you can directly browse through it) without duplicating the unchanged data.

I use the following kind of snippets for > 1 TB since years for daily backups. a linux-gui-tool that uses rsync in that way is backintime http://backintime.le-web.org/

#!/bin/bash

# when saving this as "rsync-history" call it with arguments
# sh rsync-history user@host-to-be-backed-up:/directory-on-host/    where-to-save-the-backup/ 

source=$1
target_fileprefix="$2""$source"

#keep N generations
bigger=100


rm -vrf "$target_fileprefix""$bigger"

#move 1 to 2, 2 to 3...
while [ $bigger -gt 0 ]
do
smaller=$(($bigger-1))      
mv  -f "$target_fileprefix""$smaller"\
"$target_fileprefix""$bigger" 2>>/dev/null
bigger=$(($bigger-1))
done

mkdir -p "$target_fileprefix""0" 

rsync \
-a --whole-file --delete \
--link-dest=$target_fileprefix"1"/.  $source. $target_fileprefix"0"/.