Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space or other reasons). We call this our vault.

To store to the vault, a separate machine nfs-mounts the data partition of the scanner and copies data to its own local backup harddisk:

rsync -au /nfsmount/data /pvbackup-vault >> $LOGFILE

My question is: Is this safe? Our data sometimes gets reprocessed, after having been processed once before. So I want the -u flag.

For the actual, raw data (that is sacred) I can foresee one problem: Files on the scanner get overwritten due to some error/mistake/unforeseeable circumstance and then data in the vault will be overwritten. I am not sure how to protect against that. On one side I would like to allow for data to be re-processed maybe even re-acquired on the other side I would like to create a vault that is immune to future changes, at least on the data side. Should I flag those circumstances and deal with them by hand? Tedious.

Note: I have a different incremental strategy (rsnapshot) in place to protect from user error that allows recovery of inadvertently deleted/changed data reaching back a certain number of hours/days/weeks/months.

Note2: Maybe I should mention that we are dealing with ca 250GB currently and ca 10GB per week newly acquired data. So, DVDs are out as alternative...

share|improve this question

2 Answers 2

You are doing the first part right, get the data off of the hardware scanner in the event something happens to it. The second part should be that you take backups of your intermediary backup location. So in other words you either should setup a secondary rsync job to another final resting/backup place or actually have a backup program come in and take a periodic copy for a more permanent/archival purpose.

Often you will do two things to preserve data...

  1. Take the immediate disk to disk backup (Scanner using rsync)
  2. Take an archival backup disk to tape, or some newer methods just another disk to disk or disk to web mechanism.

This ensures your scanner data is protected and you have long term archives of everything that happens. Treat your disk to disk as temporary backup until the archival run can complete, your archive is sacred.

share|improve this answer
    
Totally agree but would add one thing. Add a time-based layer to your directory structure: rsync ... /pvbackup-vault/$now/ –  khoxsey Aug 15 '12 at 20:10
    
Haven't you just recast the problem in different terms? Step 2 requires me to keep making new snapshots and then, in the future when I want some data that's gone missing, to hunt through the various snapshots to see where it happened. An additional wrinkle are the storage requirements that make storing non-incremental snapshots prohibitive - I have put a note about that in my question. –  DrSAR Aug 15 '12 at 20:15
    
Yea, thats why it may make more sense to actually get a true backup/DR system in place for the archive as most allow you to easily search through the backed up files and then browse them by date/time which makes it easy to pick out specific files from specific backups –  Brent Pabst Aug 15 '12 at 20:44
    
mh - keep in mind we are publicly funded. 'true' sounds like expensive to me. Do you pointers to how to achieve this in a cost-efficient manner? –  DrSAR Aug 15 '12 at 21:01
    
No we used a Microsoft product and it only cost us for a server and software, of course if you are a government group you usually get hosed regardless. There may be some other open source alternatives but my guess is you would be looking at a BackupExec or similar type of setup. –  Brent Pabst Aug 16 '12 at 1:28

rsync with --link-dest option might be usefull. It creates a directory for every backup-run. Files that are unchanged since last run are hardlinked to the already present versions. So you have a complete normal folder-structure from every run (you can directly browse through it) without duplicating the unchanged data.

I use the following kind of snippets for > 1 TB since years for daily backups. a linux-gui-tool that uses rsync in that way is backintime http://backintime.le-web.org/

#!/bin/bash

# when saving this as "rsync-history" call it with arguments
# sh rsync-history user@host-to-be-backed-up:/directory-on-host/    where-to-save-the-backup/ 

source=$1
target_fileprefix="$2""$source"

#keep N generations
bigger=100


rm -vrf "$target_fileprefix""$bigger"

#move 1 to 2, 2 to 3...
while [ $bigger -gt 0 ]
do
smaller=$(($bigger-1))      
mv  -f "$target_fileprefix""$smaller"\
"$target_fileprefix""$bigger" 2>>/dev/null
bigger=$(($bigger-1))
done

mkdir -p "$target_fileprefix""0" 

rsync \
-a --whole-file --delete \
--link-dest=$target_fileprefix"1"/.  $source. $target_fileprefix"0"/.  
share|improve this answer
    
yeah - not bad. It is similar to my rsnapshot solution that I am running directly on the scanner console to a separate hard drive. I was hoping to get around the incremental backups since, potentially, I will need to find out the history of a collection of files by looking through all snapshots. Additionally, the file I'm interested in might not even be present in the most recent snapshots. I suppose I can either postpone this headache into the future (your and Brent Pabst's path) or setup a procedure that flags deletions and undesired changes and asks me to handle them on an ongoing basis. –  DrSAR Aug 15 '12 at 23:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.