Initial rdiff-backup on slow connection / large data (Could performing an rsync before help?)

Question

I want to set up standard rdiff-backup to remote server. However the data is very big for the capacity of internet connection. Therefore I need some way to do the initial copy of data without rdiff-backup (as there is no way to resume initial transfer which failed after hours of thransfer).

My idea was to transfer data initially on physical media to remote server. After that transfer I synchronized the data by rync to make sure they are identical. Now I want to do intial rdiff-backup. When I run the rdiff-backup there are lots of lines

Getting delta (with hash) of XY

as if the directories were NOT identical. What exactly do these lines mean? Do they mean that the remote rdiff-backup decided that the file is not identical, so there is a delta diff transferred?

Is there any other way to do initial transfer of large data which can be resumed in case of failure (as opposed to be restarted from beginning)?

Answer 1

I use duplicity instead of rdiff-backup but from my understanding your problem is here:

My idea was to transfer data initially on physical media to remote server. After that transfer I synchronized the data by rync to make sure they are identical. Now I want to do intial rdiff-backup.

Solution that works

The simplest solution is to rdiff-backup to that physical media at the backup source, instead of copying and rsyncing data. Then transfering the backup via physical media to remote server and use it as a backup directory for remote backups. Because when copying and rsyncing there is still no initial backup and the program needs to crosscheck all files and save the metadata. Thus the Getting delta.... Which is probably not transfering whole files, just a lot of checksums.

Solution that does not work

It may be worth trying to rdiff-backup from the physical media into a directory and then remotely rdiff-backup there from remote server. Though if rdiff-backup uses some local metadata as well (duplicity does), you (or it) will have to transfer that back or recreate it.

Editted according to what worked for OP.

Answer 2

Are you really bound by rdiff-backup or you are OK using plain rsync to do incremental backups?

Here's a small script that keeps 4 incremental backups. You can change it easily to keep 9 or 99 incremental backups or any other number.

It uses the --link-dest option from rsync, creating hard links if files are unchanged.

FROM=$HOME
TO=/var/backups
LINKTO=--link-dest=$TO/`/usr/bin/basename $FROM`.1
OPTS="-a --delete -delete-excluded"
NUMBER_OF_BACKUPS=4


/usr/bin/find $TO -maxdepth 1 -type d
  -name "`/usr/bin/basename $FROM`.[0-9]"|\
  /usr/bin/sort -rn| while read dir
do
    this=`/usr/bin/expr match "$dir" '.*\([0-9]\)'`; 
    let next=($this+1)%$NUMBER_OF_BACKUPS;
    basedirname=${dir%.[0-9]}
    if [ $next -eq 0 ] ; then
             /bin/rm -rf $dir
    else
             /bin/mv $dir $basedirname.$next
    fi
done
/usr/bin/rsync $OPTS $LINKTO $FROM/ $TO/`/usr/bin/basename $FROM.0`

With that script, you can first rsync your data manually, they configure the script to be launched regularly.