Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This is a crosspost from stackoverflow because I find that I might get a much qualified audience here

The Situation

I have a pretty large Subversion repository that I am trying to backup efficiently. The repository size is about 6 GB and growing. Some large commits are around 500 to 1GB in size.

I am trying to backup this repository to an off-site location, through an Internet up-link.

Explaining the sheer size of it

To whomever is wondering, the whole production environment for various sites (config files, EXEs, data files) is kept in this one repository so that we can rollback to an existing working version and track the changes to the production setup. Code is kept on a different repository.

The How

Here is what I am actually doing:

  1. Backup of the repository to a working folder on the server using the "svnadmin hotcopy SRCDIR TGTDIR"
  2. Encrypt and compress that repository using "rsyncrypto -r SRCPATH DSTPATH KEYSPATH CERTIFICATE"
  3. Backup that encrypted version to an off-site location using "rsync -Crtv" (actually cwRsync because I am running on Windows)

The Problem

First I have to say that it works, though it still has an underlying issue.

The problem lies with the fact that I was expecting that each time the process would run, only the new revision files/data would be copied ([repos]/db/revs/0/...) thus requiring only bandwidth and time when a large commit is made. However, instead:

  • If I run only step #3 many times, rsync behaves as it should and nothing is copied because nothing has changed.
  • If I run only steps #2 & #3 many times, rsync also behaves well. The envrypted version is the same everytime and rsync doesn't have to transmit anything.
  • But, it seems that every time I run all three steps (with a new commit having been made to the repository) the whole repository is being re-uploaded in full. Thus, defeating the whole purpose of using rsync in the first place.

It is as though the files in [repos]/db/revs/0/... are changing everytime I make a hotcopy.

The Questions

Is this an expected behavior from "svnadmin hotcopy" that the [repos]/db/revs/0/... are changing from one hotcopy to another?

Any suggestion or options I could use to make this hotcopy rsync friendly or say rsyncable?

I am not quite sure that the use of 'svnadmin dump' on the whole repository would produce an "rsyncable" file.

share|improve this question
    
Can you edit your post to link to the stackoverflow question so we can monitor answers? –  Caleb Oct 19 '10 at 10:32

2 Answers 2

When you encrypt/compress the repository any small change to the underlying data (the repository) is spread throughout the encrypted/compressed file(s). So, a one byte change can propagate across the entire encrypted/compressed data set. So, from rsync's perspective, everything DID change.

share|improve this answer
    
Actually NO. That is the whole purpose of rsyncrypto. It does work its encryption in blocks and if something changes at the beginning it will propagate on only one block length and the rest of the file will stay same. This provides a slight less secure encryption at the benefit of a rsync friendly encrypted file. –  philibertperusse Sep 15 '10 at 15:30
    
You are correct, now that I looked into Rsyncrypto more. –  Craig Sep 15 '10 at 15:37
    
And even better, rsyncrypto automatically does file compression using gzip --rsyncable option, which also compress using a block size. Thus, large files becomes encrypted and smaller before transmission. If I can just manage to have similar looking files in the first place, for: subversion, exchange backup, TFS backup, SQL backups, etc. –  philibertperusse Sep 15 '10 at 22:25

Files in db/revs/ are not changing, but their timestamps do. --size-only could be the answer.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.