In the past people have checked in massive files to our subversion repository and later regretted it. To fix it, they just deleted the directory and made another checkin (leaving the massive files in the repository's history).

Since these massive files were accidents and not intended to be in the history, I wanted to filter them out using svndumpfilter. Is there any easy way to find large directories that were checked in? Perhaps sort revision diffs by size?


I basically found it by analysing the svndump file and using a small python script:

$ # dump repository to file
$ svnadmin dump /var/lib/svn/ > svn_full_dump.txt

$ # find byte offsets of 'Revision-number' ignoring non-ascii and save in file
$ egrep -boa  '^Revision-number: .+$' svn_full_dump.txt > revisions.txt

$ head revisions.txt 
75:Revision-number: 0
195:Revision-number: 1
664:Revision-number: 2
863:Revision-number: 3
1058:Revision-number: 4
1254:Revision-number: 5
1858:Revision-number: 6

$ # find size of checkins and sort by size
$ python revision_size.py  | sort -nr | head
1971768485 r1528
44453981 r2375
39073877 r1507
34731033 r2394
30499012 r484

The python file is:

#!/usr/bin/env python

f = file('revisions.txt')

last_offset = 0
last_revision = None

for l in f:
    l = l.strip()
    (offset, middle, revision) = l.split(':')
    offset = int(offset.strip())

    revision_size = offset-last_offset
    if last_revision:
        print '%s r%s'%(revision_size, last_revision.strip())

    last_revision = revision
    last_offset = offset

# will ignore last revision


update: fixed a bug in the revision_size script where the size wasn't quite matched up to the right revision.

  • 2
    Why you're writing scripts, why not a pre-commit script to keep files over a certain size (10MB? 100MB? whatever) from being able to be checked in. If you were feeling ambitious you could even add some kind of exception mechanism that would allow overlarge files if they had a particular svn property set or were listed in a .svnlargefiles file (also in the repo) or somesuch.
    – pjz
    Jul 7 '09 at 4:31
  • I faced a failure with "binary content matches" in egrep output, so that I have added "-a" option May 18 '20 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.