Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I frequently need to do a backup of a group of files, with many subfolders which contain several large, identical files.

Is there a compression scheme (.zip, .7z, etc) which can automatically detect this and not store identical files more than once?

share|improve this question
    
which operating system are you using? –  Daniel t. Mar 1 '13 at 0:01
    
I'm on Windows (7) but the files are on a network, so I could run the compression app on any OS. –  Warpin Mar 1 '13 at 0:30
    
you can google for incremental backup. –  PersianGulf Mar 1 '13 at 0:43
    
incremental backup doesn't seem to help, as new files (many identical to existing files) are created in new folders. –  Warpin Mar 1 '13 at 1:32

2 Answers 2

up vote 0 down vote accepted

Yes, it's possible: http://superuser.com/questions/479074/why-doesnt-gzip-compression-eliminate-duplicate-chunks-of-data

Here's an example I came up with:

[jay test]$ tree .
.
`-- compressme
    |-- a
    |   `-- largefile (10MB)
    `-- b
        `-- largefile (10MB, identical to ../a/largefile)

3 directories, 2 files
[jay test]$ du -sh compressme/
21M compressme/
[jay test]$ tar -cf compressme.tar compressme/
[jay test]$ du -sh compressme.tar 
21M compressme.tar
[jay test]$ lzma -9 compressme.tar
[jay test]$ du -sh compressme.tar.lzma 
11M compressme.tar.lzma
share|improve this answer

I also just went through this too.

If you compress your files into a Tar Ball, 7z's LZMA compression may or may not recognise the duplicates if they are separated too far in the Tar Ball (it's a function of Dictionary Size and a few other things).

7z has a WIM format which collects duplicates, then you can use normal LZMA compression for there. Windows Command Line Example:

7z a -twim "Example.wim" *
7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on -mhc=on "Example.7z" "Example.wim"
del "Example.wim"

It works well, give it a go.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.