Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I frequently need to do a backup of a group of files, with many subfolders which contain several large, identical files.

Is there a compression scheme (.zip, .7z, etc) which can automatically detect this and not store identical files more than once?

share|improve this question
which operating system are you using? –  Daniel t. Mar 1 '13 at 0:01
I'm on Windows (7) but the files are on a network, so I could run the compression app on any OS. –  Warpin Mar 1 '13 at 0:30
you can google for incremental backup. –  Mohsen Pahlevanzadeh Mar 1 '13 at 0:43
incremental backup doesn't seem to help, as new files (many identical to existing files) are created in new folders. –  Warpin Mar 1 '13 at 1:32

1 Answer 1

up vote 0 down vote accepted

Yes, it's possible: http://superuser.com/questions/479074/why-doesnt-gzip-compression-eliminate-duplicate-chunks-of-data

Here's an example I came up with:

[jay test]$ tree .
`-- compressme
    |-- a
    |   `-- largefile (10MB)
    `-- b
        `-- largefile (10MB, identical to ../a/largefile)

3 directories, 2 files
[jay test]$ du -sh compressme/
21M compressme/
[jay test]$ tar -cf compressme.tar compressme/
[jay test]$ du -sh compressme.tar 
21M compressme.tar
[jay test]$ lzma -9 compressme.tar
[jay test]$ du -sh compressme.tar.lzma 
11M compressme.tar.lzma
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.