Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I store a very large (TBytes) amount of logs. These logs will rarely be extracted, and when they will - only a single file will be required.

Could you recommend an ultra-efficient and extremely stable compression algorithm that's considerably better than bzip2?

share|improve this question

1 Answer 1

up vote 15 down vote accepted

lzma (aka xz) should do notably better than bzip2, but will take a bit longer.
paq (aka zp) will do quite a bit better yet, but will take ages to compress and just as long to decompress.

Both are available for Windows and *nix environments (most *nix systems have packages available)

A quick test on a smartd log:

Original       3900K
GZip            208K    0.11s
BZip2            71K    3.07s
XZ               13K    1.76s*
ZP                6K   25.68s*

*I've got -O3 compiled ports for xz and zp. The gzip and bzip2 binaries were precompiled with no optimization.

share|improve this answer
2  
See also maximumcompression.com/data/log.php –  jftuga Jun 12 '11 at 19:11
    
xz is faster than bz2. Even if your quick test shows it. –  Brendan Long Jun 12 '11 at 20:22
3  
In linux lzma is also supported by tar gnu.org/software/tar/manual/html_section/Compression.html –  leonbloy Jun 12 '11 at 20:27
    
@Brendan, just tested -Os compiled versions of xz and bzip2 on 100MB of random data. bzip2 took 45.39s while xz took 1:25.77; bzip2 should be faster for any sizable dataset and similar compiler optimizations. @leonbloy, tar also supports the use of external compressors if they can work on stdin to stdout, which all of these compressors do. –  Chris S Jun 13 '11 at 1:25
    
@ChrisS but the original 7z doesn't, also a built in alias of -J sure gives something –  Hubert Kario Jan 15 '12 at 20:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.