I am using
tar to backup a linux server to tape. I am using the
-j option to compress the file with
bzip2, however I can't see a way to adjust the block size options for bzip2 from tar. The default block size is 900,000 bytes which gives the best compression but is the slowest. I am not that bothered about the compression ratio, so am looking to make bzip2 run faster with a smaller block size.
I am using
1Sidenote: Lately I've all but given up on bzip2. I use lzma (from the lzma, lzma-utils, or lzma-sdk package, name depends on your distribution.) It usually compresses the same or better than bzip2 given the same CPU time - and when it comes to decompression it simply blows bzip2 away.– Mihai LimbăşanMay 2, 2009 at 19:53
export BZIP=--fast tar cjf foo.tar.bz2 foo
Or pipe the output of
Though you should note from the bzip2 man page:
-1 (or --fast) to -9 (or --best) Set the block size to 100 k, 200 k .. 900 k when compressing. Has no effect when decompressing. See MEMORY MANAGEMENT below. The --fast and --best aliases are primarily for GNU gzip compat- ibility. In particular, --fast doesn't make things signifi- cantly faster. And --best merely selects the default behaviour.
tar -cjf dir.tar.bz2 --options bzip2:compression-level=9 path/to/dir/
On my system (OSX El Capitan bsdtar 2.8.3) this is missing from the man page (although gzip:compression-level and xz:compression-level are listed), but testing it the option does work. Sep 5, 2016 at 16:20
tar: unrecognized option '--options'– ZN13Jul 18, 2018 at 18:40
bzip2 block sizes
bzip2 has some block size options. From the manual page
-1 (or --fast) to -9 (or --best) Set the block size to 100 k, 200 k .. 900 k when compressing. Has no effect when decompressing. See MEMORY MANAGEMENT below. The --fast and --best aliases are primarily for GNU gzip compatibility. In particular, --fast doesn't make things significantly faster. And --best merely selects the default behaviour.
As you want faster compression with less regards to compression ratio, using
bzip2, you seem to want the
bzip2 block size when using
You can set
bzip2 block size when using
tar in a couple of ways.
The UNlX way
My favorite way, the UNlX way, is one where you use every tool independently, and combine them through pipes.
$ tar --create [FILE...] | bzip2 -1 > [ARCHIVE].tar.bz2
You can read that as "create .tar with
tar -> bzip it with
bzip2 -> write it to
It is also possible to set
bzip2 options through the environment variable
BZIP2. From the manual page
bzip2 will read arguments from the environment variables BZIP2 and BZIP, in that order, and will process them before any arguments read from the command line. This gives a convenient way to supply default arguments.
So to use that with
tar, you could for example do:
$ BZIP2=-1 tar --create --bzip2 --file [ARCHIVE].tar.bz2 [FILE...]
bzip2 uses a slow compression algorithm. If you are concerned about speed, you could investigate alternative algorithms, such as those used by
lzop. Here is a nice article comparing compression tools: https://aliver.wordpress.com/2010/06/22/huge-unix-file-compresser-shootout-with-tons-of-datagraphs/
It looks like you may have the knowledge to provide good Answer here, but please consider reading How do I write a good Answer? in our help center and then revise the Answer. Your Commands/Code/Settings may technically be the solution but some explanation is welcome. Thanks in advance.– HBruijnNov 16, 2016 at 11:33
tar output to
stdout and then pipe it through
% tar cvf - _file_ | bzip2 _opts_ > output.tar.bz2
Its even easier:
% tar -cvf dir.tar path/to/dir/ && bzip2 -9 dir.tar
4Using a temporary file means you need enough hard disk space, plus bandwidth for tar to write and bzip2 to read it. This may seem trivial for small amounts of data, but when the directory in question has several hundred gigabytes, it may become a real problem. Jan 10, 2013 at 10:51
Yes, thanks. I know learned the deeper reason why
-j. These options seemed rather convenient to me. But they can save the day. Jul 4, 2015 at 9:54