Is there anyway to extract a tar.gz file faster than tar -zxvf filenamehere?
We have large files, and trying to optimize the operation.
3 Answers
pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our datasets. Once you install pigz, the tar file can be extracted with:
pigz -dc target.tar.gz | tar xf -
-
18+1. FWIW, you can also write that as
tar -xvf --use-compress-program=pigz filenamehere. (-zamounts to--use-compress-program=gzip.) Alternatively, you can even makegzipbe a symlink topigz, and keep using-zxvf.– ruakhNov 19, 2012 at 22:36 -
6@ruakh, I had to put
-xfafter--use-compress-program=pigz, or I got an error. For some reason, it was no faster than usinggzipthough.– jonderryMar 7, 2015 at 22:39 -
1For
bzip2there ispbzip2(pfor parallel).tar --use-compress-program=pbzip2 -xvf file.tar.bz2.– alfCSep 22, 2015 at 17:56 -
Is there a way to use the
pvcommand to show progress, or an equivilant, while also using the--use-compress-program=pigzflag? During compression, I can dognutar --use-compress-program="pigz | pv" -cf target.tar.gz YourData, but not sure how to do this during untar/uncompression. Jul 11, 2018 at 0:57 -
@StefanLasiewski You can use pigz with pv and tar in this way: "tar cf - /your/files | pv | pigz > compressed.tgz"– m_a_sFeb 17, 2022 at 16:49
if there are many many many small files in the tar ball, cancel the ‘v’ parameter, try again!
-
5I never use -v param. Don't know why people need that much noise in console.– EimantasMay 18, 2011 at 5:04
-
17@Eimantas When you untar something that contains many multi-gigabyte files, you will want some indication of progress. :) May 18, 2013 at 16:08
-
@TimHughes: that's really great to know, please post as a separate answer!– smciNov 28, 2017 at 9:59
-
Michael Hampton if you have a multi-gigabyte files but mixed with a big lists of small files you have a good reason to do not use -v, in my local tests it makes tar very slow specially if you have tar running in a remote server via terminal, what i do is to watch du -s directory so i can watch the directory growing... Mar 13, 2018 at 19:21
-
3It might be worth using
--checkpoint=NUMBER(display progress messages every NUMBERth record) instead of-v. Jun 21, 2018 at 18:42
If you want to see progress use something like pv.
Here is an Example:
pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -
-
Nice! You just made my day much more joyful when uncompressing tar archives of hundreds of gigabytes. Nov 2, 2021 at 23:52
$ tar -zxvfmethod is IO or CPU bound?vmstat 1 100or every 1 second, for 100 seconds, vmastat outputs. pigz was really helpful, I decompressed 108GB gz file in minutes that was taking over an hour previously.