40

Is there anyway to extract a tar.gz file faster than tar -zxvf filenamehere?

We have large files, and trying to optimize the operation.

  • 2
    Are you finding that the $ tar -zxvf method is IO or CPU bound? – EEAA May 18 '11 at 4:07
  • 1
    Believe CPU, how can I check though? – Justin May 18 '11 at 4:11
  • 5
    Not directly related, but 'z' hasn't been required since 2004/tar v1.1.5 gnu.org/software/tar/#TOCreleases :) – JamesHannah May 18 '11 at 12:06
55

pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our data sets. Once you install pigz the tar file can be extracted with:

pigz -dc target.tar.gz | tar xf -

  • 11
    +1. FWIW, you can also write that as tar -xvf --use-compress-program=pigz filenamehere. (-z amounts to --use-compress-program=gzip.) Alternatively, you can even make gzip be a symlink to pigz, and keep using -zxvf. – ruakh Nov 19 '12 at 22:36
  • 2
    @ruakh, I had to put -xf after --use-compress-program=pigz, or I got an error. For some reason, it was no faster than using gzip though. – jonderry Mar 7 '15 at 22:39
  • For bzip2 there is pbzip2 (p for parallel). tar --use-compress-program=pbzip2 -xvf file.tar.bz2. – alfC Sep 22 '15 at 17:56
  • Is there a way to use the pv command to show progress, or an equivilant, while also using the --use-compress-program=pigz flag? During compression, I can do gnutar --use-compress-program="pigz | pv" -cf target.tar.gz YourData, but not sure how to do this during untar/uncompression. – Stefan Lasiewski Jul 11 '18 at 0:57
13

if there are many many many small files in the tar ball, cancel the ‘v’ parameter, try again!

  • 3
    I never use -v param. Don't know why people need that much noise in console. – Eimantas May 18 '11 at 5:04
  • 9
    @Eimantas When you untar something that contains many multi-gigabyte files, you will want some indication of progress. :) – Michael Hampton May 18 '13 at 16:08
  • @TimHughes: that's really great to know, please post as a separate answer! – smci Nov 28 '17 at 9:59
  • Michael Hampton if you have a multi-gigabyte files but mixed with a big lists of small files you have a good reason to do not use -v, in my local tests it makes tar very slow specially if you have tar running in a remote server via terminal, what i do is to watch du -s directory so i can watch the directory growing... – Luciano Andress Martini Mar 13 '18 at 19:21
  • It might be worth using --checkpoint=NUMBER (display progress messages every NUMBERth record) instead of -v. – Stefan Lasiewski Jun 21 '18 at 18:42
6

If you want to see progress use something like pv. Here is an Example:

pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.