56

Is there anyway to extract a tar.gz file faster than tar -zxvf filenamehere?

We have large files, and trying to optimize the operation.

4
  • 2
    Are you finding that the $ tar -zxvf method is IO or CPU bound?
    – EEAA
    May 18 '11 at 4:07
  • 2
    Believe CPU, how can I check though?
    – Justin
    May 18 '11 at 4:11
  • 7
    Not directly related, but 'z' hasn't been required since 2004/tar v1.1.5 gnu.org/software/tar/#TOCreleases :) May 18 '11 at 12:06
  • @Justin You might have to install it, but vmstat will tell you about IO or CPU loading. vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity you can even run it as a continual process, vmstat 1 100 or every 1 second, for 100 seconds, vmastat outputs. pigz was really helpful, I decompressed 108GB gz file in minutes that was taking over an hour previously.
    – j0h
    Dec 6 '20 at 4:25
77

pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our data sets. Once you install pigz the tar file can be extracted with:

pigz -dc target.tar.gz | tar xf -

4
  • 16
    +1. FWIW, you can also write that as tar -xvf --use-compress-program=pigz filenamehere. (-z amounts to --use-compress-program=gzip.) Alternatively, you can even make gzip be a symlink to pigz, and keep using -zxvf.
    – ruakh
    Nov 19 '12 at 22:36
  • 6
    @ruakh, I had to put -xf after --use-compress-program=pigz, or I got an error. For some reason, it was no faster than using gzip though.
    – jonderry
    Mar 7 '15 at 22:39
  • 1
    For bzip2 there is pbzip2 (p for parallel). tar --use-compress-program=pbzip2 -xvf file.tar.bz2.
    – alfC
    Sep 22 '15 at 17:56
  • Is there a way to use the pv command to show progress, or an equivilant, while also using the --use-compress-program=pigz flag? During compression, I can do gnutar --use-compress-program="pigz | pv" -cf target.tar.gz YourData, but not sure how to do this during untar/uncompression. Jul 11 '18 at 0:57
15

if there are many many many small files in the tar ball, cancel the ‘v’ parameter, try again!

5
  • 4
    I never use -v param. Don't know why people need that much noise in console.
    – Eimantas
    May 18 '11 at 5:04
  • 15
    @Eimantas When you untar something that contains many multi-gigabyte files, you will want some indication of progress. :) May 18 '13 at 16:08
  • @TimHughes: that's really great to know, please post as a separate answer!
    – smci
    Nov 28 '17 at 9:59
  • Michael Hampton if you have a multi-gigabyte files but mixed with a big lists of small files you have a good reason to do not use -v, in my local tests it makes tar very slow specially if you have tar running in a remote server via terminal, what i do is to watch du -s directory so i can watch the directory growing... Mar 13 '18 at 19:21
  • 2
    It might be worth using --checkpoint=NUMBER (display progress messages every NUMBERth record) instead of -v. Jun 21 '18 at 18:42
9

If you want to see progress use something like pv. Here is an Example:

pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -
1
  • Nice! You just made my day much more joyful when uncompressing tar archives of hundreds of gigabytes. Nov 2 '21 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.