Is there anyway to extract a tar.gz file faster than
tar -zxvf filenamehere?
We have large files, and trying to optimize the operation.
Stack Exchange network consists of 181 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.Visit Stack Exchange
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our datasets. Once you install pigz, the tar file can be extracted with:
pigz -dc target.tar.gz | tar xf -
tar -xvf --use-compress-program=pigz filenamehere. (
--use-compress-program=gzip.) Alternatively, you can even make
gzipbe a symlink to
pigz, and keep using
--use-compress-program=pigz, or I got an error. For some reason, it was no faster than using
tar --use-compress-program=pbzip2 -xvf file.tar.bz2.
pvcommand to show progress, or an equivilant, while also using the
--use-compress-program=pigzflag? During compression, I can do
gnutar --use-compress-program="pigz | pv" -cf target.tar.gz YourData, but not sure how to do this during untar/uncompression. Jul 11, 2018 at 0:57
if there are many many many small files in the tar ball, cancel the ‘v’ parameter, try again!
--checkpoint=NUMBER(display progress messages every NUMBERth record) instead of
-v. Jun 21, 2018 at 18:42
If you want to see progress use something like
Here is an Example:
pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -
$ tar -zxvfmethod is IO or CPU bound?
vmstat 1 100or every 1 second, for 100 seconds, vmastat outputs. pigz was really helpful, I decompressed 108GB gz file in minutes that was taking over an hour previously.