1

I am trying to download a large file which is .tar.gz and (> 2GB) in size. The bash script also does lots of other things, so I want to kick off the download and then continue processing other commands in the bash script. I am using wget and piping to tar and running it in the background with &.

wget -q "https://download-of-large-file-here.tar.gz" | tar xz &
# other commands here

However, this is breaking with:

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Any idea how to make this work?

Improve this question

1 Answer 1

Reset to default
8

You will have to instruct wget to write to STDOUT, you do this by passing -O - to wget, otherwise it will save the file to disk. tar is failing because as executed it's expecting something from STDIN.

This should work:

wget -q -O - "https://download-of-large-file-here.tar.gz" | tar xz &

Where -q suppresses the default progress information from STDOUT and -O - sets the output file to STDOUT.

Improve this answer
2
  • This works, but the wget and tar process never seem to terminate. See a screenshot from htop. i.imgur.com/xwHmg8D.png
    – Justin
    Aug 12, 2015 at 20:01
  • Does tar ever finish extracting? What happens when you run the commands separately? (e.g. wget -O "test.tar.gz" "https://download-of-large-file-here.tar.gz", then tar xz test.tar.gz)
    – Brandon DeRosier
    Aug 12, 2015 at 20:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .