Unzipping files that are flying in through a pipe

Question

Can I make unzip or any similar programs work on the standard output? The situation is I'm downloading a zip file, which is supposed to be unzipped on the fly.

Related issue: http://serverfault.com/questions/25779/how-do-i-pipe-a-downloaded-file-to-standard-output-in-bash

Answer 1

This is unlikely to work how you expect. Zip is not just a compression format, but also a container format. It rolls up the jobs of both tar and gzip.bzip2 into one. Having said that, if your zip has a single file, you can use unzip -p to extract the files to stdout. If you have more than one file, there's no way for you to tell where they start and stop.

As for reading from stdin, the unzip man page has this sentence:

Archives read from standard input are not yet supported, except with funzip (and then only the first member of the archive can be extracted).

You might have some luck with funzip.

Answer 2

What you want to do is, make unzip take a ZIPped file on its standard input rather than as an argument. This is usually easily supported by gzip and tar kind of tools with a - argument. But the standard unzip does not do that (though, it does support extraction to a pipe). However, all is not lost...

Look at funzip manual page.

funzip without a file argument acts as a filter; that is, it assumes that a ZIP archive (or a gzip'd file) is being piped into standard input, and it extracts the first member from the archive to stdout. When stdin comes from a tty device, funzip assumes that this cannot be a stream of (binary) compressed data and shows a short help text, instead. If there is a file argument, then input is read from the specified file instead of from stdin.

Given the limitation on single-member extraction, funzip is most useful in conjunction with a secondary archiver program such as tar(1). The following section includes an example illustrating this usage in the case of disk backups to tape.

This goes well with the idea that most linux archives are usually TAR'ed and then ZIPped in some way (gzip, bzip, et al). This will work for you if you have a tar.ZIP.

---

It is worth noting that funzip is written by Info-ZIP original author Mark Adler. He writes in the funzip man page,

this functionality should be incorporated into unzip itself (future release).

however, no such update is seen around. I suspect that Mark found it unnecessary since other archiving methods worked easily with TAR.

Answer 3

It's not possible with Info-Zip which is the most common OSS implementation. More importantly though, it's not recommended due to the constructs of ZIP archives.

If a change of format is viable to you then consider using tar(1) instead. It is quite happy with streamed input/output and, in fact, expects it by default.

Additionally you can often tell whether applications expect streamed input/output by specifying "-" for a filename. Info-Zip, as you can imagine, doesn't treat this as a valid argument.

Answer 4

While a zip file is in fact a container format, there's no reason why it can't be read as a stream if the file can fit into memory easily enough. Here's a Python script that takes a zip file as standard input and extracts the contents to the current directory or to a specified directory if specified.

import zipfile
import sys
import StringIO
data = StringIO.StringIO(sys.stdin.read())
z = zipfile.ZipFile(data)
dest = sys.argv[1] if len(sys.argv) == 2 else '.'
z.extractall(dest)

This script can be minified to one line and created as an alias.

alias unzip-stream="python -c \"import zipfile,sys,StringIO;zipfile.ZipFile(StringIO.StringIO(sys.stdin.read())).extractall(sys.argv[1] if len(sys.argv) == 2 else '.')\""

Now unzip your stream easily.

wget http://your.domain.com/your/file.zip -O - | unzip-stream target_dir