Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a bunch of files I routinely need to tar, encrypt with gpg, and then compress. This is on a Linus VPS server, so memory is more of a consideration than speed of execution (which I really don't care about).

If I do the three steps as one command (tar | gpg | bzip2 > output.tar.gpg.bzip2) is that going to consume more memory than if I first call tar, then call gpg, then call bzip2?

The files are potentially quite large (hundreds of megabytes/gigabytes)

share|improve this question
6  
Note you will achieve far better compression ratios if you compress before encrypting. –  Joris Apr 9 '11 at 4:31
    
Not only do you achieve greater compression ratios, but it's also theoretically more secure to do compression before encryption. This is not necessarily true of the bzip2 compression, however. –  Sarah Apr 9 '11 at 6:49
    
Unrelated note: you might be interested in pbzip2, it can use multiple cores to speed up compression. –  liori Apr 9 '11 at 11:47
    
@Joris: gpg compresses it, so using bzip2 first is not important. –  Olli Apr 9 '11 at 14:39

4 Answers 4

The more programs running simultaneously, the more memory it will require. But it is a trade-off, using pipes instead of running each program separately tends to require less disk space. You only need storage for the initial input and final output. If you are piping through ssh or something similar to store the data elsewhere, you may not even need initial disk space beyond the input file. Another feature with using pipes is that data is not processed at any stage faster than the slowest stage in the pipe.

The OpenPGP format that GnuPG uses supports compression natively. This happens before encryption so it is more secure and far more efficient than post-encryption compression. Also, it will be auto-detected on decryption so you won't have to worry about adding it to the pipeline. It may also require less memory than running a separate compression program. GnuPG supports Zip, Zlib (Gzip), and Bzip2 compression.

share|improve this answer

When you do that ( tar | gpg | bzip2 > output.tar.gpg.bzip2) you are piping commands, that means you are running all commands at the same time and redirecting the outputs of each command into the input of the following command (until the last output you redirect into the file). So tar output goes to gpg input that outputs to bzip2 input that finally outputs into a file.

So, when you pipe you use more memory because you are running all commands at the same time. You also use more processing power since gpg and bzip2 are two cpu hungry programs.

share|improve this answer
    
Thanks...I guess what I mean is that let's say the tar before compression is 1GB. If I pipe tar to gpg to bzip2, is it going to try to do everything in memory (potentially using 3GB of RAM), or is it going to stage it in the final output.tar.gpg.bzip2. Does that question make sense? –  raindog Apr 9 '11 at 4:04
    
It's kind of a stream of data passing through, but it can potentially use lots of memory. –  coredump Apr 9 '11 at 4:22

You should check that bzip2 is doing something worthwhile with the size, pgp data should end up looking like a (completely) random stream, and thus shouldn't really be compressible.

share|improve this answer

All these commands: tar, bzip2 and gpg work in a a stream-like way: they take a chunk of data (let say one megabyte), transform it and push it to the next stage. After that, RAM is reused to take care of another chunk of data. So even if you have a terabyte of data to process, it will be processed piece-by-piece.

Now, when you call tar by itself, tar will work this way: read one chunk of data from disk, wrap it with some headers necessary for archiving, push it to disk and forget about it. And again. And again. In this process, you'll never need more than one chunk of data in RAM. I don't know how big will tars buffer be, but it will be probably few megabytes only. So you will need only few megabytes of RAM to make it work.

gpg and bzip2 work the same way. gpg takes a small chunk of data, encrypts it and pushes forward -- all in a loop. bzip2 takes a small chunk of data, compresses it and pushes forward -- all in a loop.

You will need less RAM if you will call these commands without a pipeline: you will need to keep only one chunk of data at the same time in RAM. With a pipeline, chunks of data will be passed from one command to another, keeping it in RAM: you will need few chunks of data: one for tar, one for gpg, one for bzip2. So several megabytes tops. You will not need RAM to store all data at once.

With pipelining you can actually get another benefit: speed. You don't need to store temporary data on disk. Chunks of data get passed from tar to bzip2 and from bzip2 to gpg using RAM only. Compare: 3GB of data, 3GB of data in a tar file, 1GB of data in a tar.bzip2 file, 1GB of data in a tar.bzip2.gpg file. Even if you don't store them at the same time (f.e. removing source files in the process), you still need to write and read 4GB more data to disk than in a pipeline version. And VPSes are usually quite slow at disk operations, so a pipeline can spare you lot of time.

(Note: this explanation is simplified. All these commands do keep some additional data in RAM, but this is rarely significant. Also, you can often change size of buffers in specific commands.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.