Log compression without indefinitely long lag?

Question

A common situation is that a program or another writes logging information to stdout, and due to its volume I pipe it through gzip on its way to disk.

Unfortunately gzip and most compression programs like it don't feel obligated to ever flush their output - and if the program freezes, even if it printed a lot of highly relevant debugging information to stdout just before that, and even if it flushed its stdout so gzip got it, gzip will hold to its data indefinitely.

I'd like something like gzip here (uncompressed data on stdin, compressed data to disk) - but that would just flush its output if it isn't getting anything for too long.

I understand that it's possible to build complicated logging system that supports all that - but with this tiny fix of periodic flushing such an extremely simple solution would work just fine.

Is there a program like that, or am I out of luck?

Answer 1

Normally, the logs are written to disk, then a daemon rotates the logs every so often, compressing the old log to save space. I'd recommend a similar tactic here.

Part of your problem is probably that gzip needs a certain amount of data to search for tokens before it can start compressing the stream. You might consider putting your log output through Huffman coding before writing to the logs if you want it to be compresses on output. There's Huffman is pretty easy to decode and there's utilities around like hffzip; plug ready made Huffman libraries to pull into your program.

Answer 2

If I recall the mechanics of it, gzip/bzip write a dictionary of the tokens used during compression, then, write the dictionary and the resulting file. Since your syslog process is accepting data but never tells your pipe that it has finished the file, your gzip/bzip process is waiting for that end of file so that it can do a quick dictionary build, compress and write the file.

If you could tell your syslog daemon to close your piped process once in a while, your gzip/bzip would process and write the file, but, now the problem you face is that you can't just append data to that compressed file. To my knowledge, there aren't any syslog daemons that support streaming compression. I know someone had written an experimental patch to syslog-ng, but a few cursory searches of Google don't find it.

As Chris mentions, it is more common to write the data, and compress it in the nightly/hourly rotation.

Answer 3

As there was no ready to use program for it, I wrote one myself.