Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

A common situation is that a program or another writes logging information to stdout, and due to its volume I pipe it through gzip on its way to disk.

Unfortunately gzip and most compression programs like it don't feel obligated to ever flush their output - and if the program freezes, even if it printed a lot of highly relevant debugging information to stdout just before that, and even if it flushed its stdout so gzip got it, gzip will hold to its data indefinitely.

I'd like something like gzip here (uncompressed data on stdin, compressed data to disk) - but that would just flush its output if it isn't getting anything for too long.

I understand that it's possible to build complicated logging system that supports all that - but with this tiny fix of periodic flushing such an extremely simple solution would work just fine.

Is there a program like that, or am I out of luck?

share|improve this question

3 Answers 3

Normally, the logs are written to disk, then a daemon rotates the logs every so often, compressing the old log to save space. I'd recommend a similar tactic here.

Part of your problem is probably that gzip needs a certain amount of data to search for tokens before it can start compressing the stream. You might consider putting your log output through Huffman coding before writing to the logs if you want it to be compresses on output. There's Huffman is pretty easy to decode and there's utilities around like hffzip; plug ready made Huffman libraries to pull into your program.

share|improve this answer

If I recall the mechanics of it, gzip/bzip write a dictionary of the tokens used during compression, then, write the dictionary and the resulting file. Since your syslog process is accepting data but never tells your pipe that it has finished the file, your gzip/bzip process is waiting for that end of file so that it can do a quick dictionary build, compress and write the file.

If you could tell your syslog daemon to close your piped process once in a while, your gzip/bzip would process and write the file, but, now the problem you face is that you can't just append data to that compressed file. To my knowledge, there aren't any syslog daemons that support streaming compression. I know someone had written an experimental patch to syslog-ng, but a few cursory searches of Google don't find it.

As Chris mentions, it is more common to write the data, and compress it in the nightly/hourly rotation.

share|improve this answer
    
bzip2 is a totally different story as it works with 100kB-900kB blocks at a time, but gzip is a straightforward stream compressor and can easily flush at any byte. Here's manual for the underlying zlib library which contains gzflush function. The problem is that this existing underlying functionality doesn't seem to be exposed by gzip program. –  taw Jul 7 '10 at 5:52
up vote 0 down vote accepted

As there was no ready to use program for it, I wrote one myself.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.