Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We rotate and compress our Apache logs each day but it's become apparent that this isn't frequently enough. An uncompressed log is about 6G, which is getting close to filling our log partition (yep, we'll make it bigger in the future!) as well as taking a lot of time and CPU to compress each day. We have to produce a gziped log for each day for our stats processing. Obviously we could move our logs to a partition with more space but I also want to spread the compression overhead throughout the day.

Using Apache's rotatelogs we can rotate and compress the log more often -- hourly, say -- but how can I concatenate all the hourly compressed logs into a running compressed log for the day, without decompressing the previous logs? I don't want to uncompress 24 hours' worth of data and recompress it because that has all the disadvantages of our current solution.

Gzip doesn't seem to offer any append or concatenate option but perhaps I've missed something obvious. This question suggests straight shell concatenation "works" in that the archive can be decompressed but that gzip -l doesn't work seems a bit dodgy.

Alternatively, perhaps this is still a bad way to do things. Other suggestions are welcome -- our only constraints are our relatively small log partitions and the need to provide a daily compressed log.

share|improve this question

4 Answers 4

up vote 6 down vote accepted

The gzip man page should have what you want, but you can concatate them directly:

cat file1.gz >> file2.gz
gzip -c file1 >> file2.gz

Compression is not as good as if it was just one file compressed, but you can recover with:

zcat old.gz | gzip > new.gz
share|improve this answer
    
Duh! How did I miss that? Thanks. –  markdrayton Aug 19 '09 at 10:31

Just tar the gzipped files together. It's effectively a concatenate, and keeps them grouped logically together. The difference in file size between doing this and decompress/recompressing them together is virtually zero.

As in, with non-trivial log files, tar'ing 24 gzipped log files together will produce a file virtually identical in size to a single gzipped archive of all 24 original files.

share|improve this answer
    
Thanks, but we need a daily gziped log, not a tarball of 24 logs. It has to be one contiguous file. –  markdrayton Aug 19 '09 at 10:30

The CustomLog directive allows you to specify a command that the logs are piped into instead of the usual log file.

You could, for example, write a shell script that simply gzips everything it get's on stdin to a file you specify as argument:

#!/bin/sh
gzip -c >> $1

It is probably not a good idea to combine this with rotatelogs, since that might corrupt the archive, but you can relatively easily emulate its behaviour.

Then you configure Apache like so:

Customlog "|/usr/local/bin/gzip_log.sh /var/log/apache/access.log" combined

Test this! The buffering of gzip could be an issue.

share|improve this answer
    
Yeah, I thought about this but I'm a little leery of piping into gzip -- I worry what'd happen if Apache were to die, or some other problem occurred that meant gzip didn't close cleanly. Broken logs wouldn't please the management. –  markdrayton Aug 19 '09 at 10:29
    
Well, you wanted alternatives ;) –  hop Aug 19 '09 at 22:40

gzip doesn't care. You can concatenate gzipped files and it will be exactly as if you had concatenated them and then gzipped them.

$ echo foo |gzip >/tmp/log.gz
$ echo bar |gzip >>/tmp/log.gz
$ zcat /tmp/log.gz
foo
bar
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.