3

Do you place any limits on the size of Apache log files - access.log and error.log?

Specifically, can you give:

  • Reasons to limit log file sizes
    • Disk space
    • Any other?
  • Reasons NOT to limit log file sizes
    • Research into performance issues or security breaches
    • Any other?
  • Methods of doing so
    • Cron job that periodically deletes the file, or the first N lines?
    • Any other?
  • Anything you might salvage before deleting
    • For example, grep out how many times a file was downloaded before deleting the access logs

I'd like get the thoughts of experienced sysadmins before I do anything.

(Marking as community wiki since this may be a matter of opinion.)

6 Answers 6

1

You should approach it in the other way, instead of limiting these log files, work out a system where there is always enough log space so that they can never fill the disk.

My primary way of doing this is simply calculating how much space is likely to be needed, and giving it lots of space, but I've also got a script which will check for diskspace in the logs directory, and if it's getting tight will automatically rotate the log files, compress old versions, and erase those which are old enough that they'll be on archived media.

1

Well, the primary reasons for not limiting the log sizes are:

  • provides a useful audit log
  • provides a detailed access log
  • serves as a database for future analysis:
    • other website referrals
    • internal navigation path
    • internal randomly chosen content log
    • useful for debugging slow-to-report bugs

The major reason for limiting log size is to not overwhelm disk space. But that's pretty lame—1.5 TB drives now cost about $120.

1
  • So there are no performance issues associated with giant log files? Apr 2, 2010 at 17:39
1

I would recommend thinking in terms of log rotation tailored to fit your log file sizes. If you have an extremely heavily utilized web site generating a large amount of log entries rotate based on a file size that is small enough to work with effectively in a text editor or whatever you view logs with and keep logs small enough to compress and send off to others for additional troubleshooting purposes. If you are generating a small amount of log entries a personal preference is to rotate on a daily basis so I can easily hone in on errors. Plus with daily rotations I can see spikes in utilization and errors based on file sizes.

Log retention should be driven by personal needs, statistical analysis needs, or corporate standards/regulations. If you want to parse the logs and look for usage patterns, statistics or for auditing purposes you may want to retain a considerable amount of log files. Log files compress to amazingly small sizes so it is easy to retain a large number of them.

We setup automated compression -> archival and eventually deletion processes based on our retention policies to keep system administration down to a minimum.

0

But why delete them when you can simply bzip2/tar/cpio them.
Text files have a great compression ratio.
Or just archive them to tapes (Cheap storage and long-lasting) with or without compression

0

Limit the size before you rotate. Smaller raw files will burn less time on your webserver while they're being compressed.

How many of the rotated logs you want to keep around is a separate question.

0

On a production machine, I don't delete them at all, just archive them as mentioned above.

But on a development machine, the logs are only useful for a short time so it makes sense to stop them from filling up too much disk space -- especially on a machine where the root file system is on an SSD instead of a HDD (much smaller drive sizes).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .