Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have been using logrotation for years and never thought too much of it being a problem until I came across a question on stackoverflow (http://stackoverflow.com/questions/1508734/disable-java-log-rotation/) where someone wants to disable log rotation.

To me with experience in having build server and even production servers cleaned up manually because logs are not rotated and discs are running out and suddenly machines come to a halt that all seems crazy, but it occurred to me that maybe it is not so obvious after all.

So what are the benefits of log rotation? And what are the drawbacks (e.g. more difficult to debug/analyze maybe)? What tools do you find useful for working with rotated log files? Splunk I assume, but what else?

share|improve this question
add comment

3 Answers

up vote 6 down vote accepted

I think the benefits of log rotation are clear:

  1. You get easily managed smaller log files instead of one huge log file.
  2. You don't run out of disk space suddenly if you configure it appropriately according to your capacity limits. (size option)
  3. Older log files can be compressed so that log files get even smaller in size, and thus saving more disk space. (compress option).
  4. You can rotate the files in a specific way / time. For example, each log file contains only information related to a specific day. This will make the search easier given that you know the date. When you don't know the date, you can just search all files or a subset of them. (daily, monthly, etc).
  5. You automatically get rid of very old files. For example, you can keep 30 files at max. (rotate 30).
  6. You can add the extension you like such as rotation date. (dateext).
  7. You can execute specific scripts before/after rotation. (prerotate, postrotate).

EDIT: I added more items to the list and included the options when applicable. For more details, man logrotate can be consulted.

share|improve this answer
    
And for httpd server, you forgot to say that it is possible to move them on another disk for further use, e.g. statistics (webalizer is your friend), without having to restart the server. –  Olivier Pons Dec 21 '10 at 9:58
    
Saving the logs in S3 would help you look at particular requests (easy grep) to check for things which are time based –  Sairam Dec 21 '10 at 11:12
    
Writing into the same file which goes to few Gigs over the course of few days and that too multiple boxes is bad. You always want to split them up in chunks which are meaningful. Think about Google's servers which log few Gigs / Tera bytes every hour . –  Sairam Dec 21 '10 at 11:14
add comment

Benefits:

  • if log files are big, you can compress or delete old data to avoid filling up the disk
  • if you know when something happened, it'll be quicker grep a single day's log file than a single perpetual log file

Drawbacks:

  • if you want to process the entire history, you'll have to specify multiple file names
  • some programs don't support rotation, and if you rotate it using a utility like logrotate, there are some edge cases to watch out for (e.g. you have to use the truncate option, which I think means you risk losing a tiny amount of data)
share|improve this answer
add comment

If you think about what you use logs for, the benefits become even more clear. Leaving aside the obvious points mentioned by others and yourself. For me the big benefit is that I can control how the logs are fed into offline analysis tools.

I've got an assortment of home-grown scripts I use for analysing logs - but, AFAIK, splunk and the various webloog analysis tools are the only 'standard' tools available off the shelf. (not counting realtime log file analyzers like fail2ban).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.