Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an application which writes its log files in a special folder. Now I'd like to add a functionality to delete these logs after a defined period of time automatically. But how long should I keep the log files? What are "good" default values (7 or 180 days)? Or do you prefer other criteria (e.g. max. used disk space)?

share|improve this question

migrated from stackoverflow.com Apr 23 '10 at 20:18

This question came from our site for professional and enthusiast programmers.

3  
As long as YOU think you will need them, obviously. –  Neil Butterworth Apr 23 '10 at 19:39
    
how long do you want to be able to check to see if there was a problem? –  BlackICE Apr 23 '10 at 19:40
2  
What's actually being written to these files? –  Seventh Element Apr 23 '10 at 19:45
    
Do you have a good reason not to use the logging system of the OS, so you can let the end user decide for themselves, centralize logging via Splunk, etc.? –  hurfdurf Apr 23 '10 at 20:42
add comment

7 Answers 7

up vote 4 down vote accepted

There's no single answer. Factors to consider:

  • legal requirements for retention or destruction
  • company policies for retention and destruction
  • how long the logs are useful
  • what questions you're hoping to answer from the logs
  • how much space they take up

I've got some logs that roll over in days, and others that are kept for years.

I tend to keep access logs longer (in some cases, forever, once I've stripped identifying information out of them) than error logs as in theory, I don't need them once I've fixed the problems.

I keep some other debugging logs (showing user activity, how they're interacting with the system, etc.) around so we can see how things change with each version.

...

And this reminds me of anecdote when I worked for a university -- management brought in an outside consultant to do a third-party review of the webserver we were building. I was told to hand over 3 months of logs for them to review if we had sized the system appropriately. I knew that was a sign they didn't know what the hell they were doing, as universities are cyclic -- webserver load trended up over time, but with spikes at the beginning of each year, finals time, etc.

share|improve this answer
    
Do you separate the different type of log files? So do you have different log files for errors, debugging informations and user activity? –  Alex Apr 23 '10 at 20:09
    
Yes ... it'd suck if I'd have to strip lines out of logs for retention. I guess it's nice to be able to have stuff together in one file when debugging, but I can grep by user identifier / IP address when debugging, so I can correlate user activity to the errors. If nothing else, keeping similar info in each file means they compress better. –  Joe H. Apr 23 '10 at 20:45
add comment

Depends on the requirements... there can be legal requirements for record keeping, and also you have to factor in how long they will be helpful for troubleshooting.

share|improve this answer
add comment

In addition to legal requirements and utility considerations, it depends quite a bit on how much disk space your logs are chewing up. I've seen very verbose programs rotate their logs every day, while very quiet programs rotate almost never.

Depending on your system, it might be worth considering the "logrotate" package instead of handling log rotation internally. Just place a file with reasonable defaults in /etc/logrotate.d/ and let your users modify it if they deem necessary.

share|improve this answer
add comment

It depends on the industry that will be using your software. There are lots of production applications that will fall under government regulatory requirements that may require any outputs, including log files, to be kept for a specific period of time.

If your application falls into one of these areas or you are unsure, you should consult your companies legal department.

share|improve this answer
add comment

You should for sure archive your logs before you run out of space. :)

In our organization, logs are archived daily. So the special "logging" folder will have only logs for current day. The archive is retained for 3 weeks on disk on a special archive location with lots of storage. This will help prod. support devs to go look if something gets reported. This is backed up to tape(yes tapes!) after 3 weeks. As a policy tapes are retained for another 35 days.

share|improve this answer
add comment

I'm running into this same decision currently. I have an app that writes non-critical exceptions to a log file and what I've settled on is setting a configurable size threshold for the log file itself. Once the log file reaches a certain size it will be archived. The archive will be cleaned once every week. I'm also thinking of setting a "Growth rate" threshold so if the log fills up too quickly customer support will be notified. Not sure yet if that's overkill.

These logs are for support only so, in my case, there is no legal or business requirement to keep them past a week.

share|improve this answer
    
Do you have one big log file only? I create one file per day and log all types of informations (errors and debug informations). The size of the file depends on the current usage (I think max. 1MB per day). But I can imagine to increase log informations in the future. What threshold size did you choose and why? –  Alex Apr 23 '10 at 20:30
add comment

The retention of log files depends on the criticality of the data being logged and the actual size of the storage medium on which the logs are stored as well as the various compliance procedures in the geographical location where the Server is hosted.

There is no hard and fast rule to the number of days the logs files should be retained though logs for at least a month ( space willing) would not be a bad idea.

Storing old logs as tar.gz files is also a nice idea if space is a constraint.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.