How long do you keep log files?

Question

I have an application which writes its log files in a special folder. Now I'd like to add a functionality to delete these logs after a defined period of time automatically. But how long should I keep the log files? What are "good" default values (7 or 180 days)? Or do you prefer other criteria (e.g. max. used disk space)?

Answer 1

There's no single answer. Factors to consider:

• legal requirements for retention or destruction
• company policies for retention and destruction
• how long the logs are useful
• what questions you're hoping to answer from the logs
• how much space they take up

I've got some logs that roll over in days, and others that are kept for years.

I tend to keep access logs longer (in some cases, forever, once I've stripped identifying information out of them) than error logs as in theory, I don't need them once I've fixed the problems.

I keep some other debugging logs (showing user activity, how they're interacting with the system, etc.) around so we can see how things change with each version.

...

And this reminds me of anecdote when I worked for a university -- management brought in an outside consultant to do a third-party review of the webserver we were building. I was told to hand over 3 months of logs for them to review if we had sized the system appropriately. I knew that was a sign they didn't know what the hell they were doing, as universities are cyclic -- webserver load trended up over time, but with spikes at the beginning of each year, finals time, etc.

Answer 2

Depends on the requirements... there can be legal requirements for record keeping, and also you have to factor in how long they will be helpful for troubleshooting.

Answer 3

In addition to legal requirements and utility considerations, it depends quite a bit on how much disk space your logs are chewing up. I've seen very verbose programs rotate their logs every day, while very quiet programs rotate almost never.

Depending on your system, it might be worth considering the "logrotate" package instead of handling log rotation internally. Just place a file with reasonable defaults in /etc/logrotate.d/ and let your users modify it if they deem necessary.

Answer 4

It depends on the industry that will be using your software. There are lots of production applications that will fall under government regulatory requirements that may require any outputs, including log files, to be kept for a specific period of time.

If your application falls into one of these areas or you are unsure, you should consult your companies legal department.

Answer 5

You should for sure archive your logs before you run out of space. :)

In our organization, logs are archived daily. So the special "logging" folder will have only logs for current day. The archive is retained for 3 weeks on disk on a special archive location with lots of storage. This will help prod. support devs to go look if something gets reported. This is backed up to tape(yes tapes!) after 3 weeks. As a policy tapes are retained for another 35 days.

Answer 6

I'm running into this same decision currently. I have an app that writes non-critical exceptions to a log file and what I've settled on is setting a configurable size threshold for the log file itself. Once the log file reaches a certain size it will be archived. The archive will be cleaned once every week. I'm also thinking of setting a "Growth rate" threshold so if the log fills up too quickly customer support will be notified. Not sure yet if that's overkill.

These logs are for support only so, in my case, there is no legal or business requirement to keep them past a week.

Answer 7

The retention of log files depends on the criticality of the data being logged and the actual size of the storage medium on which the logs are stored as well as the various compliance procedures in the geographical location where the Server is hosted.

There is no hard and fast rule to the number of days the logs files should be retained though logs for at least a month ( space willing) would not be a bad idea.

Storing old logs as tar.gz files is also a nice idea if space is a constraint.