Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there any reason to keep logging for a very high volume site? What valuable data is there to be had in the raw logs.

We do keep the logs today for "just in case" could someone provide an answer for "Just in case What?"

share|improve this question
    
Which logs? access logs or error logs? –  geeklin Aug 19 '09 at 17:59
    
BTW they compress really well. 7zip them. –  Alan Aug 19 '09 at 19:20
    
any logs? all logs? we do compress them now..but that takes up proc, mem, and disk to do..so if I don't have a compelling reason I don't want to. good answers and comments! –  Thomas Denton Aug 20 '09 at 0:46

4 Answers 4

up vote 3 down vote accepted

Try keeping your logs for at least 1/2 weeks. You can use tools like OSSEC to analyze them in real time and only store what is security relevant (multiple 400's, 500's error codes, scans, etc).

Take a look at this document for some ideas on the security value of keeping these logs:

http://www.infosecwriters.com/texts.php?op=display&id=453

share|improve this answer

You want to keep your logs, but possibly look at aggregating. I use Analog to produce weekly and monthly reports that are archived. Some historic comparisons of 404's has lead to fixing a broken change to my site.

I keep several months of IIS logs, but I've got years of montly Analog reports that have been produced from them.

Total size of my logs + reports is trivial - under 50 MB.

share|improve this answer

Assuming you track traffic elsewhere, I would suggest your most important use would be to backtrack unusual activity that may not be readily seen in pure traffic monitors. For example, many requests to the login page from one IP, or a large number of 404 errors to /admin, /administration, /config, etc.

share|improve this answer

Logs can really come in handy in certain situations. On-going situations are trending reports which can show you page view or traffic growth or referrer traffic (which search engine is sending traffic to you). If you have really large log files, it's useful to process these often and just retain the summary that your reporting tool retains.

The other thing that is useful is historical data if you need to look it up. This comes in handy if you have a sql injection attack, any other type of attack or even a DDOS where you want to track down when it started and if the zombie computers are coming from a certain region. In these situations, you'll be very thankful that you keep the logs around.

Additionally, if you have PCI or other compliance requirements, it may be a requirement to retain a certain amount of logging data.

You can turn off logging for certain folders. For example if you have a load balancer or monitoring software, you can turn off logging on the folder that it tests against. You can also turn off your \images folder or other folders that may get a lot of traffic but that don't need to be logged.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.