0

My apache log files are getting too big and I'm looking for ways to make them more manageable.

I know I can use conditional logging to only log access to specific types of files, but it seems to make more sense to log a random sample of the requests, so that I can still get an idea of what's going on without having to log every single request.

Is there something like that available?

I'm on ubuntu 8.04 with apache 2, and using cronolog for log rotating.

1
  • So you mean 'sample' in a statistically proper way, not the layperson's version, right?
    – Marcin
    Oct 26, 2009 at 15:20

3 Answers 3

1

I can think of three options to reduce the logfile size.

  1. Make the logs smaller my gzipping them. AFAIK this is a syslogd option.
  2. Only log things that may be useful. The cleanest way is to eliminate the request you know are not useful (like images, css, js etc.) based on their full URL (This way you keep the unexpected hack attempts).
  3. Make the Apache only log a subset of the traffic.

One possible way of doing this is the conditional logging you mentioned. Now conditional logging uses the SetEnvIf Apache feature. The actual syntax specs of SetEnvIf state:

 SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] ...

So how about using this to make an expression that only matches the 'even' (or 'odd') IP addresses of the Remote_Addr? You can cut it even further by limiting the IP ranges even further.

Of course you could also look at the reason for your question here: What makes the logfiles 'too big' and 'unmanageable'? What information to they hold for you?

2

Why not rotate log more often ? If each week rotation give you too much logs, turn them each day. If each day, turn them each hour. The problem of this solution is in log analyzers, like webalizer : they need to be configured accordingly.

I prefer to log everything, because when you have a problem, you never have too much informations. And with the actual disk prices, there is no issue of capacity for me.

3
  • Actually, the log files are so big that just running them through a log analyzer creates a huge strain on the system. And copying them over is even worse.
    – itsadok
    Oct 26, 2009 at 15:45
  • It is why I don't use a webalizer for stats: I prefer the image loaded in the html page. OK it works only on HTML (not download, etc) but it is lighter on CPU server. Why do you COPY them. Just MOVE them, then force-reload apache. It takes no time (just changing an inode), if you don't change of filesystem.
    – Dom
    Oct 26, 2009 at 15:56
  • I meant copy them to another server to analyze without hitting the CPU. If that sounds stupid it's because it is.
    – itsadok
    Oct 26, 2009 at 21:34
1

You can control the log format via the LogFormat directive.

If volume is the problem consider http://www.mrunix.net/webalizer/ which you can run off a cron job and produces nice graphics. I think it even looks inside older logs which have been zipped by logrorate.

Assuming there is a random distribution of error messages within the log file you could just print every 20th line in the log e.g.

perl -ne 'print unless (0 != $. % 20)' logfile.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .