How to determine concurrent user counts from apache logs

Question

We have 3 months of Apache logs and we need to determine what is the maximum number of concurrent connections that we've received.

I've used AWStats and I don't think this is something that it generates. Is there another tool that will give us this kind of information from the logs?

Answer 1

You can't do it with the default common or combined log formats without baking in some assumptions. For instance if you assume that all your responses take 500ms, then you can take the number of log-lines per second and divide in half.

What would probably be easiest is to use something like munin or cacti to poll the values from mod_status. Or you could watch netstat output for a couple dozen samples over the next few days, then use that to estimate a rough ratio of concurrent connections to traffic for your site.

Answer 2

You can look at the output of netstat -n -t over time. The number of connections ESTABLISHED on port 80 is the current number of concurrent connections (roughly). There are a number of tools that will collect this sort of data for you; we use Ganglia, which will give you a variety of network statistics if you use the "tcpconn" module.

Answer 3

Not that I'm aware of - the best solution would be polling netstat or looking at server-status, however this is not very convenient for historical data.

If you've only got the standard log files, then there's not a lot of information there either (unless your site is always very, very, VERY slow) but you could at least try to calculate an estimate of concurrent connections. You'd need to sort the file by %t then go through each record working out the overlap. It's not a trivial bit of code.

A simpler approach you could simply look at the number of hits per hour multiply by the average response time then divide by the number of seconds in an hour. Of course this assumes that all requests are evenly spaced out - but you could work out a scaling factor (by measuring the current number of connections as described above). But of course if you're only recording %T (rather than %D), then the rounding will destroy any sort of accuracy in your estimate (again unless your site is horrendously slow).