We have a pair of load-balanced/clustered Apache 2.2.17 servers that function as reverse proxy servers for our ~100 public-facing websites, and we'd like to get a good idea of our incoming web traffic and which sites are using roughly how much bandwidth/throughput.
I set up Webalizer on them a couple days ago (with a custom modification for it to parse the log files with a wildcard), and that's basically what we want, except for the fact that it lumps everything in one bucket, differentiated only by URLs relative to the server root.
In other words, I see URLs like:
/index.html /images/image.png /assets/4942t426098420g24.xml
And that masks the fact that they might actually be:
http://domain1.com/index.html http://domain2.org/images/image.png http://domain3.net/assets/4942t426098420g24.xml
I've tried searching around for others with the same problem, but it seems that most people running multiple VirtualHosts on one server want the individual stats in separate Webalizer buckets-- I want them all in one bucket, but visually differentiated.
1. Different LogFormat
I did a little brainstorming this morning and realized that I could switch my LogFormat from "combined" to "vhost_combined", which includes the VirtualHost name at the beginning of the line, but Webalizer doesn't seem to know/care about it...
2. Hacking my log files
I've considered hacking up a BASH script that parses through the log files, grabs the VirtualHost from the front of the line, and inserts it into the Request portion, for example:
www.domain1.com:80 ip.ad.dr.ess - - [06/Jun/2013:11:16:42 -0600] "GET /index.html HTTP/1.1" 200 5224 "http://www.domain1.com/referrer.html" "Mozilla/4.0 (compatible; blah blah)"
www.domain1.com:80 ip.ad.dr.ess - - [06/Jun/2013:11:16:42 -0600] "GET www.domain1.com/index.html HTTP/1.1" 200 5224 "http://www.domain1.com/referrer.html" "Mozilla/4.0 (compatible; blah blah)"
But this is a pretty ugly hack, and to be honest, if it works, I'm not sure what it will give me... Sure, I'll be able to see the domains of the top URLs, but Webalizer won't know to break things down that way, so I won't get, for example, "Top Virtual Hosts" or anything like that.
3. Modifying OSS to suit
I really don't have the time to do this, but I imagine that I could take something like Webalizer or AWStats, crack open the source code to add functionality to have it respect the
%v:%p bit at the beginning of the log line, and then include a breakdown in the monthly summary page. And if I took the time to comment well and include it as a config file option, I might even submit it as a patch to the Webalizer community... But yeah, that's a little beyond the depth of this project. (I'm not even a very good C programmer.)
So is there a better package out there that will give me this kind of single "server-level" report that intelligently aggregates Apache logs in a way that I can see my total bandwidth/throughput/usage and a breakdown by VirtualHost? Or can I use something like Webalizer or AWStats in a clever way to get what I'm looking for?
All outside-the-box solutions will be seriously considered.
Thanks in advance.