Web log analyzer that can generate a single report for multiple domains?

Question

Objective: I want to know what the most popular web sites are that are hosted on a Linux-based Apache server; minimal details would include number of visits, but it would help to know bandwidth, too, in useful date ranges (per day, per month, per year).

I've looked at awstats, webalizer, analog, visitors. These would do what I need, except that (according to my poking in the documentation) they all require setting a SiteDomain, thus limiting the report to stats for example.com.

I want multiple domains included in the report, perhaps utilizing Apache's %v log format option for including the virtualhost name. I don't want to know that / had 500 hits today; I want to know that example1.com had 300 hits and example2.com had 200 hits.

I assume analyzing Apache logs would provide me with a relatively accurate picture of this data. If you have another recommended method, I'm open to hearing it, but I basically want to know how the server's load is distributed across hosted domains. I would prefer an open source solution that works with Apache logs on Linux. Thanks for your help.

Answer 1

I guess you can achieve what you want by using something like the HostAliases config-option of awstats. maybe something like

HostAliases="REGEX[^wwww..*.com$]"

If all your domains are represented by this regular expression (start with www. and end with .com).

I don't know the directives/options for webalizer and the other tools, but i bet, most of them have a similar option.

Answer 2

this will print a single report with a summary for each vhost you have - it does not look like my work, I must have googled it somewhere: [I'll bet it was here]

#!/bin/bash
## Location of the virtual hosts directories
vhost_root=/var/www/vhosts/

## Various logwatch directories
dir_services="/usr/share/logwatch/plesk.conf/services"
dir_logfiles="/usr/share/logwatch/plesk.conf/logfiles"
dir_scripts="/usr/share/logwatch/scripts/services"
http_script="/usr/share/logwatch/scripts/services/http"

## Now iter over each directory
for domain in $( ls -Ichroot -Idefault $vhost_root ); do
if [ -d "${vhost_root}${domain}" ]
then
#echo "Making services logwatch enteries for ${domain}"
domain_us=`echo $domain | tr . _`

#echo "END_OF_SERVICES_CONF" >> ${dir_services}/http_${domain_us}.conf

echo "Title = httpd - $domain" > ${dir_services}/http_${domain_us}.conf
echo "LogFile = http_$domain_us" >> ${dir_services}/http_${domain_us}.conf
echo "\$HTTP_IGNORE_ERROR_HACKS = 0" >> ${dir_services}/http_${domain_us}.conf

#echo "END_OF_SERVICES_CONF" >> ${dir_services}/http_${domain_us}.conf

echo "Making logfiles logwatch entries for ${domain}"

#echo "END_OF_LOGFILES_CONF" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "LogFile = /var/www/vhosts/$domain/statistics/logs/access_log" > ${dir_logfiles}/http_${domain_us}.conf;
echo "LogFile = /var/www/vhosts/$domain/statistics/logs/access_log.processed" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "LogFile = /var/www/vhosts/$domain/statistics/logs/access_ssl_log" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "LogFile = /var/www/vhosts/$domain/statistics/logs/access_ssl_log.processed" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "Archive = /var/www/vhosts/$domain/statistics/logs/access_log.processed.?.gz" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "Archive = /var/www/vhosts/$domain/statistics/logs/access_ssl__log.processed.?.gz" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "*ExpandRepeats" >> ${dir_logfiles}/http_${domain_us}.conf;
echo "*ApplyhttpDate" >> ${dir_logfiles}/http_${domain_us}.conf;

## Make the script links
echo "Creating script link for ${domain}";
ln -s ${http_script} ${dir_scripts}/http_${domain_us};

fi;
done;

• if you need something more exciting than this, download & use cacti
• if you want accurate bandwidth, monitoring apache logs will not do it for you, logwatch/apache log tools only look at the size of the file that was accessed - lots of ways around that.

-sean

Answer 3

I'll suggest GoAccess. You can definitely get the stats you need in real time without setting up any sitedomain, it will also parse the log using virtualhost.