Best Way to Examine Apache Logs on Ubuntu Server?

Question

I'm getting a lot of traffic that is crushing my tiny server. Is there something I can install that will allow me to examine my Apache traffic real-time? Ideally a web interface. I'd like to see what the requests are for and which ones are taking the most resources.

Answer 1

Apache's mod_status can help you. For more complex troubleshooting, you might need to customize format of your logs (e.g. include %D to it) and write some kind of parser.

Answer 2

Have you tried using Apache's extended status?

Answer 3

well you need to parse the apache logs, there are many tools to do that. I myself wrote a few awk scripts.

echo  "Hits by source IP:"
echo "======================================================================"

awk '{print $2}' "$1" | grep -ivE "(127.0.0.1|192.168.100.)" | sort | uniq -c | sort -   rn | head -25


echo "The 25 most popular pages:"
echo "======================================================================"

awk '{print $6}' "$1" | grep -ivE '(mod_status|favico|crossdomain|alive.txt)' | grep  -ivE '(.gif|.jpg|.png)' | sed 's/\/$//g' | sort | uniq -c | sort -rn | head -25

echo

echo "The 25 most popular pages (no js or css):"
echo "======================================================================"

awk '{print $6}' "$1" | grep -ivE '(mod_status|favico|crossdomain|alive.txt)' | grep -ivE '(.gif|.jpg|.png|.js|.css)' | sed 's/\/$//g' | sort | uniq -c | sort -rn | head -25


echo "The 25 most common referrer URLs:"
echo "======================================================================"

awk '{print $11}' "$1" | \
grep -vE "(^"-"$|/www.$host|/$host)" | \
sort | uniq -c | sort -rn | head -25



echo "Longest running requests"
echo "======================================================================"

awk  '{print $10,$6}' "$1" | grep -ivE '(.gif|.jpg|.png|.css|.js)'  | awk '{secs=0.000001*$1;req=$2;printf("%.2f minutes req time for %s\n", secs / 60,req )}' | sort -rn | head -50

exit 0

Just put in a file named: "apache_stats" and run it like: "./apache_stats ".

Answer 4

Check out http://mmonit.com/

M/Monit is paid, and offers a lot more features (and a more expansive GUI). Monit is free and allows you to monitor. You can get email alerts when your server load passes a certain threshold. I use this myself to monitor server load and respond accordingly.