1

I'm looking for a log analysis tool to monitor & analyse the logs from an Apache & Tomcat webfarm.

It would need to have agents that run on the servers, and ship logs back to a central host. The host would then process them, and draw useful graphs such as requests/sec, average response time etc. The idea is to have one "console" webpage, which would present an overview of all the webapps graphs on one page, with the ability to drill into specific times on the graphs etc.

The budget for this is pretty low (read zero), so things like Splunk are out.

Currently the best candidate is Logstash & eleasticsearch, but it seems pretty complex to setup and I don't know if it can present a console view, the demos seem to be more based around searching rather than presenting an overview.

Cacti is also an option, but it would require some way of shipping the logs onto the central host, and writing custom scripts to parse the logs.

Is there another tool that could provide something like this which I've missed? It feels like a fairly common requirement, so I'm surprised I've found nothing so far that does it directly!

To clarify - Things like AWStats & Webalizer aren't really suitable (and pretty dated now as well!). I'm more interested in the health of my servers, than where my visitors came from.

1

2 Answers 2

1

I've had great success with Logstash and Elasticsearch myself, and I would put it at 'low to medium' levels of complexity. Check out the 30-minute video on the Logstash homepage, or drop $15 on the Logstash eBook. Both were very helpful in getting my environment set up such that Logstash parses the logs, throws them to Elasticsearch, which Graphite then queries to build charts-n-graphs.

Alternatively, check out Piwik - it's ostensibly a replacement for Google Analytics, but I believe they have a local-file-parse mode that is similar to how awstats works. You could set up rsyslog to throw all your http logs to a central server which is then parsed by Piwik in local-file mode, giving you your 'modern' interface.

On the other-other hand, you might find Newrelic more to your liking. It's not free (as in either beer or speech), but they are much more reasonably priced than Splunk.

1
  • Thanks, I'm going to mark this as correct for the heads up for New Relic and graphite. Think I'll give Logstash/elasticsearch/graphite a go, especially if I can find a prebuilt VM to demo with.
    – Mark
    Sep 24, 2013 at 17:45
0

To clarify: I wouldn't call things like AWStats & Webalizer "pretty dated". They sound almost EXACTLY like what you need. IS there something that these products don't do that you need?

You seem to be unwilling or at least very hesitant to do any scripting (as per your cacti comments) so a simple out of the box solution like the two you ruled out seem ideal.

If you are set in stone on not using the above solutions then mod_apache_snmp & cacti should give you exactly what you want. http://mod-apache-snmp.sourceforge.net/english/

Please note: in any product like this, there will be setup involved.

1
  • I started using Webalizer and AWStats about 8years ago, and they don't seem to have changed much since then! The Webalizer screenshots look pretty old - webalizer.org/sample Also AFAIK one thing neither of them can do is collate logs from multiple servers onto one host, and show average page response times etc.
    – Mark
    Sep 24, 2013 at 17:47

Not the answer you're looking for? Browse other questions tagged .