Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm looking for a log analysis tool to monitor & analyse the logs from an Apache & Tomcat webfarm.

It would need to have agents that run on the servers, and ship logs back to a central host. The host would then process them, and draw useful graphs such as requests/sec, average response time etc. The idea is to have one "console" webpage, which would present an overview of all the webapps graphs on one page, with the ability to drill into specific times on the graphs etc.

The budget for this is pretty low (read zero), so things like Splunk are out.

Currently the best candidate is Logstash & eleasticsearch, but it seems pretty complex to setup and I don't know if it can present a console view, the demos seem to be more based around searching rather than presenting an overview.

Cacti is also an option, but it would require some way of shipping the logs onto the central host, and writing custom scripts to parse the logs.

Is there another tool that could provide something like this which I've missed? It feels like a fairly common requirement, so I'm surprised I've found nothing so far that does it directly!

To clarify - Things like AWStats & Webalizer aren't really suitable (and pretty dated now as well!). I'm more interested in the health of my servers, than where my visitors came from.

share|improve this question

closed as off-topic by Dennis Kaarsemaker, Jenny D, Dave M, mdpc, Tim Brigham Sep 24 '13 at 19:12

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve." – Dennis Kaarsemaker, Jenny D, Dave M, mdpc, Tim Brigham
If this question can be reworded to fit the rules in the help center, please edit the question.

    
Check out Kibana (elasticsearch.org/overview/kibana) - I've not used it, but it seems to be able to do what you want. –  Jenny D Sep 24 '13 at 10:08

2 Answers 2

up vote 1 down vote accepted

I've had great success with Logstash and Elasticsearch myself, and I would put it at 'low to medium' levels of complexity. Check out the 30-minute video on the Logstash homepage, or drop $15 on the Logstash eBook. Both were very helpful in getting my environment set up such that Logstash parses the logs, throws them to Elasticsearch, which Graphite then queries to build charts-n-graphs.

Alternatively, check out Piwik - it's ostensibly a replacement for Google Analytics, but I believe they have a local-file-parse mode that is similar to how awstats works. You could set up rsyslog to throw all your http logs to a central server which is then parsed by Piwik in local-file mode, giving you your 'modern' interface.

On the other-other hand, you might find Newrelic more to your liking. It's not free (as in either beer or speech), but they are much more reasonably priced than Splunk.

share|improve this answer
    
Thanks, I'm going to mark this as correct for the heads up for New Relic and graphite. Think I'll give Logstash/elasticsearch/graphite a go, especially if I can find a prebuilt VM to demo with. –  Mark Sep 24 '13 at 17:45

To clarify: I wouldn't call things like AWStats & Webalizer "pretty dated". They sound almost EXACTLY like what you need. IS there something that these products don't do that you need?

You seem to be unwilling or at least very hesitant to do any scripting (as per your cacti comments) so a simple out of the box solution like the two you ruled out seem ideal.

If you are set in stone on not using the above solutions then mod_apache_snmp & cacti should give you exactly what you want. http://mod-apache-snmp.sourceforge.net/english/

Please note: in any product like this, there will be setup involved.

share|improve this answer
    
I started using Webalizer and AWStats about 8years ago, and they don't seem to have changed much since then! The Webalizer screenshots look pretty old - webalizer.org/sample Also AFAIK one thing neither of them can do is collate logs from multiple servers onto one host, and show average page response times etc. –  Mark Sep 24 '13 at 17:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.