Low-cost, Flexible Log Aggregation

Question

I'm starting to have quite the collection of Ubuntu VMs that I must manage. I'm starting to investigate Puppet for managing the configuration of all of them, and apticron to let me know what's out of date. But the issue I feel I should deal with sooner than later is log aggregation. I'd like to stay in the free/open source realm for now, seeing that we don't have much budget for something like splunk yet.

In addition to syslog, I would like to collect application specific logs (We are running different apps on different machines, from nginx+passenger for rails, to Apache+Tomcat for java, to PHP for expression engine, and mysql/postgresql database server), so that we can analyze the relavent data.

For now, I'm just looking to get all the logs one place.

Answer 1

Many applications can log to syslog, which means you can get the logs to a log server. apache, mysql, tomcat (log4j) can, at least.

Then you need a competent syslog server to do the aggregation. I use syslog-ng, but that's because it was the only serious alternative 7 years ago. Debian Lenny switched to rsyslog, which probably has a saner codebase and even more features.

In my experience, a good regex engine is the most important part of an aggregating syslog server. There is so much gorp you want to filter out so you can see the relevant parts. You can also point logwatch at the aggregated logs if you want to get started quickly.

EDIT: I should be explicit. Our strategy is to log everything from a specific host to one or more files in a folder for that host and simultaneously log to heavily filtered files that logs certain activities across all hosts. For example, there may be a file with failed logins across all hosts.

Answer 2

Another interesting project is Octopussy. It's an opensource log analyzer, alerter, and reporter. I haven't had a chance to set this up yet but I've heard some good things about it.

Someone else mentioned Zenoss. I use Zenoss and while it can alert based on logs I wouldn't say that's is it's strong point. It's primary task is monitoring and alerting based on snmp which is what I use it for.

Answer 3

For log aggregation and analysis you should try ECA (Event Correlation and Analysis) or SIEM (Security Information Event Manager). For example:

• AlienVault
• Zenoss
• Splunk

Answer 4

In abstract of analytics, Flume is a fantastic opensource log transport and aggregator. It has an active community and is backed by Cloudera.