Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm starting to have quite the collection of Ubuntu VMs that I must manage. I'm starting to investigate Puppet for managing the configuration of all of them, and apticron to let me know what's out of date. But the issue I feel I should deal with sooner than later is log aggregation. I'd like to stay in the free/open source realm for now, seeing that we don't have much budget for something like splunk yet.

In addition to syslog, I would like to collect application specific logs (We are running different apps on different machines, from nginx+passenger for rails, to Apache+Tomcat for java, to PHP for expression engine, and mysql/postgresql database server), so that we can analyze the relavent data.

For now, I'm just looking to get all the logs one place.

share|improve this question

closed as not constructive by Chris S Sep 4 '12 at 2:42

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

Shopping Questions are Off-Topic on any of the Stack Exchange sites. See Q&A is hard, lets go Shopping and the FAQ for more details. – Chris S Sep 4 '12 at 2:42
up vote 8 down vote accepted

Many applications can log to syslog, which means you can get the logs to a log server. apache, mysql, tomcat (log4j) can, at least.

Then you need a competent syslog server to do the aggregation. I use syslog-ng, but that's because it was the only serious alternative 7 years ago. Debian Lenny switched to rsyslog, which probably has a saner codebase and even more features.

In my experience, a good regex engine is the most important part of an aggregating syslog server. There is so much gorp you want to filter out so you can see the relevant parts. You can also point logwatch at the aggregated logs if you want to get started quickly.

EDIT: I should be explicit. Our strategy is to log everything from a specific host to one or more files in a folder for that host and simultaneously log to heavily filtered files that logs certain activities across all hosts. For example, there may be a file with failed logins across all hosts.

share|improve this answer
Interesting. Thanks. – Martijn Heemels Mar 28 '11 at 21:55

Another interesting project is Octopussy. It's an opensource log analyzer, alerter, and reporter. I haven't had a chance to set this up yet but I've heard some good things about it.

Someone else mentioned Zenoss. I use Zenoss and while it can alert based on logs I wouldn't say that's is it's strong point. It's primary task is monitoring and alerting based on snmp which is what I use it for.

share|improve this answer

LogZilla is highly scalable (hundreds of millions of events) and is 1/10 of the cost of the other software in it's class. It's also much easier to use.

share|improve this answer

For log aggregation and analysis you should try ECA (Event Correlation and Analysis) or SIEM (Security Information Event Manager). For example:

share|improve this answer

In abstract of analytics, Flume is a fantastic opensource log transport and aggregator. It has an active community and is backed by Cloudera.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.