Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have 30+ apache httpd servers, and are looking to perform analysis on the logs both for historical trending and near "real time" monitoring/alerting. I'm mainly interested in things like error rates (4xx/5xx), response time, overall request rate, etc. but it would also be very useful to pull out more compute-intensive statistics like unique client IPs and user agents per unit of time.

I'm leaning towards building this as a centralized collector/server/storage, and am also considering the possibility of storing non-apache logs (i.e. general syslog, firewall logs, etc.) in the same system.

Obviously a large part of this will probably have to be custom (at least the connection between pieces and the parsing/analysis we do), but I haven't been able to find much information on people who have done stuff like this, at least at shops smaller than Google/Facebook/etc. who can throw their log data into a hundred-node compute cluster and run Map/Reduce on it.

The main things I'm looking for are:

  • All open source
  • Some way of collecting logs from apache machines that isn't too resource-intensive, and transports them relatively quickly over the network
  • Some way of storing them (NoSQL? key-value store?) on the backend, for a given amount of time (and then rolling them up into historical averages)
  • In the middle of this, a way of graphing in near-real-time (probably also with some statistical analysis on it) and hopefully alerting off of those graphs.

Any suggestions/pointers/ideas, to either "products"/projects or descriptions of how other people do this would be greatly helpful. Unfortunately, we're not exactly a new-age-y devops shop, lots of old stuff, homogeneous infrastructure, and strained boxes.

share|improve this question

closed as off topic by sysadmin1138 Oct 12 '12 at 19:20

Questions on Server Fault are expected to relate to server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Sorry, let me clarify that I'm looking for options that are both fully Open Source by license (OSI-approved) and are not purchased products (free as in beer). – Jason Antman Jun 9 '12 at 16:08
I appreciate all of the responses about rsyslog, and it's wonderful software - I've even helped Rainer find a bug or two in it. But I'm looking for something that allows quick (i.e. doesn't store logs as plain text) searching and analysis on logs. I want to know things like acceleration of rate of 404s/minute across 30 web servers, or how many unique user-agents hit my sites in the last 2 mintues. – Jason Antman Jun 9 '12 at 16:45
Also, in terms of scale (leaning towards solutions based on MongoDB, HBase, etc. especially for capped collections), my test/evaluation will be based on 8 hosts (out of ~60 total) that generate approx. 4.5GB/15M lines of Apache logs per day. For the real time analysis I can get away with ~1 day of raw data and periodic roll-ups beyond that, but I'd really like to be able to keep 1 week of logs... – Jason Antman Jun 9 '12 at 16:50
Unfortunately, product-recommendation questions, even just if-exists research, are off-topic per the FAQ. – sysadmin1138 Oct 12 '12 at 19:20

rsyslog can work pretty well, and if the amount of data that you are attempting to log is small enough you can even get away with using the free version of Splunk. The full version is probably a more comprehensive solution that maybe in line with what you want to accomplish saving you the time of developing your own in house monitoring tools.

At my work though we just stick to syslogd, Nagios, and Ganglia for all our monitoring needs as even with the 600 or so machines they are all incredibly stable.

share|improve this answer
Sounds very interesting. How do you feed syslog data into Nagios and Ganglia (specifically Ganglia that I'm interested in)? – Jason Antman Jun 9 '12 at 16:53
So we keep the syslog data separate from any Nagios or Ganglia data. Nagios data we keep around whereas Ganglia data just operates in RR mode and only keeps data for a few days. In the end if you really want to analyze the data and have things like automated reports, reports for the lifetime of a machine, it gets a bit harder to accomplish, and personally I feel it would be much easier and generally efficient to just pull different logs with syslog and roll your own monitoring system that can accomplish exactly what you need that is portable enough to be put on every server. It is funny – Wilshire Jun 13 '12 at 16:18

Jason, you mentioned an interest in using Ganglia to monitor your Apache web servers. Have you considered using mod-sflow with Ganglia?

Using Ganglia to monitor web farms


Recently, active, idle, max worker metrics have been added. While Ganglia is great for trending cluster metrics, you will need to use a log analyzer to report on the detailed log data. mod-sflow sends counter and log data as binary XDR encoded structures over UDP. You can use sflowtool to convert the binary data into standard ASCII logs, or as the basis of your own analysis tool.

share|improve this answer

If you're looking to set up a general purpose syslog server I'd definitely recommend you have a look at rsyslog, it's a very powerful modern syslog implementation. One of the things I like about it is that it can log to a relational database rather than to flat files, which makes data crunching a lot easier.

I've never used syslog with Apache, so I can't help with that part of your question unfortunately.

share|improve this answer
I've tried this approach before. The problem with relational databases is that when logging 20,000,000+ lines/day, even if the RDBMS can keep up with the write rate, purges (aside from partitioning on date and regenerating the table/partitioning every year) end up with an endless backlog. – Jason Antman Jun 9 '12 at 16:52

You can give LogZilla a try. It is 99% open source (one file is not). It is extremely fast and dirt cheap compared to the other solutions in this class.

share|improve this answer
Aside from not being open source, and probably not being able to do the generalized analysis I asked about (it's a commercial product - where's the Nagios plugin to alert on what I want), it's at least $4,300. That's pretty expensive for no budget... – Jason Antman Jun 9 '12 at 16:10

This isn't as general a solution as you asked for, but thinking back to a session from the London PHP conference, the BBC said they had a cunning way of transporting apache log files from many servers to a central server in real time, I think they nicknamed it teleportd.

I can't remember the exact details, but the jist was that they had a small auto-restarting script running on each of the apache servers that basically opened a fifo named pipe named /var/log/apache2/access_log and used netcat to copy it to a unique tcp port on the log servers. The log servers then catted it back out again into /var/log/myApacheServer/access_log.

With care, that approach would also work for general syslog files, although an initial sync during system startup might be required.

If you're ok with semi-real time then I'd pick a much simpler solution of rotating the log files every n minutes and rsyncing them to the central server in [postrotate].

Many webstats packages, like awstats and friends, assume that the log files are sorted, so something like awstats' might be a useful preprocessor on logServer:/var/log/*/access_log before you run whatever stats you require on the results.

Cacti would draw the graphs you seek using rrdtool, but you'd need to feed it from data grepped from the webstats internal data files, which is a tad unstructured for my tastes.

This approach is scriptable but would start to get tedious with large numbers of virtualhosts though, as you wind up with a vhosts*serverCount number of TCP streams.

It's all filesystem based though, so a bit low tech in today's world, sorry.

share|improve this answer
Actually I really like the simplicity of this approach, and there's some really good ideas here. The only problems for my use case are that 1) development, and a number of other groups, have extensive scripting based on daily-rotated log files, so I'd just need to log to 2 places simultaneously. The other issue, with grep, is that we want to use this to catch DoS, misbehaving web crawlers, backend failures (503), etc. in realtime, so grep doesn't scale for our needs. But very good ideas, thanks! – Jason Antman Jun 10 '12 at 15:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.