2

We have 30+ apache httpd servers, and are looking to perform analysis on the logs both for historical trending and near "real time" monitoring/alerting. I'm mainly interested in things like error rates (4xx/5xx), response time, overall request rate, etc. but it would also be very useful to pull out more compute-intensive statistics like unique client IPs and user agents per unit of time.

I'm leaning towards building this as a centralized collector/server/storage, and am also considering the possibility of storing non-apache logs (i.e. general syslog, firewall logs, etc.) in the same system.

Obviously a large part of this will probably have to be custom (at least the connection between pieces and the parsing/analysis we do), but I haven't been able to find much information on people who have done stuff like this, at least at shops smaller than Google/Facebook/etc. who can throw their log data into a hundred-node compute cluster and run Map/Reduce on it.

The main things I'm looking for are:

  • All open source
  • Some way of collecting logs from apache machines that isn't too resource-intensive, and transports them relatively quickly over the network
  • Some way of storing them (NoSQL? key-value store?) on the backend, for a given amount of time (and then rolling them up into historical averages)
  • In the middle of this, a way of graphing in near-real-time (probably also with some statistical analysis on it) and hopefully alerting off of those graphs.

Any suggestions/pointers/ideas, to either "products"/projects or descriptions of how other people do this would be greatly helpful. Unfortunately, we're not exactly a new-age-y devops shop, lots of old stuff, homogeneous infrastructure, and strained boxes.

4
  • Sorry, let me clarify that I'm looking for options that are both fully Open Source by license (OSI-approved) and are not purchased products (free as in beer). Jun 9, 2012 at 16:08
  • I appreciate all of the responses about rsyslog, and it's wonderful software - I've even helped Rainer find a bug or two in it. But I'm looking for something that allows quick (i.e. doesn't store logs as plain text) searching and analysis on logs. I want to know things like acceleration of rate of 404s/minute across 30 web servers, or how many unique user-agents hit my sites in the last 2 mintues. Jun 9, 2012 at 16:45
  • Also, in terms of scale (leaning towards solutions based on MongoDB, HBase, etc. especially for capped collections), my test/evaluation will be based on 8 hosts (out of ~60 total) that generate approx. 4.5GB/15M lines of Apache logs per day. For the real time analysis I can get away with ~1 day of raw data and periodic roll-ups beyond that, but I'd really like to be able to keep 1 week of logs... Jun 9, 2012 at 16:50
  • Unfortunately, product-recommendation questions, even just if-exists research, are off-topic per the FAQ.
    – sysadmin1138
    Oct 12, 2012 at 19:20

5 Answers 5

2

rsyslog can work pretty well, and if the amount of data that you are attempting to log is small enough you can even get away with using the free version of Splunk. The full version is probably a more comprehensive solution that maybe in line with what you want to accomplish saving you the time of developing your own in house monitoring tools.

At my work though we just stick to syslogd, Nagios, and Ganglia for all our monitoring needs as even with the 600 or so machines they are all incredibly stable.

2
  • 1
    Sounds very interesting. How do you feed syslog data into Nagios and Ganglia (specifically Ganglia that I'm interested in)? Jun 9, 2012 at 16:53
  • So we keep the syslog data separate from any Nagios or Ganglia data. Nagios data we keep around whereas Ganglia data just operates in RR mode and only keeps data for a few days. In the end if you really want to analyze the data and have things like automated reports, reports for the lifetime of a machine, it gets a bit harder to accomplish, and personally I feel it would be much easier and generally efficient to just pull different logs with syslog and roll your own monitoring system that can accomplish exactly what you need that is portable enough to be put on every server. It is funny
    – Wilshire
    Jun 13, 2012 at 16:18
2

Jason, you mentioned an interest in using Ganglia to monitor your Apache web servers. Have you considered using mod-sflow with Ganglia?

Using Ganglia to monitor web farms

mod-sflow

Recently, active, idle, max worker metrics have been added. While Ganglia is great for trending cluster metrics, you will need to use a log analyzer to report on the detailed log data. mod-sflow sends counter and log data as binary XDR encoded structures over UDP. You can use sflowtool to convert the binary data into standard ASCII logs, or as the basis of your own analysis tool.

1

If you're looking to set up a general purpose syslog server I'd definitely recommend you have a look at rsyslog, it's a very powerful modern syslog implementation. One of the things I like about it is that it can log to a relational database rather than to flat files, which makes data crunching a lot easier.

I've never used syslog with Apache, so I can't help with that part of your question unfortunately.

1
  • I've tried this approach before. The problem with relational databases is that when logging 20,000,000+ lines/day, even if the RDBMS can keep up with the write rate, purges (aside from partitioning on date and regenerating the table/partitioning every year) end up with an endless backlog. Jun 9, 2012 at 16:52
0

You can give LogZilla a try. It is 99% open source (one file is not). It is extremely fast and dirt cheap compared to the other solutions in this class.

1
  • Aside from not being open source, and probably not being able to do the generalized analysis I asked about (it's a commercial product - where's the Nagios plugin to alert on what I want), it's at least $4,300. That's pretty expensive for no budget... Jun 9, 2012 at 16:10
0

This isn't as general a solution as you asked for, but thinking back to a session from the London PHP conference, the BBC said they had a cunning way of transporting apache log files from many servers to a central server in real time, I think they nicknamed it teleportd.

I can't remember the exact details, but the jist was that they had a small auto-restarting script running on each of the apache servers that basically opened a fifo named pipe named /var/log/apache2/access_log and used netcat to copy it to a unique tcp port on the log servers. The log servers then catted it back out again into /var/log/myApacheServer/access_log.

With care, that approach would also work for general syslog files, although an initial sync during system startup might be required.

If you're ok with semi-real time then I'd pick a much simpler solution of rotating the log files every n minutes and rsyncing them to the central server in [postrotate].

Many webstats packages, like awstats and friends, assume that the log files are sorted, so something like awstats' logresolvemerge.pl might be a useful preprocessor on logServer:/var/log/*/access_log before you run whatever stats you require on the results.

Cacti would draw the graphs you seek using rrdtool, but you'd need to feed it from data grepped from the webstats internal data files, which is a tad unstructured for my tastes.

This approach is scriptable but would start to get tedious with large numbers of virtualhosts though, as you wind up with a vhosts*serverCount number of TCP streams.

It's all filesystem based though, so a bit low tech in today's world, sorry.

1
  • Actually I really like the simplicity of this approach, and there's some really good ideas here. The only problems for my use case are that 1) development, and a number of other groups, have extensive scripting based on daily-rotated log files, so I'd just need to log to 2 places simultaneously. The other issue, with grep, is that we want to use this to catch DoS, misbehaving web crawlers, backend failures (503), etc. in realtime, so grep doesn't scale for our needs. But very good ideas, thanks! Jun 10, 2012 at 15:36

Not the answer you're looking for? Browse other questions tagged .