I've recently taken over administration of a decent size network (900 FreeBSD or Linux hosts across 10 physical sites) where we get a lot of email destined for root, mostly from cfengine (v2). We've also installed a lot of cron jobs over the years, some of which make noise from time to time (and some of which make noise on every run!) Over the years, sysadmins on the team have grown to accept the messages and filter them out to /dev/null so it rarely gets read.
Obviously this isn't ideal and we should fix all the noise - and we will. But first I'd like to do something to wrap our heads around the problem, and try to point to the most "noisy" programs, and to keep stats on the volume of emails so that we can see stats on the changes we make.
I was thinking I'd set up a machine which receives copies of these emails and then process them into a database, where I'd run stats on how many repeats of similar messages we get. There are a lot of pre-existing systems out there for aggregating traditional log files, but the only one that similar to what we're after is Sentry, but I would have to write some processing stuff to make it work. I could write something myself if nothing like this exists.
Has anyone else faced this sort of issue and is aware of a solution for aggregating lots of error email reports and performing analysis on them? I'm looking for:
- Stats over time on # of emails per hour or day
- Lists of most common messages, by message content, subject or From: address
- Any other stats which would point to where simple changes could have the most impact on volume of messages
