Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server with a faulty power button that likes to reboot itself. Usually there are warning signs, like the acpid log file in /var/log starts spamming garbage for about 10hrs or so.

Is there an easy way I can have something monitor the acpid log and email me when it has new activity?

I wouldn't consider myself extremely advanced so any "guides" you may have for accomplishing something like this would be very helpful and much appreciated. Thank you!

share|improve this question
    
Is changing the power button and/or the server a plausible solution? –  Air 2 days ago
add comment

8 Answers

You could use something like LogWatch. Or even a simple script like this (it's pseudo code you'll need to modify it for your enviroment):

 #!/bin/bash
 GREP_STRING=`grep -c <error string> <acpid log location>`
 if [ $GREP_STRING -ne 0 ] 
 then
    <send email notification>
 fi

Put that in cron to run every hour or so and you should get an email letting you know when it's getting wierd.

share|improve this answer
    
logwatch works great for me. –  J.Zimmerman Jul 25 '09 at 0:48
2  
The problem with this script is that it will send the same error over and over until the file gets rotated –  chmeee Jul 25 '09 at 15:43
    
On Ubuntu/Debian logwatch can be installed with: aptitude install -y logwatch –  Air 2 days ago
add comment

You can use OSSEC HIDS to set up rules on log files and, at the same time, get security information from your host.

Setting it up is very easy:

  • Download the source
  • Uncompress it and run ./install.sh
  • Choose local install
  • Answer the questions (email, checks, etc.)
  • Edit /var/ossec/rules/local_rules.xml as specified below
  • Start OSSEC with /var/ossec/bin/ossec-control start

local_rules.xml

<group name="local,syslog,">
  <rule id="100001" level="13">
    <regex>^.*Your string.*$</regex>
    <description>I've just picked up a fault in the AE35 unit. It's going to go 100% failure in 72 hours</description>
  </rule>
</group>

Rules can be very flexible and complex. See this table to get an idea of the parameters involved in a rule.

If you don't want or need the other security features you can deactive them by removing the include lines under the rules tag.

share|improve this answer
add comment

I would suggest Nagios its what we run where I work for monitoring multiple machines with are network. Its very good i've not used it specifically for what your doing but you can certainly set it up to email you when errors occur.

There is a guide here for installing it on Ubuntu http://beginlinux.com/blog/2008/11/install-nagios-3-on-ubuntu-810/ and one here for installing on http://www.debianhelp.co.uk/nagiosinstall.htm.

share|improve this answer
add comment

And you can send it with something like this:

EMAILMSG="/tmp/logreport.$$"
echo "Something to put in the email" >> $EMAILMSG

cat $EMAILMSG | mail -s "Whatever Subject You Like" user@domain.com
rm -f $EMAILMGS
share|improve this answer
add comment

Download and install Splunk (www.splunk.com) on the server. It's similar to logwatch, but provides you with a search engine for your logs.

You can configure it to index your logs, you can then search the logs and find patterns, find the errors, and then look at what other logs are doing at that specific point of failure.

It can also be set to send alerts or execute scripts at certain thresholds. So if a particular error starts being spammed to your log, you can script it to automatically restart the offending service.

We use splunk in our server cluster and it has been a lifesaver!

share|improve this answer
    
+1 for Splunk it looks pretty good i'm giving it a try later tonight. –  Mark Davidson Jul 25 '09 at 18:26
add comment

I'm using Zabbix with IPMI tools to restart faulty servers on demand. Also, I think OSSEC is a good choice too, but you really need to experiment and debug before put it in prod...

share|improve this answer
add comment

At a previous employer, we used logsurfer+ to monitor logs in real time and send email alerts. It does take a lot of time and configuration to tune for false positives, but we had a ruleset that worked quite well for a variety of findings and alerting, far more valuable than Nagios was for similar purposes.

Unforunately I don't have access to the config file anymore to provide samples of what we filtered, but the site should provide more information and examples.

share|improve this answer
add comment

You can also take a look at my Octopussy project.

share|improve this answer
add comment

protected by Michael Hampton Apr 6 at 19:12

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.