Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have over 100 servers that generate hundreds of emails every day using scheduled scripts (cron jobs) when the system/data is backed up or any other activity takes place. The email is sent to our network admin group, so a few people receive them every day. The issue is that we don't have enough time to go through each of them and verify if all the servers have been able to run their scheduled scripts and send the email. Is there an automated way to verify emails, using a client mail app or on the server side? I was thinking of generating an email only if the script fails, but what happens if that server is down?

share|improve this question
    
Hopefully your monitoring system would notice if the server's down. You do have one of those, don't you? –  HopelessN00b Aug 15 '12 at 15:09
1  
Do your emails contain identifiable tokens for success and failure? If so, you can use a categorisation, colourisation or even a simple move-to-folder rule in your email client, which searches for those tokens. If not, well that's your first issue: standardising all your emails so they can be machine-read. –  jimbobmcgee Aug 15 '12 at 15:14
    
You are discovering that e-mail isn't a very scalable monitoring solution. Every small shop runs into this wall eventually. Rather than automating the reading of these e-mails, I'd suggest reevaluating your method. –  Aaron Copley Aug 15 '12 at 16:21
    
yes, we do have monitoring system in place, but the scripts are mostly used to backup data or create some kind of new logs etc. –  ashvagan Sep 7 '12 at 14:01

1 Answer 1

Send email only if the scripts fail. And as HopelessNoob mentioned, for 100 servers, you should already have monitoring system in place to figure if a server is down. If not, it's high time you had one.

share|improve this answer
    
What happens when e-mail fails? Or networking stack all together..? –  Aaron Copley Aug 15 '12 at 16:22
    
Thats why you engineer redundant systems and have a monitoring system to monitor every critical piece of your infrastructure. –  Chida Aug 15 '12 at 16:38
    
You're missing my point. If a host only sends e-mail when there's a failure, but the failure is due to the host being offline... You're going to have a bad time. –  Aaron Copley Aug 15 '12 at 16:41
    
Hundreds of emails are sent on success and nobody is looking at them. As your systems grow, even more emails will come in. this is setting up for failure. If a backup fails, send email. If the server running the cron itself fails, then your monitoring system should alert you. Either way, you will get emails of bad events, which will be less, and you can focus on fixing it than trying to find a needle in a haysack for a failed email. I'm not sure if I understood in any other way. –  Chida Aug 15 '12 at 16:48
    
The problem is assuming that he has a monitoring system. It doesn't sound like you got all the way through the question because he specifically asks: I was thinking of generating an email only if the script fails, but what happens if that server is down? –  Aaron Copley Aug 15 '12 at 16:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.