Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two independent alert notification systems for my servers. The server is a virtual machine on Linode and one of the alerts comes from Linode. The other monitoring system we use is New Relic. They are both watching out for IO utilization.

Every night I get alerts from both of them as the server is using too much IO. I run quite a few tasks in the middle of the night but the one I confirmed that can cause IO-warnings is running the backups. The backup is done by s3cmd sync. I tried ionice but it still generates the warnings.

Getting warnings every night reduces the efficacy of warnings when they happen for real. For Linode I could raise the level at which a warning is issued, but it might mean making the whole thing useless as the level is too high.

What would be the proper solution for this?

share|improve this question
3  
Add some code that turn off notifications to the start of the cron job that handles the backups. Then turn them back on at the end. –  Buttle Butkus Dec 6 '12 at 10:27
    
@ButtleButkus how do I turn notifications off for linode and new relic? How do I avoid something crashing in the middle and notifications left as off? –  J. Pablo Fernández Dec 6 '12 at 10:29
    
If there's a "maintenence mode" for the alerts then I would set the alerts into maintenence mode for the duration of the backup window. –  joeqwerty Dec 6 '12 at 14:25

1 Answer 1

First, ask yourself the important question: "Is this alert ever going to tell me anything useful?"
Is your server's workload such that it is I/O bound, and an I/O alert would warn you of a problem before one actually happens? Are there other, better indicators you can monitor (say response time on HTTP requests to the server?)

If the alert is not going to be helpful, disable it.
As you've correctly pointed out, "noise alerts" reduce the effectiveness of the monitoring system, so if this alert is just noise shut it off and be done with it.


If the alert is actually useful in your environment, but not useful during certain time periods, you need to check your monitoring systems and see if you can disable alerting during specific time periods (most can - I'm not sure about Linode or New Relic, but their documentation or tech support staff can certainly help you).

If your monitoring system can not suspend alerting during specific time periods you can accomplish the same thing with procmail rules (or similar mail filtering). This requires you to deliver the message to a filtered mailbox, but you can forward it on to pagers, cell phones, etc. from there as needed.
(If you use a mail filtering solution like this, make sure to document it so the next admin knows why they don't get alerts during that window! One day there might be a real problem, and missing the alert could be a Bad Thing.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.