Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am running into a couple of issues with some applications we've deployed and maintain. I have the feeling we have approached this with some anti-patterns up to now, but I would like to see how to make this more flexible and stable.

In one situation, we have a server at a client which pushes data to us to parse every night (yes, Windows Task Scheduler). This is highly unstable however, so once every month this doesn't happen because of reasons out of our control. This heavily impacts our business since we run with stale data in that situation.

In another scenario we have a lot of background job processes that should be running. We already keep them up using bluepill ( http://www.github.com/arya/bluepill ) but obviously restarts happen, both automatically and manually, and people forget things or systems mess up.

What I would like to track is events that should occur or should be available. Like the existence of a process, the execution of a program, or the creation/age of a file, and track it when they don't happen or exist.

We develop most things in Ruby on Rails, use NewRelic, Bluepill and Munin, and run on Ubuntu. I've been toying around with counting ps aux | grep processname | wc -l in Munin scripts, or capturing the age of a file and raising alerts over 24-26 hours, stuff like that.

Is there better tooling to track things that should happen, and raise alerts if they don't?

P.S. I know some things are suboptimal, like manually having to define bluepill for applications and then forgetting to do so. The same goes for the push based approach of the first application, a dedicated daemon that manages that on the client side that we control and can track its connection to us might be a much better solution.

share|improve this question
    
This is... just way too broad to be a good question. But in the general case: (pseudocode) if [thing you're looking for] exists, break; else, [trigger alert];. –  HopelessN00b Nov 8 '12 at 18:16
    
yes, I know I can use cronjobs and such. But since monitoring tools exist we can streamline those things. Just wondering if some tools have a more native approach to tracking absence of some thing or event, since most tend to focus on presence. –  Kamiel Wanrooij Nov 8 '12 at 18:30
    
This got closed as it appears to be searching for an off-the-shelf solution to your monitoring problem, which triggers our shopping reflexes. However, your question is answerable without resorting to product recommendations. So I'm reopening it. –  sysadmin1138 Nov 25 '12 at 13:46
    
Thanks! I am indeed looking for a sensible method of dealing with this rather than just a list of products. –  Kamiel Wanrooij Nov 30 '12 at 15:02

2 Answers 2

Most quality monitoring frameworks that work well in the Linux space have the ability to have custom-written probes. These can often be used over SSH, which allows you to essentially write bash scripts that get run every time the monitoring system probes your assets. From the sounds of it, you'll need two custom probes:

  1. If the modification time-stamp of a file is older than now-X, alert.
  2. If $processName doesn't show up in the running processes list, alert.

Alternately, you can tie these scripts to your snmpd process, so when a specific OID is accessed they're run and return their value. Darned near everything can do SNMP.

In my experience, pre-built probes rarely handle the file-exists or file-too-old events, but some do include the ability to test that certain long-running executables are running.

share|improve this answer

We use monit or mmonit (http://mmonit.com/monit/) for these kind of things. You can define it to look at files, timestamps, if something exist or even to run your script and check for output.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.