Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am using Monit 5.5 with the CHECK PROGRAM directive to execute an external script that does some work to verify if my app is functioning properly. I want to do this check every few cycles so as to avoid being a burden on the app. My config is this:

CHECK program mydaemon with path "/usr/local/sbin/my_check.sh"
  ALERT monit@mycompany.pagerduty.com ON { exec }
  START PROGRAM "/etc/init.d/mydaemon start"
  STOP PROGRAM "/etc/init.d/mydaemon stop"

  if status = 1 for 2 cycles then restart

  # Trick monit into doing a restart + hitting our local alert
  if status = 1 for 4 cycles then exec "/bin/true"

  if status = 1 for 6 cycles then unmonitor

  every 3 cycles

Things work almost as expected - every 3 cycles monit performs the check or takes an action, though as you might guess by the comment in the config, what I am seeing in my logs is that on cycle 3, 4, and 5 monit will also perform the restart action:

May 24 14:03:24 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:03:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:03:54 monit[19488]: 'mydaemon' trying to restart
May 24 14:03:54 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:03:54 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:04:24 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:24 monit[19488]: 'mydaemon' trying to restart
May 24 14:04:24 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:04:24 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' exec: /bin/true
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' trying to restart
May 24 14:04:54 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:04:54 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' exec: /bin/true
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' trying to restart
May 24 14:05:25 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:05:25 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon

Why is monit performing my restart action on unspecified cycles?

PS - My monit cycle length is 10s, hence the actions in the log snippit are 30s apart.

share|improve this question
    
This is difficult to follow. –  ewwhite May 27 '13 at 17:43
    
@ewwhite - which parts are confusing? –  Dave S. May 27 '13 at 17:46

1 Answer 1

up vote 1 down vote accepted
+50

Monit acts exactly as you've instructed it to.

Let's break the logic down:

Cycle 1
Check result: 1 consecutive failure
Action: none

Cycle 2
Check result: 2 consecutive failures
Action: restart (the first condition is met)

Cycle 3
Check result: 3 consecutive failures
Action: restart (the first condition is still met, the last two cycles exited with status = 1)

Cycle 4
Check result: 4 consecutive failures
Action: restart AND exec /bin/true (both the first and second conditions are met)

Cycle 5
Check result: 5 consecutive failures
Action: restart AND exec /bin/true (both the first and second conditions are still met)

Since your program always returns 1, the first condition will always be met after the check result at the second cycle, since (at least) the last 2 cycles will always have failed until you unmonitor.

share|improve this answer
    
Niiiiice... Is the OP asking Monit to do too much in this case? –  ewwhite May 27 '13 at 22:30
3  
Trying to make a piece of software understand what you mean and not what you actually tell it to do. That's a pretty tall order if you ask me... –  Mathias R. Jessen May 27 '13 at 22:32
    
Thanks, this helps a lot with what was perhaps my core misunderstand - that the condition for "2 cycles" is not "on the second cycle". Is there a way to achieve what I was hoping for? –  Dave S. May 28 '13 at 0:22
    
Can't think of anything... but, wouldn't you expect your program to return 0 after the first restart? –  Mathias R. Jessen May 28 '13 at 16:13
    
The check program only will return 0 if the underlying daemon is properly doing its job. In some cases a restart doesn't always fix the problem, so my goal here was to implement a rough back-off policy. I suspect what I will need to do instead is have my check program look at additional info and run less frequently (perhaps every 6 cycles), giving the daemon more time to fully boot and return to proper operation. –  Dave S. May 28 '13 at 16:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.