2

I am using Monit 5.5 with the CHECK PROGRAM directive to execute an external script that does some work to verify if my app is functioning properly. I want to do this check every few cycles so as to avoid being a burden on the app. My config is this:

CHECK program mydaemon with path "/usr/local/sbin/my_check.sh"
  ALERT [email protected] ON { exec }
  START PROGRAM "/etc/init.d/mydaemon start"
  STOP PROGRAM "/etc/init.d/mydaemon stop"

  if status = 1 for 2 cycles then restart

  # Trick monit into doing a restart + hitting our local alert
  if status = 1 for 4 cycles then exec "/bin/true"

  if status = 1 for 6 cycles then unmonitor

  every 3 cycles

Things work almost as expected - every 3 cycles monit performs the check or takes an action, though as you might guess by the comment in the config, what I am seeing in my logs is that on cycle 3, 4, and 5 monit will also perform the restart action:

May 24 14:03:24 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:03:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:03:54 monit[19488]: 'mydaemon' trying to restart
May 24 14:03:54 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:03:54 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:04:24 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:24 monit[19488]: 'mydaemon' trying to restart
May 24 14:04:24 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:04:24 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' exec: /bin/true
May 24 14:04:54 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:04:54 monit[19488]: 'mydaemon' trying to restart
May 24 14:04:54 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:04:54 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' exec: /bin/true
May 24 14:05:25 monit[19488]: 'mydaemon' status failed (1) for /usr/local/sbin/my_check.sh -- Error: testing!
May 24 14:05:25 monit[19488]: 'mydaemon' trying to restart
May 24 14:05:25 monit[19488]: 'mydaemon' stop: /etc/init.d/mydaemon
May 24 14:05:25 monit[19488]: 'mydaemon' start: /etc/init.d/mydaemon

Why is monit performing my restart action on unspecified cycles?

PS - My monit cycle length is 10s, hence the actions in the log snippit are 30s apart.

2
  • This is difficult to follow.
    – ewwhite
    May 27, 2013 at 17:43
  • @ewwhite - which parts are confusing?
    – Dave S.
    May 27, 2013 at 17:46

1 Answer 1

1
+50

Monit acts exactly as you've instructed it to.

Let's break the logic down:

Cycle 1
Check result: 1 consecutive failure
Action: none

Cycle 2
Check result: 2 consecutive failures
Action: restart (the first condition is met)

Cycle 3
Check result: 3 consecutive failures
Action: restart (the first condition is still met, the last two cycles exited with status = 1)

Cycle 4
Check result: 4 consecutive failures
Action: restart AND exec /bin/true (both the first and second conditions are met)

Cycle 5
Check result: 5 consecutive failures
Action: restart AND exec /bin/true (both the first and second conditions are still met)

Since your program always returns 1, the first condition will always be met after the check result at the second cycle, since (at least) the last 2 cycles will always have failed until you unmonitor.

5
  • Niiiiice... Is the OP asking Monit to do too much in this case?
    – ewwhite
    May 27, 2013 at 22:30
  • 3
    Trying to make a piece of software understand what you mean and not what you actually tell it to do. That's a pretty tall order if you ask me... May 27, 2013 at 22:32
  • Thanks, this helps a lot with what was perhaps my core misunderstand - that the condition for "2 cycles" is not "on the second cycle". Is there a way to achieve what I was hoping for?
    – Dave S.
    May 28, 2013 at 0:22
  • Can't think of anything... but, wouldn't you expect your program to return 0 after the first restart? May 28, 2013 at 16:13
  • The check program only will return 0 if the underlying daemon is properly doing its job. In some cases a restart doesn't always fix the problem, so my goal here was to implement a rough back-off policy. I suspect what I will need to do instead is have my check program look at additional info and run less frequently (perhaps every 6 cycles), giving the daemon more time to fully boot and return to proper operation.
    – Dave S.
    May 28, 2013 at 16:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .