0

I've asked this question around a week a go at StackOverflow but no answer just yet, probably is no possible but don't know where to look for that answer, I hope someone can help over here.

I'm using monit to scan logs for errors and then push those alert into a monitoring system called DataDog...

All seem to work as expected but now I've need to grab what is causing the alarm.

Using a very simple rule I'm able to catch line on log that provoke this error and run the specific script to alert; until here all ok:

montirc file:

check file testmonit with path /var/log/testmonit.log
      if MATCH "(ERROR.*)" then
         exec "/usr/bin/python /opt/scripts/bin/dd_notify.py test-error"

This config make what I want, it actually raise the alarm I wanted

But now I need to know "What caused this alarm"; so for example if this line appears on the log:

ERROR failure to complete process due lock file....

On monit logs I can see:

[UTC Mar  6 11:59:08] error    : 'testmonit' content match [ERROR failure to complete process due lock file....]
[UTC Mar  6 11:59:08] info     : 'testmonit' exec: /usr/bin/python

Which is perfect... what I want is capture this:

[ERROR failure to complete process due lock file....]

To send this string into my monitoring system (DataDog) I can't find any documentation that actually allows me to use the MATCH content, or groups (which I can see is being supported by the MATCH regex)

So in short:

There's any monit variable(like $DESCRIPTION for mail) that refer to MATCH line that trigger the rule?

(I've tried $DESCRIPTION, $HOST...etc but this seems to work only for email)

I'd a look into google several times (and also here) but I can't find the answer...

Please if you think this has been addressed before feel free to point me to right direction.

An update:

Sorry I forgot to say that I'm running this on:

 Ubuntu 16.04 LTS and 
 Ubuntu 12

Monit version is:

This is Monit version 5.25.1 Built with ssl, with ipv6, with compression, with pam and with large files Copyright (C) 2001-2017 Tildeslash Ltd. All Rights Reserved.

Following DevOps reccommendation, I've upgraded monit version.

And tried to use MONIT_DESCRIPTION or $MONIT_DESCRIPTION without success rule file is like this:

check file pd-error with path /var/log/testmonit.log if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error " $MONIT_DESCRIPTION

What I want is pass the content that has been matched as an additional argument for dd_notify.py program;

but what i get is (which is the result of executing dd_notify.py):

{ "ALARM": { "pd_error": 67 }, "MESSAGES": { "pd_error": "$MONIT_DESCRIPTION" } }

what I want is $MONIT_DESCRIPTION content which is actually:

[UTC Apr 3 21:53:22] debug : 'pd-error' Pattern 'ERROR' match on content line [Apr 3 21:52:30 ams01 MainProcess[1376]: cel ery.worker.job ERROR Task tasks.telemetry.gather_and_send_telemetry_info[f090d579-9ec2-40e5-9fb2-91436eb4fc8a] fail]

But I'm not having any luck at the moment... what I'm missing here?

Thank you.

  • Seeing "$MONIT_DESCRIPTION" in the (JSON) output of your python script, mostly, your are not reading correctly the enthronement variable in your python script. Check stackoverflow.com/questions/4906977/… – DevOps Apr 19 '18 at 16:05
1

Jsut tried with Monit 5.23.0 and there is an environment variable for it.

MONIT_DESCRIPTION=content match:
[ERROR failure to complete process due lock file....]
[ERROR failure to complete process due lock file....]

Monit will output every occurrence of the matching content

Also the syntax as changed in Monit 5.16.0 but old is still functional. Changelog is available here : https://mmonit.com/monit/changes/

Also be aware that you can change the various limits of Monit https://mmonit.com/monit/documentation/monit.html#LIMITS

If you are willing to upgrade, Monit provides precompiled generic binaries that I'm using to have a newer versions than the ones in the Ubuntu repositories.

| improve this answer | |
  • Hi thanks for answering!, I will try this and will get back to you I'm really happy you answered... I will come back to you as soon as I test it! – Larry Mar 15 '18 at 18:51
  • Hello DevOps, I've tried your suggestion but can't get what I want... not sure if I'm doing it correctly: I've tried to use it as MONIT_DESCRIPTION or $MONIT_DESCRIPTION without any success... what I want is pass the match content as an argument to another program that is being executed by the rule... I will update my post to explaint it better... – Larry Apr 3 '18 at 21:45
2

Thanks for DevOps for put me on the rigth track to finish with this issue, I've finally succeed on what I wanted to do, and also can explain (from my understanding) why it wasn't working for me before

Thing is as DevOps says, there's a variable MONIT_DESCRIPTION that in fact has the error string, but this variable is only "reachable" at bash environment.

As I was doing:

check file pd-error with path /var/log/testmonit.log
if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error "

and from dd_notify.py I was trying to do :

error = os.environ['MONIT_DESCRIPTION']

but I always got 'key_error' because 'MONIT_DESCRIPTION' wasn't accesible from python

then I tried to call my program using a bash wrapper like:

check file pd-error with path /var/log/testmonit.log
  if CONTENT = "ERROR" then exec "/bin/bash /opt/scripts/bin/wrapper.sh"

and on 'wrapper.sh' code I have:

/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error

and then I got what I was looking for:

{
  "ALARM": {
    "pd_error": 294
  }, 
  "MESSAGES": {
    "pd_error": "content match:\nMay 16 18:07:08 ams01 MainProcess[1358]: celery.worker.job ERROR Task fds.realtime.tasks.telemetry.gather_and_send_telemetry_info[abe35540-55ef-40db-984a-
12287f5648ab] raised unexpected: ConnectionError()#012Traceback (most recent call last):#012  File \"/usr/lib/python2.7/dist-packages/celery/app/trace.py\", line 240, in trace_task#012   
 R = retval = fun(*args, **kwargs)#012  File \"/usr/lib/python2.7/dist-packages/celery/app/trace.py\", line 438, in __protected_call__#012    return self.run(*args, **kwargs)\n...\n"
  }
}

Which is great!

So basically I was not able to access monit enviroment variable from Python... so I wrapped into a bash script and then got it...!

Thank you!

| improve this answer | |
  • To add for posterity.. The variable is multi line. This for some reason means, if you access it as $MONIT_DESCRIPTION, you get only the first line which is a very helpful content. Whereas using it with quotes in bash "$MONIT_DESCRIPTION" works as expected and gives the full match – Karthik T Oct 11 '19 at 6:54
  • It was the spaces in the variable rather than the multiline nature that was my issue above. – Karthik T Oct 11 '19 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.