36

Is there any way supervisord can automatically restart a failed/exited/terminated job and send me a notification email with a dump of the last x lines of log file?

50

There is a plugin called superlance.

You install it with pip install superlance or download it at: http://pypi.python.org/pypi/superlance

The next thing you do is you go into your supervisord.conf and add the following lines:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m email1@example.com
events=PROCESS_STATE

This should be followed by a "supervisorctl update". When a process "exits" you will now get a notification sent to email1@example.com.

If you only want to listen to some selected apps you can exchange the -a for a -p program1 or if it is a group group1:program2 One example would be

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -p program1 -p group1:program2  -m email1@example.com
events=PROCESS_STATE

Regarding the automatic restart: you should make sure that autorestart is set to true (it is set to unexpected by default). This way the package will be restarted 3 times. If after that it still exits, it gives up, but you can change that with startretries.

Example program:

[program:cat]
command=/bin/cat
autorestart=true
startretries=10
| improve this answer | |
  • Despite being inactive for a few years this thread is still valid - I just tested superlance 1.0.0 (dated oct 2016) with supervisor 3.1.4 (CentOS7) and the crashmail worked just fine. – David Ramirez Oct 4 '18 at 19:48
  • How do we test it? – Curious Developer Jul 17 at 14:43
  • If you have it configured correctly, you should see crashmail as process when running supervisorctl. You can test the mail by doing ps aux | grep programthatshouldnotcrash, then killing that process with kill -9 123456789 where 123456789 is the pid of the program that is currently running via supervisor. Note: The documentation says to listen to PROCESS_STATE_EXITED instead of PROCESS_STATE. – Sumurai8 Sep 4 at 13:09
0

I tried installing superlance and running crashmail like this:

sudo apt-get install python-pip
sudo pip install superlance

after i do:

sudo nano /etc/supervisor/supervisord.conf

and after i added:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m mymail@mail.fr
events=PROCESS_STATE

and I do not receive anything....

My crashmail file is:

#!/usr/bin/python

-- coding: utf-8 --
import re
import sys

from superlance.crashmail import main

if name == 'main':
sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0])
sys.exit(main())
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.