Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run a python script several days ago using 'python a.py &' on my Ubuntu 12.04. I assume it's running all the time, but just now I found it crashed 2 days ago. Is there any log information on ubuntu that I can find how it crashed? Any debug message recorded by the system?

Also, is there a way to set up a alert checking, like send me an email if this python script is not running anymore? I need it to be running all the time. Thanks

share|improve this question
    
Did it crash or did you exit your session? If you leave your session the process will die. Use nohup or make it a daemon. Otherwise you'll need to use a secondary method for monitoring if it's running. Cheater way setup a cronjob to verify it every minute and start it. Otherwise use nagios, zabbix, zenoss or some other monitoring tool to do process monitoring and then perform a restart action if the process dies. –  John May 2 '13 at 22:49
    
I ssh, and I use '&' for backend running, and then I log out. It's usually fine. But I don't know how it got crashed. Do you know any way to get the error message? –  Jiechao Li May 2 '13 at 22:51
    
I've seen things like that die if they try to write to stdout or stderr and no valid device is available. To troubleshoot, you can either explicitly redirect both stdout and stderr when you background your program, or else background it from a screen session, and see if you ever see things get written to the terminal inside screen when you reattach to it. –  Mike Renfro May 2 '13 at 22:56
    
Use monit. mmonit.com/monit/documentation/monit.html to monitor your script. Create a proper init/upstart script so that your daemon is started when the system boots. –  Zoredache May 2 '13 at 23:03
    
Thanks, Zoredache. I will look into monit. –  Jiechao Li May 3 '13 at 5:35
add comment

1 Answer

up vote 2 down vote accepted
nohup python a.py 2>&1 > myscriptoutput &

In this case stderror/out will go to that file "myscriptoutput" and will not use tty. Once you close your ssh session generated HUP signal will be ignored by python process. And as of monitoring your script: put a cron job to run every 2 minutes and do something like this:

    #!/bin/bash
    if ps -aux | grep -v grep | grep a.py; then
        exit 0
    else
        #Generate email
        echo "Script a.py is not running" | mailx -s "My script is dead a.py" admin@mycomp.org 
        exit 0
fi

There are actually much better ways to do it of course but this is will get you started.

Hope this helps.

share|improve this answer
    
Danila, thanks a lot for your answer. That is a good way to do that. I am gonna look into monit first, that seems a good tool. –  Jiechao Li May 3 '13 at 5:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.