How to prevent termination of long-running Python script

Question

I have a cron job running a Python script on a VM-based remote Ubuntu server. The script takes a couple hours to run, and I'm noticing the kernel appears to be terminating the script before it completes, as the log shows:

myscript.py: line 11:  4890 Terminated

I've monitored my script's CPU and memory usage via top while it's running, and it never gets excessive.

How do I find out why my script's being terminated and how to prevent premature termination?

Answer 1

You may be exceeding some limits set by default or in /etc/security/limits.conf. You may want to run the command ulimit -a as a cronjob. This should display the limits you are getting under cron.

It is possible the job is mistakenly being killed by an idle terminal monitoring program or runaway process killer. There are a large number of such programs, most of which can be programmed to ignore known long running processes.

Edit: The default values have limits that could be exceeded. These are the limits I get by default:

:~$ ulimit -a | grep -v unlim
core file size          (blocks, -c) 0
scheduling priority             (-e) 0
pending signals                 (-i) 61167
max locked memory       (kbytes, -l) 64 
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
max user processes              (-u) 61167

Of these open files and stack size are the two I would expect to be most likely to be exceeded. Open files can be monitored by counting the entries in /proc/XXX/fd, where XXX is the process id of your script. I am not aware of any easy way to monitor stack size. Running the program from a script that increases the stack size limit may help determine if this is the problem.

I would also check all the logs written around the time the program is terminated to see if there is anything logged. If you can modify the program to be more verbose when exiting.

Answer 2

If it is the kernel you should see mention of it in dmesg. Otherwise you have to look for something else that is terminating your script (maybe a too broad killall python somewhere else?).