up vote 2 down vote favorite
2
share [g+] share [fb]

running linux i have a few processes which tend to crash occasionally (game servers), which end up using 100% cpu. I'm looking for a program or script to check the cpu usage of a list of processes by name and if they are at 100% for more than X time, say 30 seconds, kill them. I tried ps-watcher but wasn't able to determine how to accomplish this. Just killing the process at 100% usage wont work as it will hit that for brief periods during normal operation. I've also found this script which seems to do what i want, however it is limited to one process: link Any help is greatly appreciated!

link|improve this question
Can you please post again a link to the script cause this one pastebin.com/m1c814cb4 seems not to be valid any more. – Nikolay Kolev Jan 11 at 15:57
feedback

2 Answers

up vote 6 down vote

Try monit.

You could use a configuration like this, to accomplish your task:

check process gameserver with pidfile /var/run/gameserver.pid
  start program = "/etc/init.d/gameserver start" with timeout 60 seconds
  stop program  = "/etc/init.d/gameserver stop"
  if cpu > 80% for 2 cycles then alert
  if cpu > 95% for 5 cycles then restart
  if totalmem > 200.0 MB for 5 cycles then restart
  if loadavg(5min) greater than 10 for 8 cycles then stop
  if failed port 12345 type tcp with timeout 15 seconds
    then restart
  if 3 restarts within 5 cycles then timeout

Details about this configuration can be found in monit's documentation.

link|improve this answer
Thank you for the reply! Is there any way to monitor the process without having to start it with monit? I have a ton of servers running on the machine which are managed through a web interface, having to launch them with monit isn't ideal. – user30153 Dec 28 '09 at 1:10
Sure, the start program and stop program lines are just for the case when monit needs to restart your process. You can still start it with your normal init script. monit can also check if the program is already running (e.g. by its PID file or process name). – joschi Dec 28 '09 at 2:00
Fantastic, i think i've got it figured out. The only problem is it's dependence on a pid file, i'm going to have to generate one for over 200 processes, and create rules for each one i suppose. Thanks for the help! – user30153 Dec 28 '09 at 8:37
feedback
up vote 0 down vote

Below is a sample BASH script that may help you get some hints for your own needs.

#!/bin/bash

CPU_LOAD=$(uptime | cut -d"," -f4 | cut -d":" -f2 | cut -d" " -f2 | sed -e "s/\.//g")
CPU_THRESHOLD=700

if [ $CPU_LOAD -gt $CPU_THRESHOLD ] ; then
  kill -9 $(ps -eo pid | sort -k 1 -r | grep -v PID | head -n 1)
fi

exit 0

Please take note that the value of your $CPU_THRESHOLD should depend on the number of (CPU) cores you have on your system. A detailed explanation about this topic can be found at http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages .

You can either call your script from inside the /etc/inittab or a cronjob for every number of minutes you prefer. Please take note also that the example script will kill the top-most process if the $CPU_LOAD is greater than the $CPU_THRESHOLD.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.