I have an odd bug with a rails app I've created, causing it (randomly, maybe once every couple days) to have all (or some instances) go to 100% CPU usage and stay there forever. This causes nginx to serve an error to the users. So I decided to find a way to automatically kill the offending ruby process once it does this.
Due to the fact that I'm using nginx + passenger (not passenger standalone) I had to use a gem which creates the PID files for each rails process. This works correctly, I get the right number of PID files with the right PID in them in the /var/tmp/ directory.
So in order to monitor these processes I decided to use Monit. This is the config I used (note the /bin/true is because I don't need to actually start the process, once killed a new one will be spawned automatically).
check process instance1 with pidfile /var/tmp/rack.1.pid
start program = "/bin/true" with timeout 30 seconds
stop program = "/bin/bash -c '/bin/kill -s SIGTERM \'/bin/cat /var/tmp/rack.1.pid\''"
if cpu > 15% for 2 cycles then restart
However, it always fails to kill the process. I've tried a couple stop 'scripts', this is another I've tried:
stop program = "/bin/bash -c 'kill -9 `cat /var/tmp/rack.1.pid`' && rm -f /var/tmp/rack.1.pid"
Neither of them work, both result in the error below. I've tried using /bin/sh instead, same thing.
[EDT Oct 16 19:09:06] error : 'instance1' cpu usage of 15.6% matches resource limit [cpu usage>10.0%]
[EDT Oct 16 19:09:36] error : 'instance1' cpu usage of 15.8% matches resource limit [cpu usage>10.0%]
[EDT Oct 16 19:09:36] info : 'instance1' trying to restart
[EDT Oct 16 19:09:36] info : 'instance1' stop: /bin/bash
[EDT Oct 16 19:10:06] error : 'instance1' failed to stop
So I have a couple questions: What am I doing wrong here? Is it something obvious that I just don't see?
Is there any way to see the actual error that monit gets? I've enabled logging, but I can't see any other logging options in the doc.
