Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Every so often (maybe once every 3 months), all of a sudden our instance gets 100% CPU usage and locks out and becomes very unresponsive.

I'd like to find a way to detect this condition and initiate a reboot; what's the best way of achieving this?

We have a Cloudwatch subscription. We are running Ubuntu Server 10.04.

share|improve this question
1  
Why not fix the runaway process instead? –  Ignacio Vazquez-Abrams Nov 21 '10 at 15:54

1 Answer 1

up vote 3 down vote accepted

Seriously, fix the issue. There could be tons of reasons why your CPU usage is at 100%, and restarting the server or killing processes when this happened is just going to be a recipe for disaster. If this is a third-party utility--either list it here or file a bug with the maintainers.

If this is a home-grown utility, it very clearly sounds like you have an issue with your application. My first intuition would be to check boundaries on any loops you have that might be using heavy processing power. Another would be to check to see whether or not termination conditions are being met. In all honesty, though, this is a debugging issue and should be dealt with in house.

As for your actual question, there's probably not anything built in--but with the correct permissions, you could certainly configure Nagios to trigger a remote restart (i.e., via ssh) when it detects certain conditions. You can use the system load indicators (usage != load) to monitor, say, the 15-minute intervals for a certain max value. You could also do this via a cron script that, say, runs every minute

I hope you fix the bug rather than work around it--future maintainers will be thankful, especially if its you!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.