Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Not so far ago LA on my server raised to 400 and I couldn't even login to server using ssh. Does exists any software, that can prevent this situations by automatically killing processes that making huge load on server?

PS. Debian 6.0.5

share|improve this question

3 Answers 3

up vote 1 down vote accepted

You can use a watchdog like Monit to watch over the processes you care about, and restart them if they consume excess resources.

Something like this would be used to monitor Apache:

 check process apache with pidfile /var/run/httpd.pid
       start program = "/etc/init.d/httpd start"
       stop program  = "/etc/init.d/httpd stop"
       if cpu > 40% for 2 cycles then alert
       if totalcpu > 60% for 2 cycles then alert
       if totalcpu > 80% for 5 cycles then restart
       if mem > 100 MB for 5 cycles then stop
       if loadavg(5min) greater than 10.0 for 8 cycles then stop

So, if the cpu% for the Apache process or any of its children are over 40%, send an alert. If it's above 80%, do a restart of Apache.

Monit will also start up Apache if it's not running for some reason, which is a reasonable way to keep critical services up (if you don't have something like Upstart available).

This assumes that you have a set of processes that you can target for this sort of monitoring. Presumably, you suspect a particular application may be a problem.

share|improve this answer
    
It is great! Thanks, this is that I need. –  Drakmail Jul 5 '12 at 6:59

Load is an aggregate value of tasks with processing work left to run.

It is a measure of the remainder of processing left to do after each process has been given a fair time slice of CPU. Effectively it puts a number on how well the system is dealing with competition of resources between processes.

IOWait can also increase load if processes are causing large IO usage which would point to a memory problem (using far too much of it and aggressively swapping) or an underlying I/O problem. If its just one process that uses up loads of I/O it would be punished usually anyway without too much affecting other processes.

I dont think you can just simply kill one process to fix a problem like this as the problem stems from a few processes demands on the CPU. You could kill a process group or a thread pool for one particular process though.

share|improve this answer
    
Ideally, I want something, that will be kill all suspicious processes which using too many CPU or IO with something like blacklist of processes, that never will be killed (like ssh). –  Drakmail Jun 27 '12 at 21:00
    
I dont want to sound harsh, but that feels like a band-aid to me. You should really be figuring out what circumstances cause the load to go out of control and remedy the problem at the source. –  Matthew Ife Jun 27 '12 at 23:49
    
Sounds good, but problem does not repeat now :( I has some suggestions, why it could be, but I'm afraid that verifying of it my kill my server again. –  Drakmail Jun 28 '12 at 5:36

When your LA raises and you can't login via ssh, try Grey Goo a tiny available and reliable remote command execution server and client designed purely for emergency situations:

https://code.google.com/p/greygoo/

share|improve this answer
    
Thanks, nice tool. –  Drakmail Jul 5 '12 at 7:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.