1

Not so far ago LA on my server raised to 400 and I couldn't even login to server using ssh. Does exists any software, that can prevent this situations by automatically killing processes that making huge load on server?

PS. Debian 6.0.5

4 Answers 4

3

You can use a watchdog like Monit to watch over the processes you care about, and restart them if they consume excess resources.

Something like this would be used to monitor Apache:

 check process apache with pidfile /var/run/httpd.pid
       start program = "/etc/init.d/httpd start"
       stop program  = "/etc/init.d/httpd stop"
       if cpu > 40% for 2 cycles then alert
       if totalcpu > 60% for 2 cycles then alert
       if totalcpu > 80% for 5 cycles then restart
       if mem > 100 MB for 5 cycles then stop
       if loadavg(5min) greater than 10.0 for 8 cycles then stop

So, if the cpu% for the Apache process or any of its children are over 40%, send an alert. If it's above 80%, do a restart of Apache.

Monit will also start up Apache if it's not running for some reason, which is a reasonable way to keep critical services up (if you don't have something like Upstart available).

This assumes that you have a set of processes that you can target for this sort of monitoring. Presumably, you suspect a particular application may be a problem.

0
2

When your LA raises and you can't login via ssh, try Grey Goo a tiny available and reliable remote command execution server and client designed purely for emergency situations:

https://code.google.com/p/greygoo/

0
0

Load is an aggregate value of tasks with processing work left to run.

It is a measure of the remainder of processing left to do after each process has been given a fair time slice of CPU. Effectively it puts a number on how well the system is dealing with competition of resources between processes.

IOWait can also increase load if processes are causing large IO usage which would point to a memory problem (using far too much of it and aggressively swapping) or an underlying I/O problem. If its just one process that uses up loads of I/O it would be punished usually anyway without too much affecting other processes.

I dont think you can just simply kill one process to fix a problem like this as the problem stems from a few processes demands on the CPU. You could kill a process group or a thread pool for one particular process though.

3
  • Ideally, I want something, that will be kill all suspicious processes which using too many CPU or IO with something like blacklist of processes, that never will be killed (like ssh).
    – Drakmail
    Jun 27, 2012 at 21:00
  • I dont want to sound harsh, but that feels like a band-aid to me. You should really be figuring out what circumstances cause the load to go out of control and remedy the problem at the source. Jun 27, 2012 at 23:49
  • Sounds good, but problem does not repeat now :( I has some suggestions, why it could be, but I'm afraid that verifying of it my kill my server again.
    – Drakmail
    Jun 28, 2012 at 5:36
0

Here is a simple bash script that can kill a process if system load is higher than a certain limit.

In this example the script kills the "named" process if the load is higher than 2.5.

If the load drops below this level then it checks if named is running, and starts it if needed.

#!/bin/bash

NOTIFY="2.5"

TRUE="1"    
FTEXT='load average:'
LOAD5MIN="$(uptime | awk -F "$FTEXT" '{ print $2 }' | cut -d, -f2 | sed 's/ //g')"

RESULT=$(echo "$LOAD5MIN > $NOTIFY" | bc)

if [ "$RESULT" == "$TRUE" ]; then
    echo "High load"
    killall -9 named
else
    pgrep named || /etc/init.d/bind9 start 1>/dev/null 2>/dev/null
fi

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .