How to get into server when it's overloaded and ssh takes ages to connect

Question

My debian server sometimes gets overloaded (due to lots of apache processes and a badly-written website). The software developers are taking their time fixing the problem, but until they fix it, I have to cope with a server that keeps overloading.

When it overloads, I struggle to get an SSH connection to the box because it is under so much load. If and when I manage to get an SSH connection, I am always able to restart apache which rescues the server from the crazy website code.

But sometimes it can take me an hour or so to get a connection as my SSH connection keeps timing out.

Can anyone suggest a way I can guarantee that I can always SSH in, even if the server gets under a big load?

Answer 1

Renice sshd process to raise its priority. Should help when resources are tight

Answer 2

The ideal solution to this problem is to enable some sort of out-of-band access method. The simplest is probably a serial console. You can get serial consoles that allow you to ssh in and get access to the serial console of your server. Many decent hosting companies can supply this sort of access on dedicated and colo hosted servers. Alternatively, you might have something like a IP enabled KVM or lights out management like HP's iLO or Dell's DRAC.

Answer 3

If the reason you can't connect is due to bandwidth and not system resources then you could always apply traffic shaping at the border firewall to the network. If it is a system resources issue write a script that looks for system resources being exhausted and if say 3 runs in a row it detects a resource issue then restart apache automatically.

Answer 4

Perhaps you could write a script to watch the server load and if it gets above some point, restart apache (and whatever else you do when you get in there). If you create a cron job to run every minute or so, then maybe you can catch the load on the way up and fix it before the server gets too hammered. Otherwise the job might take a while to run, just like your problem establishing an ssh session...but my gut tells me that it'd have a better chance regardless.

Answer 5

try to see what exactly happens. maybe you can keep dynamically generated content still served from apache, but let static content served by better suited server like ngingx / lighthttpd / mathopd - on the same hsot [ if you have anohter ip or can use another port ] or on another host .