2

Is it possible to keep a Linux remote console responsive (I'm using SSH but the box is also slow locally) when the system is under high load.

I'd like to be able to remote login to the server to monitor it, even though the system appears to be over loaded.

2 Answers 2

3

If the server is about to swap itself to death and there are huge number of processes fighting for CPU and other resources, then it's a bit difficult thing to do. Sometimes if a server is very overloaded, even a new ssh connection can be hard to establish.

In the situations like that I usually have another server in the same network and I have a ssh connection in screen from that server to the overloaded server. When the overloaded server becomes too overloaded, I ssh to that another server and resume my screen session.

But you have other options, too.

It's possible to partition processes to several domains using cpusets kernel feature. That way you can give one domain to your applications and allow it to use only some cores and some amount of memory. Then you can create another domain for system administration, so you should have some room left for your ssh needs.

Before doing this, though, you should be certain where does the high system load come from. Is it because of extremely high CPU usage? Are the applications constantly accessing disk and making I/O operations very slow? Are the applications eating up all the RAM and swapping causes the slowness? Is the memory/other resource usage expected or is there a bug lurking around in your application, causing it to suddenly eat up all the RAM or CPU?

If the latter, then you can install psmon or similar software. psmon can monitor your system and shoot down & restart the watched process(es) if it takes more than X megabytes of RAM or constantly takes more than Y% of CPU, for example.

And if the high resource usage is expected, then you should consider upgrading your hardware.

4
  • or optimising your software/data model Nov 5, 2010 at 14:44
  • Yeah, sure, if you have access to do that. If you are a poor soul just administering the server & OS, then all you can do is to try to keep the server running and possibly report your findings to software vendor ... :-) Nov 5, 2010 at 14:49
  • Thanks for you response. The system appears to be overloaded by CPU usage (over 80&). There is only one dominant process running which is a web server and as I want this application to make use of all available hardware, partitioning doesn't seem to solution in this case. Nov 5, 2010 at 15:02
  • Well then. Is this slowness going on all the time, or does your pager alert you every now and then and that's the point when you has to rush? Nov 5, 2010 at 15:05
1

This might be tricky depending on what high load means for you. It would be best to log in before the server gets under high load. Since this is not an option (can't be logged in all day) I would try limiting the normal users a little bit (just enough to get you logged in and save the day).

I would suggest to fiddle with /etc/limits.conf (see man limits.conf) to limit memory and increase priority for non root users. So there is still a resource window for administrative users to log in.

1
  • Interesting, I'll give it a go. Nov 5, 2010 at 15:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .