Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is it possible to keep a Linux remote console responsive (I'm using SSH but the box is also slow locally) when the system is under high load.

I'd like to be able to remote login to the server to monitor it, even though the system appears to be over loaded.

share|improve this question
add comment

2 Answers

This might be tricky depending on what high load means for you. It would be best to log in before the server gets under high load. Since this is not an option (can't be logged in all day) I would try limiting the normal users a little bit (just enough to get you logged in and save the day).

I would suggest to fiddle with /etc/limits.conf (see man limits.conf) to limit memory and increase priority for non root users. So there is still a resource window for administrative users to log in.

share|improve this answer
    
Interesting, I'll give it a go. –  MattChurchy Nov 5 '10 at 15:09
add comment

If the server is about to swap itself to death and there are huge number of processes fighting for CPU and other resources, then it's a bit difficult thing to do. Sometimes if a server is very overloaded, even a new ssh connection can be hard to establish.

In the situations like that I usually have another server in the same network and I have a ssh connection in screen from that server to the overloaded server. When the overloaded server becomes too overloaded, I ssh to that another server and resume my screen session.

But you have other options, too.

It's possible to partition processes to several domains using cpusets kernel feature. That way you can give one domain to your applications and allow it to use only some cores and some amount of memory. Then you can create another domain for system administration, so you should have some room left for your ssh needs.

Before doing this, though, you should be certain where does the high system load come from. Is it because of extremely high CPU usage? Are the applications constantly accessing disk and making I/O operations very slow? Are the applications eating up all the RAM and swapping causes the slowness? Is the memory/other resource usage expected or is there a bug lurking around in your application, causing it to suddenly eat up all the RAM or CPU?

If the latter, then you can install psmon or similar software. psmon can monitor your system and shoot down & restart the watched process(es) if it takes more than X megabytes of RAM or constantly takes more than Y% of CPU, for example.

And if the high resource usage is expected, then you should consider upgrading your hardware.

share|improve this answer
    
or optimising your software/data model –  The Unix Janitor Nov 5 '10 at 14:44
    
Yeah, sure, if you have access to do that. If you are a poor soul just administering the server & OS, then all you can do is to try to keep the server running and possibly report your findings to software vendor ... :-) –  Janne Pikkarainen Nov 5 '10 at 14:49
    
Thanks for you response. The system appears to be overloaded by CPU usage (over 80&). There is only one dominant process running which is a web server and as I want this application to make use of all available hardware, partitioning doesn't seem to solution in this case. –  MattChurchy Nov 5 '10 at 15:02
    
Well then. Is this slowness going on all the time, or does your pager alert you every now and then and that's the point when you has to rush? –  Janne Pikkarainen Nov 5 '10 at 15:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.