5

I have a linux server for one of my web applications. Every once in a while, a process (a long running script maybe) might go out of hand, consume too much memory / cpu cycles maybe and block all other processes.

In such situations, I can't ssh into the server, and I need to restart the server through a management panel. I'd prefer to login into the machine and deal with the problematic process only.

Is it possible to arrange the resources in a linux machine such that, however a process is consuming resources, there is always enough resources available for an ssh connection?

  • maybe have a look at what causes your application to "go out of hand"... – SnakeDoc Nov 3 '15 at 18:16
5

You can use 'nice' to prioritize certain software.

You could also look into installing monit, which you can instruct to restart a certain package if a certain threshold is met.

A monit config along these lines will automatically restart Apache:

check process apache
   with pidfile "/usr/local/apache/logs/httpd.pid"
   start program = "/etc/init.d/httpd start" with timeout 60 seconds
   stop program = "/etc/init.d/httpd stop"
   if 2 restarts within 3 cycles then timeout
   if totalmem > 100 Mb then alert
   if children > 255 for 5 cycles then stop
   if cpu usage > 95% for 3 cycles then restart
   if failed port 80 protocol http then restart
   group server
   depends on httpd.conf, httpd.bin
2

In simple way - no. You can use nice to set SSH to highest priority, but if there is not enough memory to handle new connection, ssh will not work (don't forget that after successful login server has to start shell). You can use OOM killer to automatically kill process with too many ram consumed, but it didn't work if you will have thousands of process (like crazy apache forking) and every consumes little bit of RAM (1000 x 4MB RAM = 4GB RAM consumed without OOM limit).

Hard restart is simplest and fastest solution. If you need some services to be running 24/7, you need to use two machines in HA setup. You can use zabbix or another monitoring tool for warning and have time to solve it before whole server crashs too.

0

One simple solution is to renice your sshd process to -20. This may still make it unresponsive if it cannot fork (i.e. there are too many processes or so little memory), and it will be slow if it need to be reloaded from swap, but it significantly reduces the likelyhood. This also has the side effect that every ssh login will have a -20 priority, and - if you are not careful - each process started from an ssh shell will have such high priority. You may or may not want that to happen.

There is also a way to positively disable swapping for a process, but you would need to patch your sshd for that to call mlockall or madvise, but you probably do not want to do that. See https://stackoverflow.com/questions/578137/can-i-tell-linux-not-to-swap-out-a-particular-processes-memory for details.

0

You can use memory cgroups to ensure that Apache and the processes it creates cannot starve the rest of the system of memory. The system is likely becoming unresponsive because Apache is consuming enough memory that the system starts swapping.

You can setup the crogups with cgcreate, cgset, cgclassify and cgexec tools.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.