Automatically Kill/Restart Process(es) When Memory is Critically Low

Question

I have a Debian Wheezy VPS box where am running a couple of Django apps in production. Ideally, would have tried addressed my current memory footprint issues by optimizing the apps, adding more RAM or augmenting with Swap. But the problem is that I doubt there's much memory optimization I'd milk from optimizing the Django apps (the stack being open-source and robust), and adding RAM is a cost constraint for me (this is a remote VPS), also, the host doesn't offer options to use Swap!

So, in the meantime (as I wait to secure more resources to afford more RAM), I wish to mitigate the scenarios where the server runs out memory so that I just have to request a VPS restart (as in, at that point, I can't even SSH into the box!).

So, what I would love in a solution is the ability to detect when a process (or generally, total system memory usage) exceeds a certain critical amount (for now, example the FREE RAM falls to say 10%) - which I've noticed occurs after the VPS's been up for long, and when also traffic is suddenly much to some of the heavy apps (most are just staging apps anyway).

So, I wish to be able to kill/restart the offending process(es) - most likely Apache. Which solution when done manually in these situations has restored sane memory usage levels - a hint that possibly one or more of the Django apps has a memory leak?

---

In brief:

1. Monitor overall system RAM usage
2. When FREE RAM falls below a given critical threshold (say below 10%), kill/restart the offending process(es) - or simpler, if we assume from my current log analysis (using linux-dash) that Apache is often the offender, then kill/restart it.
3. Rinse and repeat...

Answer 1

The linux kernel has a so-called OOM Killer built-in. It is the "Out of memory killer". So when your box has exhausted its ram & swap, the kernel will start killing stuff to make the server accessible.

You can tweak the priorities of processes, to determine the "likelihood" of a process being killed. Read more at this link, see section "Configuring the OOM Killer".

Basically, you adjust the likelihood in the /proc/*/oom_adj file. Eg. raise the likelihood of killing any of the currently running apache instances?

pgrep apache2 |sudo xargs -I %PID sh -c 'echo 10 > /proc/%PID/oom_adj'

Or lower the likelihood that SSH will get killed:

pgrep sshd |sudo xargs -I %PID sh -c 'echo -17 > /proc/%PID/oom_adj'

Also i recommend completely disabling swap on a server where you have this issue, because swap is so slow that it can grind the server to a virtual standstill, even though theres still swapspace left, thus never triggering the OOM killer.

Answer 2

If these apps are running inside an apache2 server, you can tune the server. Consider:

• Limit the MaxRequestWorkers (This limits number of workers using memory).
• Limit the MaxConnectionsPerChild (This recycles servers so that they don't consume to much memory. This is useful if the applications are leaking memory.

If your processes are leaking memory, you can use /etc/security/limits.conf to limit the amount of memory a server can contain. This will prevent servers from growing too large. The same effect can be achieved on a temporary basis using the ulimit command. It may be best to use ulimit to discover an appropriate size, and then set those values in the limits.conf file. If your server supports it, drop a file into /etc/security/limits.d rather than editing /etc/security/limits.conf.

Answer 3

First of all I would say restart is not a problem of solution and the better way is to found the offending process and why it's consuming high memory. Like mentioned above linux already have OOM mechanism to find the offending process and to kill it to release memory pressure

Other way to find it out using Kdump,configure this parameter vm.panic_on_oom = 1(/etc/sysctl.conf),this will generate vmcore when system is going out of memory.You can find more info about it here

http://people.redhat.com/anderson/crash_whitepaper/

Also limits.conf has lot of limitation,better Solution is to use cgroups to restrict the memory utilization per process

So in /etc/cgconfig.conf you can define the control group like this(here I am restricting my app to use only 256MB of memory)

    group test {
                 memory {
                         memory.limit_in_bytes = 256m;
                 }
          }

and then in /etc/cgrules.conf I can define that your app utilzation(in your case django can't exceed beyond 256)

    *:django        memory          test/

For more info about cgroup you can refer

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

But the idea of restarting the app seems to be bad.