Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Similar to this question, we have a computing server with 96GB of RAM that is used to run large jobs in parallel.

Occasionally, the total amount of physical RAM is exceeded, which causes the server to become unresponsive, forcing a reboot. To me, this is not acceptable behavior, so I'm looking for ways to fix this.

I know one way would be to set limits using "ulimit -v". However I'd like to avoid going down that route if possible, as I may occasionally have a signle very large process (as opposed to many small ones), so setting a useful threshold is going to be difficult.

I suspect the problem may come from the fact that the system has 20GB of swap: instead of killing the offending process(es), the system will allocate memory on disk which will make it unresponsive. Is reducing the amount of swap a good idea?

Any insight or experiences with a similar problem highly appreciated!

EDIT

I made a few experiments using the following leaking C++ program:

#include <vector>
#include <unistd.h>

using namespace std;

int main(int argc,char * argv[])
{
        while(true) {
                vector<double>* a = new vector<double>(50000000);
                sleep(1);
        }
}

I ran it a first time with a 256MB swap file. The system completely hung for about 5 minutes, than came back to life. In the logs, I saw that the OOM killer had succesfully killed my leaky program.

I ran it a second time with no swap. This time, the machine didn't come back to life for at least ten minutes, at which point I rebooted the machine. This came as a surprise for me, as I expected the OOM killer to fire up earlier on a machine with no swap.

What I don't understand is the following: why does linux wait until the system is completely hung to do something about the offending process? Is it too much to expect of an OS to not be completely killed by one badly coded process?

share|improve this question
add comment

2 Answers 2

If you want your server to still be responsive, you need to do your best to avoid swapping. However, reducing the swap amount or disabling it will not solve your problem.

You need either to control your jobs memory usage or install more memory ships in the server machine.

You can try cgroups (control groups) to control your processes CPU and memory usage.

share|improve this answer
    
Why exactly will removing the swap not fix my problem? How exactly will cgroups help me with this problem? Is it not about the same as using ulimit? –  static_rtti Jan 21 '13 at 13:48
    
@static_rtti: You can create a group of processes and impose some limits on them. At least, this should solve your problem of creating one or many processes. For swap, removing swap can make things even worse as your server crashes when it needs to allocate more memory and it is completely exhausted. –  Khaled Jan 21 '13 at 13:56
    
isn't the OOM killer supposed to save the day when there is no memory left? Also, the cgroup thing looks eerily similar to what you can do with ulimit. –  static_rtti Jan 21 '13 at 15:54
add comment

Do you have the oom logs in /var/log/messages.

I would suggest to check the /proc/meminfo and monitor it at peak usage hours. The Committed_AS parameter shows how much memory will be needed if the kernel allows all the memory requests by all processes at that time. If that value is below free memory, then there is potential chance that the system will eventually go OOM.

As always, fit the workload that the hardware can accommodate. It is best to either increase RAM or reduce workload.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.