Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a Linux machine which runs ~10 in house written processes.

Every other day(!) the machine completely runs out of RAM, goes into swap and becomes unresponsive. This happens quickly over a period of a couple of seconds, so it's not feasible to sit watching the machine until it dies. It's a sudden leak, not a gradual one, so top(1) doesn't give any indications something bad may about to happen.

What is the best way of identifying which process(es) are causing the trouble?

share|improve this question
    
give us a hint about these processes: what are they doing, is it interpreted or native code? What are the developers saying? –  cstamas Jun 25 '11 at 13:01

2 Answers 2

up vote 3 down vote accepted

You can look into disabling memory overcommit so the kernel would just kill the leaking process instead of the machine.

You can disable overcommit by using the following two sysctl:

vm.overcommit_ratio=60
vm.overcommit_memory=2

The first number depends on how much memory and how much swap you have. The allocation limit is going to be:

amount_of_swap + overcommit_ratio/100 * amount_of_ram
share|improve this answer

You can set memory limits for each process with bash ulimit command to prevent them from eating all system memory. To get notification I recommend using monit to monitor each process memory usage.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.