Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On our cluster we would sometimes have nodes go down when a new process would request too much memory. I was puzzled why the OOM killer does not just kill the guilty process.

The reason turned out to be that some processes get -17 oom_adj. That makes them off-limits for OOM killer (unkillabe!).

I can clearly see that with the following script:

#!/bin/bash
for i in `grep -v 0 /proc/*/oom_adj | awk -F/ '{print $3}' | grep -v self`; do
  ps -p $i | grep -v CMD
done

OK, it makes sense for sshd, udevd, and dhclient, but then I see regular user processes get -17 as well. Once that user process causes an OOM event it will never get killed. This causes OOM kiler to go insane. NFS rpc.statd, cron, everything that happened to to be not -17 will be wiped out. As a result the node is down.

I have Debian 6.0 (Linux 2.6.32-3-amd64).

Does anyone know where to contorl the -17 oom_adj assignment behaviour?

Could launching sshd and Torque mom from /etc/rc.local be causing the overprotective behaviour?

share|improve this question

2 Answers 2

It gets inherited from the process that spawned it. If SSH is set to -17 then Bash will be. If you restart via Bash, you'll spawn it even further.

[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
11395
[i-180ae177] root@migrantgeek ~ # cat /proc/11395/oom_adj 
0
[i-180ae177] root@migrantgeek ~ # for pid in `pgrep bash`; do echo -17 >  /proc/$pid/oom_adj; done
[i-180ae177] root@migrantgeek ~ # /etc/init.d/mysqld  restart
Stopping MySQL:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
11523
[i-180ae177] root@migrantgeek ~ # cat /proc/11523/oom_adj 
-17

Editing the init script to change the value at the end of the startup process should fix this.

share|improve this answer
    
So -17 on init (pid 0) is a bad idea? –  Aleksandr Levchuk May 21 '11 at 1:21
    
@Seth, I think you are right. I will test now and then mark this as the correct answer. –  Aleksandr Levchuk May 21 '11 at 1:42
    
@Seth, I un-17'd all processes but still when I open a new ssh connection the sshd and the shell get -17 –  Aleksandr Levchuk May 21 '11 at 2:13
    
Hmmm. Can you run the following? –  Seth May 21 '11 at 4:27
    
for pid in pgrep ssh; do cat /proc/$pid/oom_adj; done –  Seth May 21 '11 at 4:27

On our clusters we disable overcommit with sysctl:

vm.overcommit_ratio=60
vm.overcommit_memory=2

You should fix the ratio depending on how much memory and swap you have.

Once overcommit is disabled the kernel just returns NULL to the process that is trying to allocate too much memory. It solved all our memory crashes on the cluster nodes.

share|improve this answer
    
I would caution against using overcommit - on a mixed web app + DB (Python + PostgreSQL) system, with background processes also allocating quite a lot of RAM, the overcommit=2 setting with ratio=100 just changed the OOM killing behaviour into many fork() failures (no more COW forking i.e. vfork), and many Python memory allocation errors (MemoryError). There wasn't really enough RAM on this system, but due to the forking issue and the need for some apps to allocate more virtual memory than will ever be used, the cure was worse than the disease. The solution was to add more RAM and/or swap. –  RichVel Apr 4 '13 at 14:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.