On our cluster we would sometimes have nodes go down when a new process would request too much memory. I was puzzled why the OOM killer does not just kill the guilty process.

The reason turned out to be that some processes get -17 oom_adj. That makes them off-limits for OOM killer (unkillabe!).

I can clearly see that with the following script:

for i in `grep -v 0 /proc/*/oom_adj | awk -F/ '{print $3}' | grep -v self`; do
  ps -p $i | grep -v CMD

OK, it makes sense for sshd, udevd, and dhclient, but then I see regular user processes get -17 as well. Once that user process causes an OOM event it will never get killed. This causes OOM kiler to go insane. NFS rpc.statd, cron, everything that happened to to be not -17 will be wiped out. As a result the node is down.

I have Debian 6.0 (Linux 2.6.32-3-amd64).

Does anyone know where to contorl the -17 oom_adj assignment behaviour?

Could launching sshd and Torque mom from /etc/rc.local be causing the overprotective behaviour?


2 Answers 2


It gets inherited from the process that spawned it. If SSH is set to -17 then Bash will be. If you restart via Bash, you'll spawn it even further.

[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
[i-180ae177] root@migrantgeek ~ # cat /proc/11395/oom_adj 
[i-180ae177] root@migrantgeek ~ # for pid in `pgrep bash`; do echo -17 >  /proc/$pid/oom_adj; done
[i-180ae177] root@migrantgeek ~ # /etc/init.d/mysqld  restart
Stopping MySQL:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]
[i-180ae177] root@migrantgeek ~ # pgrep mysqld_safe
[i-180ae177] root@migrantgeek ~ # cat /proc/11523/oom_adj 

Editing the init script to change the value at the end of the startup process should fix this.

  • So -17 on init (pid 0) is a bad idea? May 21, 2011 at 1:21
  • @Seth, I think you are right. I will test now and then mark this as the correct answer. May 21, 2011 at 1:42
  • @Seth, I un-17'd all processes but still when I open a new ssh connection the sshd and the shell get -17 May 21, 2011 at 2:13
  • Hmmm. Can you run the following?
    – Jim
    May 21, 2011 at 4:27
  • for pid in pgrep ssh; do cat /proc/$pid/oom_adj; done
    – Jim
    May 21, 2011 at 4:27

On our clusters we disable overcommit with sysctl:


You should fix the ratio depending on how much memory and swap you have.

Once overcommit is disabled the kernel just returns NULL to the process that is trying to allocate too much memory. It solved all our memory crashes on the cluster nodes.

  • I would caution against using overcommit - on a mixed web app + DB (Python + PostgreSQL) system, with background processes also allocating quite a lot of RAM, the overcommit=2 setting with ratio=100 just changed the OOM killing behaviour into many fork() failures (no more COW forking i.e. vfork), and many Python memory allocation errors (MemoryError). There wasn't really enough RAM on this system, but due to the forking issue and the need for some apps to allocate more virtual memory than will ever be used, the cure was worse than the disease. The solution was to add more RAM and/or swap.
    – RichVel
    Apr 4, 2013 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.