Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Every once in a while, my server will kill named. This gets logged:

Apr 16 17:00:08 li127-203 kernel: Out of memory: Kill process 15723 (named) score 38 or sacrifice child
Apr 16 17:00:08 li127-203 kernel: Killed process 15723 (named) total-vm:92096kB, anon-rss:5492kB, filers:0kB

It looks like I'm running out of memory, but why is it always named that gets killed and not some other process? Is there any way to prevent this?

I'm running CentOS 6.2 on a Linode VPS.

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

The OOM killer is a fickle mistress. BIND is getting targeted because it's considered to be the best target by the OOM killer's logic. It's explained well in the code comment on the linked page, but it bears pasting:

/*
 * oom_badness - calculate a numeric value for how bad this task has been
 * @p: task struct of which task we should calculate
 * @p: current uptime in seconds
 *
 * The formula used is relatively simple and documented inline in the
 * function. The main rationale is that we want to select a good task
 * to kill when we run out of memory.
 *
 * Good in this context means that:
 * 1) we lose the minimum amount of work done
 * 2) we recover a large amount of memory
 * 3) we don't kill anything innocent of eating tons of memory
 * 4) we want to kill the minimum amount of processes (one)
 * 5) we try to kill the process the user expects us to kill, this
 *    algorithm has been meticulously tuned to meet the principle
 *    of least surprise ... (be careful when you change it)
 */

But that only tells you why it targeted BIND. The solution probably isn't "get it to target something else", since anything else that's sucking up a lot of memory is probably also important. The solution is more likely to be "don't trigger the OOM killer".

Can you increase the memory or swap space available to this system, or decrease the memory used by other processes? What else is running on this system? Is BIND configured as authoritative, or recursive (which would allow you to limit its cache memory usage)?

share|improve this answer
    
I can increase the swap space, but I'd like to figure out why this is happening before I resort to that. BIND had been running fine until just recently, now it keeps getting killed. I just looked at my running processes and mysql is consuming around 130 MB. Is that normal? Also, I see there's a ton of apache processes running. How can I limit how many processes that it spawns? –  edc1591 Apr 16 '12 at 21:42
    
I've also been considering switching over to nginx. Would that help with the memory problem? –  edc1591 Apr 16 '12 at 21:43
    
It's happening because the system ran out of memory; that's not necessarily BIND's fault, but BIND is getting the punishment for it. (again, is it configured as recursive?) MySQL will use as much RAM for cache as it can, but 130 MB is very reasonable, and Apache can certainly be a hog; its worker process count is determined by which MPM you're using and how you've configured the limits for it. –  Shane Madden Apr 16 '12 at 21:45
    
nginx might be a good choice as it tends to run lighter than Apache, but if, for instance, dynamic content code running within Apache is the cause of the bulk of the memory usage, then changing web servers may not help at all. –  Shane Madden Apr 16 '12 at 21:49
1  
The OOM killer is tangible evidence of badness. It should be disabled, and malloc() allowed to return failure as god and nature intended :-) <end-rant> –  voretaq7 Apr 19 '12 at 3:14
show 6 more comments

There's actually a file in the process tree that allows you to disable or alter the OOM behavior, located at /proc/PROCESS_ID/oom_adj (documentation). To disable, you can do:

echo 17 > /proc/PROCESS_ID/oom_adj

So if you wanted to always apply this to named, you could use:

echo 17 > /proc/$(pidof named)/oom_adj

Keep in mind, though, that $(pidof named) may return more than run pid, although I think named only has one process running.

Hope this helps!

share|improve this answer
    
Thanks, this could help in the short term, but as Shane said in his answer, it's probably not a good idea to just let something else get killed instead. –  edc1591 Apr 16 '12 at 21:44
    
That may be true here, but there are plenty of other places where you may want to prevent the DNS server from being arbitrarily killed, regardless of the memory situation--the point here is, there are other martyrs that can have their memory reclaimed. :) –  Andrew M. Apr 16 '12 at 22:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.