I have a testing web server.

One time the server was so unresponsive that I had to restart it.

Viewing the logs I could see the server was out if memory and oom killer killed mysqld.

But reading some docs about oom killer I know that mysqld wasn't necessarily (but maybe it was) the cause of the out of memory situation.

So only using the log files can I know what process(es) caused the oom condition?

  • I hate it when my servers run out of mana and I need to rest them... ^^ – Oskar Duveborn Nov 14 '09 at 23:52

How do you define the "cause" of the OOM situation? Is it the process using the most memory? Perhaps you have a DB that always takes 3GB of memory to run and thus uses the most memory on the machine. Is it the "cause" of the problem? Probably not.

Ultimately the cause of the problem is "An unexpected situation which may or may not have been the fault of the sysadmin."

Sometimes you can know; for instance if you had process accounting setup (+1 to @JamesHannah) and you saw 3000 httpd or sshd processes (and that was unusual) you could probably blame that daemon.

With that in mind, I present comments from The Source:

 * oom_badness - calculate a numeric value for how bad this task has been
 * @p: task struct of which task we should calculate
 * @p: current uptime in seconds
 * The formula used is relatively simple and documented inline in the
 * function. The main rationale is that we want to select a good task
 * to kill when we run out of memory.
 * Good in this context means that:
 * 1) we lose the minimum amount of work done
 * 2) we recover a large amount of memory
 * 3) we don't kill anything innocent of eating tons of memory
 * 4) we want to kill the minimum amount of processes (one)
 * 5) we try to kill the process the user expects us to kill, this
 *    algorithm has been meticulously tuned to meet the principle
 *    of least surprise ... (be careful when you change it)

"So the ideal candidate for liquidation is a recently started, non privileged process which together with its children uses lots of memory, has been nice'd, and does no raw I/O. Something like a nohup'd parallel kernel build (which is not a bad choice since all results are saved to disk and very little work is lost when a 'make' is terminated)."

Comment block and quote shameless stolen from http://linux-mm.org/OOM_Killer

| improve this answer | |

You can see which processes (with pid) were considered by the OOM killer and which ones were actually killed by running dmesg. but I don't know how to make that go to a log file.

| improve this answer | |

It's only really possible if you had some kind of forensics software installed before the incident, something like sysstat, psacct or similar. Otherwise, you're in the dark.

| improve this answer | |

As was somewhat hinted on before, something like psacct is perfect for this situation with minimal overhead. I use atop and I let logs stick around for at least a year. This doesn't amount to too much data and allows you to trace back all kinds of interesting things you may come up with after the fact. http://www.atcomputing.nl/Tools/atop/

The default is 5 minute time slices and that works out well for tracing back to why the server ran out of memory. The great thing about this is typically you can go back to when the process first started growing in memory and then look at points in the log that might hint to what was going on with that particular application.

| improve this answer | |

We just recently had this problem with a RHEL guest running on VMware. If you are in the same situation, check out the following knowledge base article from VMware: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002704

| improve this answer | |

As an aside, if this is a recurring problem and you want to be able diagnose the cause in the future, a technique which we've used in the past is to create a much larger swap file using:


Obviously, this wont fix the problem, but it should give you sufficient time to get in and diagnose the cause before the machine runs out of memory and keels over.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.