4

We use Nagios to monitor our servers. Now.. linux uses all the RAM we give them so there is no way to detect if we are about to run out of memory. We have a couple of times had some server crashes due to excessive memory usage.

Is there any way to detect if ubuntu(linux) launches the OOM killer? I know its written in the log files but is there anyway to detect when it is launched?

/RJ

1

You can not see when OOM passes by, the only way is to check the logs and hope it's there (sometimes it doesn't get recorded).

There are tools to check memory usage with Nagios, I personally use check_mem.pl

3
  • But how can it monitor RAM usage when linux uses all RAM for cache? linuxatemyram.com When I look at a server with 4GB RAM it has almost none free RAM when looking in top. – Ronnie Jespersen Aug 9 '12 at 21:07
  • There is a switch for that, refer to its manual, it clearly states how you can program it to see the real memory usage and considering how much cache is used. – Lucas Kauffman Aug 9 '12 at 22:41
  • Do you have an example under which conditions oom-killer will not be recorded? I was convinced that oom-killer will always log to dmesg and - provided init and syslog are not badly misconfigured - will always end up in logs. – anx Mar 29 '18 at 15:39
2

Perhaps by logtail.

oom-killer leaves a trace of run in syslog and dmesg. You can put a cron job or a script to invoke via some monitoring tool that "logtails" syslog and warn you if there are lines like the one below between runs

Feb 24 13:35:29 hostname kernel: [22472693.216224] foobar:23 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

dmesg isn't "logtailable" but dmesg -c clears the dmesg buffer after printing so the script can simply call dmesg -c | grep <string> | wc -l A value of 1 or greater means that oom-killer was invoked since the las run/boot. But this will destroy your dmesg stored info.

1

Create /etc/rsyslog.d/30-oom-killer.conf as below:

if ($msg contains [
    "oom",
    "kill"
]) then {
    /var/log/oom-killer.log
    stop
}

Add /var/log/oom-killer.log to /etc/logrotate.d/rsyslog.

0

Obviously it is launched when you don't have enough free virtual memory and an appication requests a memory region from the kernel. So what you really should monitor is your free memory size. And load average is a subject to monitor too because it always gets high when there is not enough free memory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.