I'm using cgroup to partition my processes and I'm getting Out Of Memory messages in my kernel logs.

However, I can't find which partition causes them. I've checked the memory controller cgroup but there are no obvious ways to use it.

The problem is, by the time I see 'task killed' message in system logs the task is dead, its /proc entry is gone and cgroup's tasks file doesn't have the pid of the killed task.

  • I thought you wanted to know what caused the OOM. Your problem seems to be about figuring out how the system responded to the OOM. These are two different things. – David Schwartz Apr 21 '12 at 3:08
  • I need to be able to recover from OOM (which might be caused by a variety of reasons). – Cyberax Apr 23 '12 at 19:08

Answering my own question. I've used SystemTap to hook into the OOM killer:

#!/usr/bin/env stap
#include <linux/cgroup.h>

function find_mem_cgroup:string(task:long) %{
    struct cgroup *cgrp;
    struct task_struct *tsk = (struct task_struct *)((long)THIS->task);

    /* Initialize with an empty value */ 
    strcpy(THIS->__retvalue, "NULL");

    cgrp = task_cgroup(tsk, mem_cgroup_subsys_id);
    if (cgrp)
        cgroup_path(cgrp, THIS->__retvalue, MAXSTRINGLEN);

probe kernel.function("oom_kill_task") {
    cgroup = find_mem_cgroup($p)
    exename = kernel_string($p->comm)
    printf("pid\t%d\tmem-cgroup\t%s\texe-name\t%s\n", $p->pid, cgroup, exename)

Works like this:

cyberax@cybnb:~/work/shell$ sudo stap -g oom.stap
pid    3966    mem-cgroup    /task1/1/    exe-name    oom_generator.p

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.