Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm using cgroup to partition my processes and I'm getting Out Of Memory messages in my kernel logs.

However, I can't find which partition causes them. I've checked the memory controller cgroup but there are no obvious ways to use it.

The problem is, by the time I see 'task killed' message in system logs the task is dead, its /proc entry is gone and cgroup's tasks file doesn't have the pid of the killed task.

share|improve this question
    
I thought you wanted to know what caused the OOM. Your problem seems to be about figuring out how the system responded to the OOM. These are two different things. –  David Schwartz Apr 21 '12 at 3:08
    
I need to be able to recover from OOM (which might be caused by a variety of reasons). –  Cyberax Apr 23 '12 at 19:08

1 Answer 1

Answering my own question. I've used SystemTap to hook into the OOM killer:


#!/usr/bin/env stap
%{
#include <linux/cgroup.h>
%}

function find_mem_cgroup:string(task:long) %{
    struct cgroup *cgrp;
    struct task_struct *tsk = (struct task_struct *)((long)THIS->task);

    /* Initialize with an empty value */ 
    strcpy(THIS->__retvalue, "NULL");

    cgroup_lock();
    cgrp = task_cgroup(tsk, mem_cgroup_subsys_id);
    if (cgrp)
        cgroup_path(cgrp, THIS->__retvalue, MAXSTRINGLEN);
    cgroup_unlock();
%}

probe kernel.function("oom_kill_task") {
    cgroup = find_mem_cgroup($p)
    exename = kernel_string($p->comm)
    printf("pid\t%d\tmem-cgroup\t%s\texe-name\t%s\n", $p->pid, cgroup, exename)
}

Works like this:

cyberax@cybnb:~/work/shell$ sudo stap -g oom.stap
pid    3966    mem-cgroup    /task1/1/    exe-name    oom_generator.p
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.