8

We have a debian linux webserver. It's just running apache2. Our mysql server is on another host. However we sometimes run cron tasks on the webserver to do regular tasks.

However recently one of the cron tasks had a bug and started to gobble up the memory. The Linux OOM killer killed apache. Which of course brought down our web site. The memory hungry cron kept running. However in this case, I'd like the OOM killer to kill that script, and not apache.

Is there some way to configure the kernel so that I can say don't kill processes called 'apache2' (or at least make apache2 be the last thing it kills)? Both apache and the regular crons are run as the same user (www-user).

2
  • Have you considered simply running that task on a separate host?
    – Zoredache
    Jan 21, 2010 at 18:04
  • @Zoredache That's one option, yes. I'm partially curious if this is possible Jan 21, 2010 at 18:24

5 Answers 5

3

It doesn't sound like you are addressing the root cause of the issue by actually debugging why this cron job is using up so much memory.

You can try setting this option

echo 1 > /proc/sys/vm/oom_kill_allocating_task

which will tell the OOM killer to kill the process that triggered the OOM condition, but this is not guaranteed to be your cron job. You can also use "ulimit -m" in your script to set the max amount of resident memory to use. I think your best bet would be to evaluate why the cronjob is using up so much memory and if its perhaps best suited for another host or to be rewritten to consume less memory.

1
  • Yes I know the root cause should be addressed, but I also would like some way to 'prioritize' apache. E.g. I'd rather kill sshd than apache on this server. I'd like to tell the computer this. However I had forgotten about ulimit, that's a good suggestion. Jan 21, 2010 at 21:47
3

The OOMKiller is configurable to an extent. After you have launched a process you can set the value of /proc/<pid>/oom_adj to a negative integer. This will affect the affinity of OOMKiller towards the process and its children . When your system hits an out of memory condition, other processes will be killed.

1

You can also change the Virtual Memory over commit behaviour. For example, you can change the value of /proc/sys/vm/overcommit_memory to '2' -- meaning do NOT overcommit memory. (Do not just change this value without understanding what it does.)

In 'no overcommit' mode, whatever new process asks for more ram, will receive an error when it tries to allocate. So rather than have the OOM killer go and wack your old, long running process(es), the new guy asking for RAM is told 'no'.

...and then you need to fix your memory issue. Find the leak, redesign the memory-consuming process, add more ram to the box, etc.

0

Short answer: No, the OOM-killer is not configurable, and there is/has been resistance to changing that.

Just a few ideas off the top of my head:

  1. Increase swap space - if the scripts are 32-bit, then it should be easy to give them too much space to exhaust.

  2. Increase physical memory. At 1 pretty much.

  3. Use ulimit to restrict the amount of memory the scripts can take.

2
  • more swap space & ulimit, good idea. Jan 21, 2010 at 21:49
  • The OOM-killer is configurable, as others have noted.
    – MarkR
    Jan 22, 2010 at 8:05
0

It says here that you can set a OOM_DISABLE "flag" on a process: http://linux-mm.org/OOM_Killer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .