Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a debian linux webserver. It's just running apache2. Our mysql server is on another host. However we sometimes run cron tasks on the webserver to do regular tasks.

However recently one of the cron tasks had a bug and started to gobble up the memory. The Linux OOM killer killed apache. Which of course brought down our web site. The memory hungry cron kept running. However in this case, I'd like the OOM killer to kill that script, and not apache.

Is there some way to configure the kernel so that I can say don't kill processes called 'apache2' (or at least make apache2 be the last thing it kills)? Both apache and the regular crons are run as the same user (www-user).

share|improve this question
    
Have you considered simply running that task on a separate host? –  Zoredache Jan 21 '10 at 18:04
    
@Zoredache That's one option, yes. I'm partially curious if this is possible –  Rory Jan 21 '10 at 18:24
add comment

5 Answers

It doesn't sound like you are addressing the root cause of the issue by actually debugging why this cron job is using up so much memory.

You can try setting this option

echo 1 > /proc/sys/vm/oom_kill_allocating_task

which will tell the OOM killer to kill the process that triggered the OOM condition, but this is not guaranteed to be your cron job. You can also use "ulimit -m" in your script to set the max amount of resident memory to use. I think your best bet would be to evaluate why the cronjob is using up so much memory and if its perhaps best suited for another host or to be rewritten to consume less memory.

share|improve this answer
    
Yes I know the root cause should be addressed, but I also would like some way to 'prioritize' apache. E.g. I'd rather kill sshd than apache on this server. I'd like to tell the computer this. However I had forgotten about ulimit, that's a good suggestion. –  Rory Jan 21 '10 at 21:47
add comment

The OOMKiller is configurable to an extent. After you have launched a process you can set the value of /proc/<pid>/oom_adj to a negative integer. This will affect the affinity of OOMKiller towards the process and its children . When your system hits an out of memory condition, other processes will be killed.

share|improve this answer
add comment

Short answer: No, the OOM-killer is not configurable, and there is/has been resistance to changing that.

Just a few ideas off the top of my head:

  1. Increase swap space - if the scripts are 32-bit, then it should be easy to give them too much space to exhaust.

  2. Increase physical memory. At 1 pretty much.

  3. Use ulimit to restrict the amount of memory the scripts can take.

share|improve this answer
    
more swap space & ulimit, good idea. –  Rory Jan 21 '10 at 21:49
    
The OOM-killer is configurable, as others have noted. –  MarkR Jan 22 '10 at 8:05
add comment

It says here that you can set a OOM_DISABLE "flag" on a process: http://linux-mm.org/OOM_Killer

share|improve this answer
add comment

You can also change the Virtual Memory over commit behaviour. For example, you can change the value of /proc/sys/vm/overcommit_memory to '2' -- meaning do NOT overcommit memory. (Do not just change this value without understanding what it does.)

In 'no overcommit' mode, whatever new process asks for more ram, will receive an error when it tries to allocate. So rather than have the OOM killer go and wack your old, long running process(es), the new guy asking for RAM is told 'no'.

...and then you need to fix your memory issue. Find the leak, redesign the memory-consuming process, add more ram to the box, etc.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.