I'm running Apache webserver and would like to improve a little bit how the OOM situation is handled.
I'm avare of the OOM scores and already did some customizations in that matter so when something bad happens, Linux is killing the correct processes. But its not enough.
The problem is that sometimes when OOM occurs the server gets overloaded and afterwards crashes and must be restarted. I would like to handle that without the full restart of the server. So I need to somehow "hook" a script on OOM killer invocation which would kill all apache (and its CGIs) processes, thus freeing the memory and start it (Apache) again.
I know this would work, because if OOM occurs and I'm fast enough to login to the server and kill the Apache manualy, everything is OK then.
FYI I'm running now nearly a hundred of those webservers, thats why I'm looking for fully automatic solution.
One possible solution would of course be to use some watchdog that would parse the syslog and detect OOMs in this manner - I already have something like that, which notifies about OOM killings by e-mail. This aproach can solve some situations but if the OOM is realy bad, the server is too much overloaded and my script does not even start (its run by cron). It can be improved by using inotify to watch the syslog or by piping the syslog directly (i.e. by fifo) to the script.
But still I'm wondering - isn't there any way how to "hook" the script directly to OOM killer? So i would put something like that in some /etc/.. file:
oom_action="sh /path/to/my/script.sh kill"
Or its simply not possible to do it like that?
I'm using Centos 6, Apache 2.2 and PHP as FastCGI.