Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm involved in maintaining an Ubuntu VPS which runs our django websites (nginx/apache/mod_wsgi) and we've been having some memory spikes which have either caused the database to die, or induced kernel panic when the memory management system can't find any killable processes. I'm working on fixing the memory spikes, but I'm wondering whether there's anything I can do to better deal with the problem if it occurs again.

Are there any tools I could use to detect the memory spikes and then, say, kill the offending process and email the server admin to fix it up? Killing off one website so that the server can remain operational is certainly preferable to the whole thing falling over.

Also, we were charged $600 for after-hours service because we had to get the hosting company to restart the server - is this standard practice among hosting companies? Another provider I work with provides a panel with which I can stop and start the server myself, and given that a restart was all that was needed, $600 seems mightily excessive. (That's NZD, it's around $445 USD)

share|improve this question
    
What Apache MPM are using? How are you using mod_wsgi? Certain combinations are bad for memory usage. Read 'blog.dscpl.com.au/2009/03/…;. This may not be the issue, but worth a read anyway in case it is. Also read 'blog.dscpl.com.au/2009/11/…;. –  Graham Dumpleton Dec 22 '10 at 6:34

2 Answers 2

up vote 1 down vote accepted

$600 is definitely expensive for an off hours restart. I'd suggest looking for another host. For comparison, you could get a few months of hosting from a place that offers automatic restarts for that much.

Check out monit for automatically killing processes. You can configure it to watch cpu/memory or even the status of a server, and have it take action if anything looks abnormal.

share|improve this answer
    
Thanks, I'm investigating monit now and it looks excellent. –  Greg Dec 22 '10 at 4:19
1  
Monitoring is important, but having your monitoring system automatically kill things is kind of a duct-tape-bandaid approach. –  mattdm Dec 22 '10 at 4:23

You could configure the system to not overcommit memory. Use sysctl to set vm.overcommit to 0.

445 USD seems like a lot, but: did you read the fine print before you had a problem? Did you have a plan for what to do if a reboot were needed? It's one of those things that can be obvious in retrospect, but should be planned for in advance. And, honestly, $445 may be steep, but it isn't a terrible price for that lesson.

share|improve this answer
    
Thanks for your answer. Just so I'm clear, if I turn off overcommit completely, that will significantly reduce the amount of (apparent) available memory? I'd have hoped it was possible to fix the problem without disabling a core kernel feature. –  Greg Dec 22 '10 at 4:18
    
No, the apparent available memory will be the same. However, processes that try to allocate more memory than available will get errors (which they will probably not handle, and therefore they will die), instead of being told "yep, that worked! nooooo problem!" even when the system is squeezed. It's not really a "core kernel feature". –  mattdm Dec 22 '10 at 4:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.