2

I'm involved in maintaining an Ubuntu VPS which runs our django websites (nginx/apache/mod_wsgi) and we've been having some memory spikes which have either caused the database to die, or induced kernel panic when the memory management system can't find any killable processes. I'm working on fixing the memory spikes, but I'm wondering whether there's anything I can do to better deal with the problem if it occurs again.

Are there any tools I could use to detect the memory spikes and then, say, kill the offending process and email the server admin to fix it up? Killing off one website so that the server can remain operational is certainly preferable to the whole thing falling over.

Also, we were charged $600 for after-hours service because we had to get the hosting company to restart the server - is this standard practice among hosting companies? Another provider I work with provides a panel with which I can stop and start the server myself, and given that a restart was all that was needed, $600 seems mightily excessive. (That's NZD, it's around $445 USD)

Improve this question
1

2 Answers 2

Reset to default
1

$600 is definitely expensive for an off hours restart. I'd suggest looking for another host. For comparison, you could get a few months of hosting from a place that offers automatic restarts for that much.

Check out monit for automatically killing processes. You can configure it to watch cpu/memory or even the status of a server, and have it take action if anything looks abnormal.

Improve this answer
2
  • Thanks, I'm investigating monit now and it looks excellent.
    – Greg
    Dec 22, 2010 at 4:19
  • 1
    Monitoring is important, but having your monitoring system automatically kill things is kind of a duct-tape-bandaid approach.
    – mattdm
    Dec 22, 2010 at 4:23
0

You could configure the system to not overcommit memory. Use sysctl to set vm.overcommit to 0.

445 USD seems like a lot, but: did you read the fine print before you had a problem? Did you have a plan for what to do if a reboot were needed? It's one of those things that can be obvious in retrospect, but should be planned for in advance. And, honestly, $445 may be steep, but it isn't a terrible price for that lesson.

Improve this answer
2
  • Thanks for your answer. Just so I'm clear, if I turn off overcommit completely, that will significantly reduce the amount of (apparent) available memory? I'd have hoped it was possible to fix the problem without disabling a core kernel feature.
    – Greg
    Dec 22, 2010 at 4:18
  • No, the apparent available memory will be the same. However, processes that try to allocate more memory than available will get errors (which they will probably not handle, and therefore they will die), instead of being told "yep, that worked! nooooo problem!" even when the system is squeezed. It's not really a "core kernel feature".
    – mattdm
    Dec 22, 2010 at 4:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .