Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 0 down vote favorite

My web server gets inexplicable memory spikes every once in a while (before eventually crashing) and I can't figure out what is causing it or how to solve it. I've spent the whole morning looking at error and access logs and I can't find anything unusual.

I have 3 images:

  1. server status graph:
    http://i.stack.imgur.com/6QnsQ.png

  2. top:
    http://i.stack.imgur.com/ZYY1P.png

  3. ps aux: (edit - site won't let me post the 3rd because I'm too new)

Any help in solving this mystery would be greatly appreciated.

share|improve this question
2  
Why are there no swap file? Is this a OpenVZ container? – pauska Jan 17 at 18:11
Not sure. It's a Media Temple Dedicated Virtual server 4.0. I'll have to investigate. – wired Jan 17 at 18:35
Any reason you're trying to diagnose the memory spike rather than the crash? – David Schwartz Jan 17 at 18:45
I don't know. It starts with the memory spiking and eventually it stops responding to http requests until I restart. – wired Jan 17 at 18:55
I've read that Virtuozzo (Parallels) based vps' don't have a swap. – wired Jan 18 at 1:46
show 1 more comment

3 Answers

up vote 1 down vote

Can you please pastebin the /proc/meminfo output.

Also, I don't know when you took the snapshot of top but I see that you are not using any cached memory. You have any knowledge why so or did you notice this before?

Along with that, can you provide the relevant bits and pieces of the /var/log/messages before the crash.

BTW, one httpd process is in Z state but that won't cause the problem.

Also, look at the sar logs to see if you can identify any memory usage pattern.

share|improve this answer
Thank you for the response. Unfortunatlly the server finally crashed and I had to reboot. Here is ps aux (before crash) and meminfo (after) pastebin.com/ASvSUMrd – wired Jan 17 at 18:24
Here is /var/log/messages around the time of the spike (7:50 i think), but it doesn't look too helpful. i.imgur.com/1Xsrb.png – wired Jan 17 at 18:58
What appears on surface is that you seriously need to bump up the RAM for the server. It just looks like a classical scalability issue, might I say capacity planning ;) You have too many processes running given the limited amount of RAM. Tunings can't help much here and I don't see any parameter which I can suggest you to tune. – Soham Chakraborty Jan 17 at 19:16
Thank you. It's just befuddling that there are these memory spikes with no apparent connected increase in traffic. As soon as the server is restarted, it's back to normal. i.imgur.com/1FCFG.png I'm worried an upgrade won't make much difference. – wired Jan 18 at 1:50
up vote 0 down vote

you have 2G of RAM and you're running MySQL, Apache's HTTPd, qmail, spamd... all this needs memory, so you need to bump up your memory AND set SWAP on your system as well.

share|improve this answer
up vote 0 down vote

Finding what caused the memory spike on a live server is not an easy task. It will be even trickier since you have a range of services, any of which could be the culprit.

As a cheat, I have left top running through a terminal sorted by memory usage. This way, when the server crashes, it leaves my terminal at the last top screen which can at least tell me which process is the likely culprit.

It is then working with that service to try and figure out the culprit.

In our case, it turned out to be caused by the maximum memory for PHP set to too high a value.

The reason this causes a problem is that apache does not clear the memory once a child uses up a certain amount of memory until that child is reaped which may not happen for a long time.

This can build up and eventually finish off all the memory.

I have no experience with qmail so can't comment on its memory usage but mysql on default settings tend to use around 50m as shown on your top.

Good luck

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.