Server grinds to a halt every few days

Question

I have an Ubuntu machine that works fine most of the time, but every few days one of two things will happen:

• I'll be logged into the machine and start getting a "cannot allocate memory" message anytime I try to run any command
• I'll try to SSH into the machine but I'll get an error that says "Connection closed by remote host"

If I log into my host's web interface and restart the server, the problem will completely go away but it always comes back.

I'm not sure how to troubleshoot this. Every time I do top it looks normal. Any suggestions?

Edit: here is my top output (a couple days in, sorted by memory usage): http://pastebin.com/x1CbFLrA

Answer 1

I would agree with sybreon in that this sounds like a memory leak. If a machine runs fine for a while, eventually become unusable, and works right away after a reboot, then it's almost certainly a memory leak.

The error message you posted in regards to "cannot allocate memory" confirms that as well.

The top command, by default, sorts output via CPU usage. While in top, hit the M button which will sort the output by memory usage. Then, keep an eye on the %MEM column. If you have a process that leaks memory, then it will eventually move its way up to the top.

I would also keep an eye on the 3rd column in the "Mem:" row, which lists the free memory on the machine.

Answer 2

Sounds like a load issue. What are the specifications for the server and what is its workload?

Install Ganglia: http://ganglia.sourceforge.net/. It is my preferred load troubleshooting tool. You will like it.

Enable and look at the logs for more clues.

Answer 3

That's a very small box; is it perhaps a VPS? If so, the box might be oversubscribed, and you're unable to allocate RAM even though your machine thinks there's still free memory.

Alternately, based on:

Just a few minutes after I checked top, which showed the most memory-hogging process using only 8.4%, I started getting my regular error again: -bash: fork: Cannot allocate memory

You might just have MaxClients set too high in your Apache config. The largest one I see on the output of top is 32M or so; on a 1GB VPS, that'd mean a MaxClients of 32 or higher is potentially unsafe*.

*: Yes, this is wildly inaccurate in some circumstances. It's still a good rule of thumb.