Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an Ubuntu machine that works fine most of the time, but every few days one of two things will happen:

  • I'll be logged into the machine and start getting a "cannot allocate memory" message anytime I try to run any command
  • I'll try to SSH into the machine but I'll get an error that says "Connection closed by remote host"

If I log into my host's web interface and restart the server, the problem will completely go away but it always comes back.

I'm not sure how to troubleshoot this. Every time I do top it looks normal. Any suggestions?

Edit: here is my top output (a couple days in, sorted by memory usage): http://pastebin.com/x1CbFLrA

share|improve this question
    
By any chance, have you recently been messing with any sysctls (or anything in the /proc or /sys filesystems)? –  BMDan May 17 '11 at 1:48
    
No, I have not. –  Jason Swett May 17 '11 at 12:12
add comment

3 Answers 3

I would agree with sybreon in that this sounds like a memory leak. If a machine runs fine for a while, eventually become unusable, and works right away after a reboot, then it's almost certainly a memory leak.

The error message you posted in regards to "cannot allocate memory" confirms that as well.

The top command, by default, sorts output via CPU usage. While in top, hit the M button which will sort the output by memory usage. Then, keep an eye on the %MEM column. If you have a process that leaks memory, then it will eventually move its way up to the top.

I would also keep an eye on the 3rd column in the "Mem:" row, which lists the free memory on the machine.

share|improve this answer
    
All right, I'll keep an eye on that and report back. Thanks. –  Jason Swett May 17 '11 at 12:12
    
Note, with regards to "free" memory, the following: linuxatemyram.com –  BMDan May 17 '11 at 22:27
    
Just a few minutes after I checked top, which showed the most memory-hogging process using only 8.4%, I started getting my regular error again: -bash: fork: Cannot allocate memory –  Jason Swett May 17 '11 at 23:47
add comment

Sounds like a load issue. What are the specifications for the server and what is its workload?

Install Ganglia: http://ganglia.sourceforge.net/. It is my preferred load troubleshooting tool. You will like it.

Enable and look at the logs for more clues.

share|improve this answer
    
I have a gig of RAM, 2266MHz processor, 20G hard drive. The workload is almost nothing - a hundred hits a day at most - but I'll try Ganglia. Thanks. –  Jason Swett May 17 '11 at 0:45
    
Could be a memory leak too. –  sybreon May 17 '11 at 0:49
add comment

That's a very small box; is it perhaps a VPS? If so, the box might be oversubscribed, and you're unable to allocate RAM even though your machine thinks there's still free memory.

Alternately, based on:

Just a few minutes after I checked top, which showed the most memory-hogging process using only 8.4%, I started getting my regular error again: -bash: fork: Cannot allocate memory

You might just have MaxClients set too high in your Apache config. The largest one I see on the output of top is 32M or so; on a 1GB VPS, that'd mean a MaxClients of 32 or higher is potentially unsafe*.

*: Yes, this is wildly inaccurate in some circumstances. It's still a good rule of thumb.

share|improve this answer
    
Yes, it is a VPS. I'll check MaxClients. Thanks. –  Jason Swett May 20 '11 at 12:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.