Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I recently made the move to The Cloud from a bare metal server for personal use and I couldn't be happier. Except for one thing: My server crashes about once a week.

Rackspace has been really great and extremely helpful and I wish to stay with them and recommend them to others. But with this issue I have been told that the server is just running out of memory and I need to look into it. While I'm fine with that and this does seem like my issue, I've never heard of a Linux machine crashing from running out of memory. In my experience it would, worst case scenario, run really slowly or the kernel would start killing processes.

Some Details:

  • Running Gentoo Linux (up to date)
  • 512MB RAM, 1G Swap
  • Services installed & running:
    • 2 Apache 2 procs (1 minimal, for serving static/cached resources and proxying; 2nd has cgi, mod_perl and mod_jk)
    • 2 Tomcat Instances (1 has 2 apps I made, the other is just for Nexus)
    • Mysql
  • When the crash happens:
    • server can be pinged
    • ssh connection hangs indefinitely
    • console will allow username but password prompt never comes up. after 60 seconds it times out and I'm presented with another login prompt. repeat.
    • services (http, tomcat, mysql) cannot be contacted; The connection will hang and not report 404, server not found, etc.
    • Softboot will not work, hard boot needs to be performed often more than once (even with a 30 min wait between reboot attempts)

I've run this same setup on servers throughout the years but with only 256MB RAM. The only difference here is I'm on a virtual machine.

My question could take one of two forms: Has anyone had a similar problem with Rackspace cloud or other cloud hosts? (and if so, what was the solution?) or; What's a good way to track down my issue? I setup a cron that outputs uptime and free to a file every minute so I can examine it after the next crash but that seems hacky.

Full Disclosure: I'm a software developer by trade so that's where most of my experience is, but I have about 15 years experience using Linux for desktop and servers both for personal and professional use.

share|improve this question
    
Really slow can be indistinguishable from crashed for a remote user. If it can be pinged it's not truly crashed. Rackspace's explanation sounds reasonable to me. –  ceejayoz May 18 '12 at 14:32
    
@ceejayoz Agreed. Users aside, if I could at least get in while it was "really slow" I could diagnose the issue. In this case I've waited hours to get into the server in hopes to catch a runaway process but it eventually needed to be rebooted. –  Josh Johnson May 18 '12 at 14:36
1  
You should get as much status information from your server as possible. Definitely monitoring its load at few minutes interval? Then see if any strange patterns show before the server stops responding. –  Dmitri Chubarov May 18 '12 at 14:44
    
A tool like ganglia could be helpful. –  Dmitri Chubarov May 18 '12 at 14:50
    
@DmitriChubarov Much appreciated, I've installed ganglia and look forward to seeing the data. –  Josh Johnson May 18 '12 at 15:41
show 4 more comments

1 Answer

Another great way to track your memory usage would be to install sar on your linux box. If you're using debian it can be listed as sysstat. Using sar -r will give you a picture of what's going on in your memory and network statistics.

Lindsey

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.