I recently made the move to The Cloud from a bare metal server for personal use and I couldn't be happier. Except for one thing: My server crashes about once a week.
Rackspace has been really great and extremely helpful and I wish to stay with them and recommend them to others. But with this issue I have been told that the server is just running out of memory and I need to look into it. While I'm fine with that and this does seem like my issue, I've never heard of a Linux machine crashing from running out of memory. In my experience it would, worst case scenario, run really slowly or the kernel would start killing processes.
- Running Gentoo Linux (up to date)
- 512MB RAM, 1G Swap
- Services installed & running:
- 2 Apache 2 procs (1 minimal, for serving static/cached resources and proxying; 2nd has cgi, mod_perl and mod_jk)
- 2 Tomcat Instances (1 has 2 apps I made, the other is just for Nexus)
- When the crash happens:
- server can be pinged
- ssh connection hangs indefinitely
- console will allow username but password prompt never comes up. after 60 seconds it times out and I'm presented with another login prompt. repeat.
- services (http, tomcat, mysql) cannot be contacted; The connection will hang and not report 404, server not found, etc.
- Softboot will not work, hard boot needs to be performed often more than once (even with a 30 min wait between reboot attempts)
I've run this same setup on servers throughout the years but with only 256MB RAM. The only difference here is I'm on a virtual machine.
My question could take one of two forms: Has anyone had a similar problem with Rackspace cloud or other cloud hosts? (and if so, what was the solution?) or; What's a good way to track down my issue? I setup a cron that outputs
free to a file every minute so I can examine it after the next crash but that seems hacky.
Full Disclosure: I'm a software developer by trade so that's where most of my experience is, but I have about 15 years experience using Linux for desktop and servers both for personal and professional use.