I have a godaddy server that has been becoming unresponsive periodically. It was difficult to troubleshoot because I can't ssh into it when it becomes unresponsive. I figured out what was happening by adding a cron job that piped output from "top" to log files every 5 minutes. The next time I power cycled it after it became unresponsive I checked those logs and found that the ram was maxed out, but the swap was mostly unused.
I'm working on reducing ram usage by the two app servers on that machine (it turns out too many connections were getting opened. Each one used up 30m, so after 40 got opened the server runs out of ram), but what I'd really like to know is how to ensure I can ssh into the machine.
If the swap file isn't full then I'd think there'd be enough space for the server to respond, even if it did so slowly. Is there any way I can reserve a bit of ram so that I can always ssh into the machine?
Here is an example of what it looks like when the server is running normally:
top - 15:13:21 up 3:12, 2 users, load average: 0.15, 0.30, 0.33 Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie Cpu(s): 2.4%us, 1.8%sy, 0.0%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 2064980k total, 1611252k used, 453728k free, 45852k buffers Swap: 2096472k total, 0k used, 2096472k free, 790212k cached
Here is the last top log that got logged before the server stopped running:
top - 14:45:08 up 15:20, 0 users, load average: 0.27, 0.16, 0.10 Tasks: 141 total, 2 running, 139 sleeping, 0 stopped, 0 zombie Cpu(s): 2.7%us, 1.9%sy, 0.0%ni, 95.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2064980k total, 2007652k used, 57328k free, 60496k buffers Swap: 2096472k total, 100k used, 2096372k free, 689584k cached
Note that the cron job logging the "top" output also stops running when the server runs out of ram, so the whole server just grinds to a halt apparently.