We're running an Ubuntu instance on Amazon EC2, and we recently had an unexpected (and currently unexplained) crash (not related to their recent outage). The problem was basically that the machine would not respond to anything (SSH / HTTP), although everything in the AWS console would suggest the instance was running fine (CPU Usage was at 0 in CloudWatch).
Ultimately, we have resorted to rebooting the instance, which seems to have restored it, for now. I have:
- checked all the logs I could think of (
dmesg
,syslog
, etc.) and I could not find anything error-y; - checked the system log from the aws console before rebooting the machine, nothing error-y;
- looked at CloudWatch and I can't see anything out of place, the CPU Utilization just seems to suddenly drop to 0 without any spikes before that. All other metrics are steady as well;
- ensured there is enough storage space on the instance (so no out of hdd memory errors occur).
I don't know what else to do / check.
Is there anything I can do to prevent this from happening again? It has caused us quite some downtime. If not, what should I do if it happens again? Anything else besides rebooting?
Update
It's done it again. The instance was installed barely a couple of weeks ago (soon after amazon's outage). It runs perfectly fine until one moment when it just goes unresponsive. Again, there was someone logged in 10 minutes before, running htop
and everything was fine.
Update #2
It crashed again. Here are some graphs from AWS CloudWatch (we have to EBS volumes, one for booting and one for data):
1. DiskReadBytes (instance metric)
Note: the hang happened after the spike close to 19:00
2. VolumeReadBytes (EBS metric, data EBS)
3. VolumeReadBytes(EBS metric, boot EBS)
I have no idea what an interrupted line means.