2

We're running an Ubuntu instance on Amazon EC2, and we recently had an unexpected (and currently unexplained) crash (not related to their recent outage). The problem was basically that the machine would not respond to anything (SSH / HTTP), although everything in the AWS console would suggest the instance was running fine (CPU Usage was at 0 in CloudWatch).

Ultimately, we have resorted to rebooting the instance, which seems to have restored it, for now. I have:

  • checked all the logs I could think of (dmesg, syslog, etc.) and I could not find anything error-y;
  • checked the system log from the aws console before rebooting the machine, nothing error-y;
  • looked at CloudWatch and I can't see anything out of place, the CPU Utilization just seems to suddenly drop to 0 without any spikes before that. All other metrics are steady as well;
  • ensured there is enough storage space on the instance (so no out of hdd memory errors occur).

I don't know what else to do / check.

Is there anything I can do to prevent this from happening again? It has caused us quite some downtime. If not, what should I do if it happens again? Anything else besides rebooting?


Update

It's done it again. The instance was installed barely a couple of weeks ago (soon after amazon's outage). It runs perfectly fine until one moment when it just goes unresponsive. Again, there was someone logged in 10 minutes before, running htop and everything was fine.


Update #2

It crashed again. Here are some graphs from AWS CloudWatch (we have to EBS volumes, one for booting and one for data):

1. DiskReadBytes (instance metric)

DiskReadBytes

Note: the hang happened after the spike close to 19:00

2. VolumeReadBytes (EBS metric, data EBS)

VolumeReadBytes

3. VolumeReadBytes(EBS metric, boot EBS)

VolumeReadBytes

I have no idea what an interrupted line means.

2
  • What systems are running on it that you added (like Apache, Tomcat, etc)? May 3, 2011 at 20:35
  • Apache (+PHP with APC) and MySQL. I did not check their logs, thinking one of them failing couldn't bring the entire system to a freeze.
    – Felix
    May 3, 2011 at 21:20

1 Answer 1

1

It's unlikely that an unmodified Ubuntu would do this, I would suspect one of the added apps has caused this. I don't notice RAM usage in your list, maybe it would be worth storing the output of top every hour in a log file using cron. You may simply have a leak or not enough RAM.

1
  • RAM usage was fine. I was logged in right before the hang and was running htop and everything looked normal.
    – Felix
    May 5, 2011 at 18:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .