7

I have an Amazon EC2 instance running LAMP on Ubuntu Natty/11.04. On three separate occasions within the last few months, two of which in the last two weeks, the server has just... stopped. It becomes unresponsive and stops responding to connection attempts (SSH or otherwise), but the EC2 control panel still reports it as running. Each time I had to reboot the instance through the console, with ensuing data loss.

So, now I'm trying to diagnose the issue, but I'm coming up blank and I need advice on what else to check for. Syslog contains nothing suspicious -- on each occasion, the last thing that happened was munin running its regular five-minute cronjob, although since I don't know exactly when the machine stopped working, I can't say how close the cron log is to the point of freezing. After that, it's as if the machine was simply not running until the point where it was restarted, after which point syslog contains what looks to me like normal dmesg output.

There seems to be no correlation between traffic volume and the time of these freezes. Each occasion has been far removed from peak traffic times.

What else can I look at to attempt to figure out what has been causing these issues? What might the issue be?

ADDENDUM: The server was not under heavy load at any occasion when it went down. CPU and memory use were both well and safely under limits. There was plenty of free disk space (tens of gigabytes). There is nothing strange in Apache or MySQL logs either, they just stop operating at that time. This is a medium/high-CPU instance.

3
  • If you don't have paid support, post this along with the failed instance ID and AMI ID to the EC2 forum. You should get a response from their support team. Nov 24, 2011 at 9:32
  • Well, we have had pretty abysmal experiences with the support forum so far. Usually all we get is a "we'll get back to you on that" from a support guy and then they never follow up. Besides, I was hoping to get some general troubleshooting advice so I can at least attempt to research this myself.
    – pjohansson
    Nov 24, 2011 at 10:03
  • Could your cron jobs be using any ports not allowed by your security groups? Oct 22, 2015 at 23:26

2 Answers 2

4

First thing you should do is setup some monitioring to let you know when the server becomes unresponsive. You can do this by using pingdom and/or cloudwatch to check service uptime and system stats like cpu and ram. Both have free plans for small accounts. This will allow you to get an idea when it goes down and should make it easier to hunt the logs as to what was going on at that moment. Usully something like this might be caused by the system not having enough resources, you dont mention what is the size of you instance but something like a micro could be just pegging 100% cpu by a simple cron job and at which point server just locks up.

Aloso check other logs beside syslog, check all app logs to see if any of them are throwing an error before your system goes down.

2
  • We do have monitoring tools set up, but I only know within an accuracy of 15 minutes when the server went down. As mentioned, I can tell where in the logs the server went down -- but there is nothing incriminating there. Syslog and Apache/MySQL/PHP logs are all clean. This is a Medium-sized instance and the CPU was not under any heavy load when the server went down.
    – pjohansson
    Nov 24, 2011 at 13:13
  • 2
    In this case you might just have a bad instance. AWS does not say much about this but as the company I work for is pretty much all AWS we will see once in a blue moon a server being deployed on a bad instance maybe because they over provisioned the box. Your only choice in that case is new instance and move.
    – Zuhaib
    Nov 25, 2011 at 3:00
-4

Could be a bad NIC on the host machine.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .