20

I found out that my CentOS 5 server on EC2 was rebooted, without my command to do so. I did not even log in on the day it was rebooted. When I look at 'last' in linux, it says:

jeroen   pts/0        128.97.......    Thu Apr  7 15:02 - 16:28  (01:25)    
reboot   system boot  2.6.18-xenU-ec2- Wed Apr  6 15:48         (1+05:27)   
jeroen   pts/2        128.97.......    Tue Apr  5 19:31 - 23:17  (03:45)

So it looks like nobody logged on that day. Any suggestions to what could have happened? Does EC2 reboot instances under any circumstances? Or might it be hacked?

0

2 Answers 2

31

This happens occasionally when Amazon is looking to decommission the physical server your instance is running on. They might be killing the server to replace a failing hardware component or the server might have simply reached its end of life. Either way from what I understand, their process works something like this:

  • The physical server gets marked as decommissioned so no new instances get launched on it.
  • If they can (e.g. they're not dealing with a critical hardware failure), Amazon will wait for some period of time to see if the instances running on the server shutdown or reboot on their own (rebooting an EC2 instance usually results in it getting launched on a different physical server).
  • After that period completes, Amazon will force the remaining instances to reboot moving them to other physical servers.

As a general rule of thumb, due to this and other quirks in the environment, you should treat any individual server running on EC2 as Ephemeral. Any server may reboot or flat out disappear at any time. Accordingly automating deployment with something like chef or puppet, having solid monitoring, and designing your application to work around failure is critical.

4
  • That's terrible - I get the push to create ephemeral instances but if they are going to retire a host they should xen motion the guests to another host
    – user209462
    Feb 15, 2014 at 3:22
  • So, no vMotion (or its equivalent)? D'oh.
    – Massimo
    Feb 15, 2014 at 3:28
  • Your automation and monitoring framework should be checking for that. That's also the reason why your applications should be written to use Elastic Load Balancers.
    – Magellan
    Feb 15, 2014 at 3:55
  • Also, you guys are commenting on a 4 year old question. You aren't going to get an answer from the OP.
    – Magellan
    Feb 15, 2014 at 7:58
3

Maybe an EC2 problem. See Reboot?

I had the same problem and I found in AWS Service Health Dashboard that there had been a problem in EC2.

Conclusion : Reboots should be expected, but no more frequently than you'd expect them with commodity hardware.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .