Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I found out that my CentOS 5 server on EC2 was rebooted, without my command to do so. I did not even log in on the day it was rebooted. When I look at 'last' in linux, it says:

jeroen   pts/0        128.97.......    Thu Apr  7 15:02 - 16:28  (01:25)    
reboot   system boot  2.6.18-xenU-ec2- Wed Apr  6 15:48         (1+05:27)   
jeroen   pts/2        128.97.......    Tue Apr  5 19:31 - 23:17  (03:45)

So it looks like nobody logged on that day. Any suggestions to what could have happened? Does EC2 reboot instances under any circumstances? Or might it be hacked?

share|improve this question

2 Answers 2

up vote 10 down vote accepted

This happens occasionally when Amazon is looking to decommission the physical server your instance is running on. They might be killing the server to replace a failing hardware component or the server might have simply reached its end of life. Either way from what I understand, their process works something like this:

  • The physical server gets marked as decommissioned so no new instances get launched on it.
  • If they can (e.g. they're not dealing with a critical hardware failure), Amazon will wait for some period of time to see if the instances running on the server shutdown or reboot on their own (rebooting an EC2 instance usually results in it getting launched on a different physical server).
  • After that period completes, Amazon will force the remaining instances to reboot moving them to other physical servers.

As a general rule of thumb, due to this and other quirks in the environment, you should treat any individual server running on EC2 as Ephemeral. Any server may reboot or flat out disappear at any time. Accordingly automating deployment with something like chef or puppet, having solid monitoring, and designing your application to work around failure is critical.

share|improve this answer
    
That's terrible - I get the push to create ephemeral instances but if they are going to retire a host they should xen motion the guests to another host –  Logic Feb 15 at 3:22
    
So, no vMotion (or its equivalent)? D'oh. –  Massimo Feb 15 at 3:28
    
Your automation and monitoring framework should be checking for that. That's also the reason why your applications should be written to use Elastic Load Balancers. –  Magellan Feb 15 at 3:55
    
Also, you guys are commenting on a 4 year old question. You aren't going to get an answer from the OP. –  Magellan Feb 15 at 7:58

Maybe an EC2 problem. See Reboot?

I had the same problem and I found in AWS Service Health Dashboard that there had been a problem in EC2.

Conclusion : Reboots should be expected, but no more frequently than you'd expect them with commodity hardware.

share|improve this answer
    
Oups... link is forums.aws.amazon.com/thread.jspa?messageID=52417 –  fafanoulele Apr 11 '11 at 13:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.