After a routine reboot of a Linux ec2 instance with 1 small root volume and 1 small attached volume I was not able to ssh into the instance. It is not clear why rebooting the instance caused it to be inaccessible from ssh. The instance showed as running in the AWS console but ssh, http,etc was not responsive. I tried to create an AMI from this running instance. However, the AMI was never created. Instead I just saw "pending" in the AMI section of the AWS console for hours. Eventually I de-registered the AMI. Next I tried to stop the ec2 instance. However I am not able to stop the instance - it has been stuck in the stopping state for hours. I also tried force stopping the instance with no success. I then tried to detach the volumes but they constantly report "detaching" Does anyone have any suggestions about how to handle this? It seems that Amazon does not offer any kind of email or phone support unless you are a premium member. Thanks very much for your help.
6 Answers
If you don't have premium support, then "stuck" instances (where you can't stop/terminate them) and "stuck" volumes (where you can't detach/delete them) can be reported to Amazon on the EC2 forum:
Amazon AWS EC2 Forum
https://forums.aws.amazon.com/forum.jspa?forumID=30
Make sure you list the specific instance/volume ids involved.
Nobody but Amazon can really help in these situations.
Fortunately, you should not be charged for instance hours once it enters the "stopping" or "terminating" state.
Your original problem about not being able to connect to the instance through ssh could also be posted on the EC2 forum for help from the community, but it is a common problem and has many potential causes. I've written an article to help start diagnose this and to point out pieces of information you should include in your forum post:
Solving: "I can't connect to my server on Amazon EC2"
http://alestic.com/2009/08/ec2-connectivity
I've also written an article describing a method that can be used to diagnose EBS boot instances by looking at the log files on the disk even if you can't connect to the instance:
Fixing Files on the Root EBS Volume of an EC2 Instance
http://alestic.com/2011/02/ec2-fix-ebs-root
However, you won't be able to use this approach until Amazon helps you detach your EBS volume.
-
1Thank you for your detailed suggestions and your articles - they are very helpful. Sep 13, 2011 at 15:06
I know this post is old, but you can also click stop again, and the dashboard will ask you if you want to do a forced stop. Sometimes, I know this from experience, you have to do two or three forced stops to get it to work.
Credit to Robin Scott.
I ran into a related situation after I created an Elastic Beanstalk environment within an ec2 instance. To terminate my ec2 instance, I had to terminate my eb environment first, which I did by visiting:
https://console.aws.amazon.com/elasticbeanstalk/?region=us-west-2
Where the value supplied to region=
is the region that contains the environment to be shut down. On that page, one can easily deactivate multiple environments swiftly:
Force detach the system volume. Then you will be able to stop instance. Re-attach volume and start instance. Viola!
I recently faced a similar same issue. While trying to start an instance, it got stuck in pending state. After reaching out to AWS support, they provided a workaround which worked well. This might also be applicable in other similar situations where an instance is stuck in a state which does not allow further actions through AWS Console.
You can use the AWS CLI to force stop the instance, instead of AWS console. Although an instance stack in Pending state cannot be managed through AWS console, using the AWS CLI allows you to stop it and start it again. The following command should allow you to stop the instance and force it move to the "Stopped" state:
aws ec2 stop-instances --instance-ids <You instance Id>
You can find more information about how to install and use AWS CLI here: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
on reboot internal IPs and public DNS change, so if you dont have elastic IP associated with your system then this might be the reason why you cant SSH in to the system.
-
Thank you for your suggestion. There is an elastic IP associated with the instance and it was associated to the instance after the reboot. Sep 13, 2011 at 15:10
-
3Rebooting an EC2 instance does not change internal IP addresses, does not change public IP addresses, and does not disassociate any Elastic IP address from the instance. Stopping and starting an instance will do all these things, but rebooting will not. Sep 13, 2011 at 20:22
-
1"If you reboot an instance, the IP address is maintained." aws.amazon.com/articles/… "A reboot or restart of an instance has no impact on the mapping of an Elastic IP to it." aws.amazon.com/articles/1346 Sep 13, 2011 at 20:37