Unable to connect to EC2 instance after "reboot"

Question

I am not able to connect to my m1.small instance after rebooting it.

I have already associated the public IP with this instance. Upon checking the system log, this seems to be the issue:

cloud-init-nonethttp://11.84: waiting 10 seconds for network device
cloud-init-nonethttp://21.85: waiting 120 seconds for network device
cloud-init-nonethttp://141.85: gave up waiting for a network device.
Cloud-init v. 0.7.3 running 'init' at Sun, 18 May 2014 07:02:55 +0000. Up 142.54 seconds.
ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0 | False | . | . | 02:43:xx:xx:xx:xx |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

A bunch of these follow the above message:

2014-05-18 07:02:56,178 - url_helper.pyWARNING: Calling http://169.254.169.254/2009-04-04/meta-data/instance-id failed 0/120s: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : Errno 101] Network is unreachable)

This is obviously related to the network interface not being working correctly. I have tried this so far:

1. Relaunch a new instance from the custom AMI (created from EBS) of the failing instance. The same error shows up in the logs.

2. Attach a new network interface to the EC2 instance. The error still persists. eth1 shows up in the list but the "up" column is False.

Answer 1

It's due to someone made a networking change that caused the instance to be inaccessible. A common network settings issue is when someone sets a static IP address, which Amazon EC2 ignores per AWS policy.

1. Stop the instance
2. Detach the volume from the non-accessible instance
3. Attach the volume to a running instance
4. ssh running instance
5. mkdir bad (create a mounting directory)
6. Mount the volume into bad/
7. cd /bad/etc/sysconfig/network-scripts (/bad is the mount directory)

8. vi ifcfg-eth0

   DEVICE=eth0 
   BOOTPROTO=dhcp 
   ONBOOT=yes 
   TYPE=Ethernet 
   USERCTL=yes 
   PEERDNS=yes 
   IPV6INIT=no 
   

9. Detach the volume
10. Attach the volume to the non-accessible instance
11. Start the non-accessible instance

Find it working.

Answer 2

@kapeels,

I just went trough the same problem, on my case I'm using an internal DNS server set on the VPC DHCP Options set, if the DNS server set there is down all EC2 servers boots are going to have the same problem.

From what I can see in the logs, the server makes few API calls using a hostname, this hostname is translated using the DNS from DHCP options set, but because my server was down it could get the IP for the hostname and returns the 169.* IP.

When I changed my VPC DHCP options set to AmazonProvidedDNS again, it worked.

So basically AWS totally ignores your security groups to make their API calls but relies on your DNS server to solve the hostname. :)