3

I am not able to connect to my m1.small instance after rebooting it.

I have already associated the public IP with this instance. Upon checking the system log, this seems to be the issue:

cloud-init-nonethttp://11.84: waiting 10 seconds for network device
cloud-init-nonethttp://21.85: waiting 120 seconds for network device
cloud-init-nonethttp://141.85: gave up waiting for a network device.
Cloud-init v. 0.7.3 running 'init' at Sun, 18 May 2014 07:02:55 +0000. Up 142.54 seconds.
ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0 | False | . | . | 02:43:xx:xx:xx:xx |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

A bunch of these follow the above message:

2014-05-18 07:02:56,178 - url_helper.pyWARNING: Calling http://169.254.169.254/2009-04-04/meta-data/instance-id failed 0/120s: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : Errno 101] Network is unreachable)

This is obviously related to the network interface not being working correctly. I have tried this so far:

  1. Relaunch a new instance from the custom AMI (created from EBS) of the failing instance. The same error shows up in the logs.

  2. Attach a new network interface to the EC2 instance. The error still persists. eth1 shows up in the list but the "up" column is False.

6
  • Possibly time to log a support request with AWS – Drew Khoury May 18 '14 at 15:15
  • I couldn't find any way to contact AWS technical support directly since I have not subscribed to any 1-to-1 support plan. Isn't it a bit odd that I have to rely on their public forum for something like this (my instance being down due to failure of their software)? – kapeels May 18 '14 at 17:16
  • Try their forum they're very helpful. – Drew Khoury May 18 '14 at 21:06
  • Have you tried detaching the EIP, rebooting and seeing if the instance passes both status checks? – Scott Moore May 19 '14 at 17:01
  • Yes, I did that. The checks still fails. They are not even replying on the support forums. This is the worst customer support I have experienced in the past few years. – kapeels May 21 '14 at 11:34
2

It's due to someone made a networking change that caused the instance to be inaccessible. A common network settings issue is when someone sets a static IP address, which Amazon EC2 ignores per AWS policy.

  1. Stop the instance
  2. Detach the volume from the non-accessible instance
  3. Attach the volume to a running instance
  4. ssh running instance
  5. mkdir bad (create a mounting directory)
  6. Mount the volume into bad/
  7. cd /bad/etc/sysconfig/network-scripts (/bad is the mount directory)
  8. vi ifcfg-eth0

    DEVICE=eth0 
    BOOTPROTO=dhcp 
    ONBOOT=yes 
    TYPE=Ethernet 
    USERCTL=yes 
    PEERDNS=yes 
    IPV6INIT=no 
    
  9. Detach the volume
  10. Attach the volume to the non-accessible instance
  11. Start the non-accessible instance

Find it working.

0

@kapeels,

I just went trough the same problem, on my case I'm using an internal DNS server set on the VPC DHCP Options set, if the DNS server set there is down all EC2 servers boots are going to have the same problem.

From what I can see in the logs, the server makes few API calls using a hostname, this hostname is translated using the DNS from DHCP options set, but because my server was down it could get the IP for the hostname and returns the 169.* IP.

When I changed my VPC DHCP options set to AmazonProvidedDNS again, it worked.

So basically AWS totally ignores your security groups to make their API calls but relies on your DNS server to solve the hostname. :)

1
  • 1
    FYI, AWS us-east-1 had DNS issues this evening. – ceejayoz Oct 3 '14 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.