2

I am trying to understand why my server is failing to connect to the metadata server (and thus not able to log in with the SSH key).

Similar problems online seem to be caused by Security Group problems, that I am confident is not the problem here.

It is a single host, with IP of 192.168.160.100. The guest is given a fixed IP of 10.225.134.63. The network has a neutron DHCP server at 10.225.134.64.

On the host, I have run the following commands,

# curl http://192.168.160.100:8775
1.0
...
2008-09-01
2009-04-04

# curl http://169.254.169.254
curl: (7) Failed connect to 169.254.169.254:80; No route to host

/etc/neutron/metadata_agent.ini

nova_metadata_ip = 192.168.160.100

/etc/nova/nova.conf

metadata_host=192.168.160.100

Start up log

cloud-init[827]: Cloud-init v. 0.7.5 running 'init' at Fri, 29 Sep 2017 09:50:08 +0000. Up 6.40 seconds.
cloud-init[827]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: | Device |  Up  |    Address    |      Mask     |     Hw-Address    |
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: |  lo:   | True |   127.0.0.1   |   255.0.0.0   |         .         |
cloud-init[827]: ci-info: | eth0:  | True | 10.225.134.63 | 255.255.240.0 | fa:16:3e:0b:36:43 |
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: ++++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++++
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
cloud-init[827]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
cloud-init[827]: ci-info: |   0   |   0.0.0.0    | 10.225.128.1 |    0.0.0.0    |    eth0   |   UG  |
cloud-init[827]: ci-info: |   1   | 10.225.128.0 |   0.0.0.0    | 255.255.240.0 |    eth0   |   U   |
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+

cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [0/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [1/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [2/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
...
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [119/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from   ['http://10.225.134.64//latest/meta-data/instance-id'] after 126 seconds
2
  • 1
    192.168.160.100 and 10.225.134.63 are not on the same subnet. Why would they talk to each other? Is there a router or something between them?
    – bgtvfr
    Oct 2 '17 at 10:50
  • Yes, there is a router between them. I am able to ping from 192.168.160.100 to 10.225.134.63 and SSH starts a connection from 192.168.160.100 to 10.225.134.63 (but fails at the key authorization). Oct 2 '17 at 10:53
0

It turns out that some routes were missing in the Management Network namespace. They were added using the following commands.

# Where,
#  Management network ID = qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa
#  Management network device = tap54ff6yfd-fb
#    from `sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip addr`
sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip route add 169.254.0.0/16 dev  tap54ff6yfd-fb  proto kernel  scope link  src 169.254.169.254
sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip addr add 169.254.169.254/16 brd 169.254.255.255 scope global dev tap54ff6yfd-fb
2
  • Please can you give us more details on the solution. How did you identify the network id ( for me I have three qdhcp-*), how did you identify on which one you should run which command. Thanks.
    – hubert
    Mar 1 at 5:33
  • Sorry, it is a long time now and I really can't remember. Mar 8 at 8:07
0

Even I faced the same issue so tried to reboot the Instance and after some time in the Route Ipv4 info, am able to identify the 3rd destination with which the Instance trying to take the meta-data. This is not the exact solution to the problem but a kind of alternative.

[ 131.326590] cloud-init[548]: Cloud-init v. 18.4-0ubuntu1~16.04.2 running 'init' at Thu, 29 Nov 2018 13:01:17 +0000. Up 104.95 seconds. [ 131.373908] cloud-init[548]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++ [ 131.398661] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.419546] cloud-init[548]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 131.515524] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.567586] cloud-init[548]: ci-info: | ens3 | True | 20.0.0.9 | 255.255.255.0 | global | fa:16:3e:fb:ba:81 | [ 131.595173] cloud-init[548]: ci-info: | ens3 | True | fe80::f816:3eff:fefb:ba81/64 | . | link | fa:16:3e:fb:ba:81 | [ 131.621358] cloud-init[548]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 131.652707] cloud-init[548]: ci-info: | lo | True | ::1/128 | . | host | . | [ 131.681617] cloud-init[548]: ci-info: | sit0 | False | . | . | . | . | [ 131.716687] cloud-init[548]: ci-info: | tunl0 | False | . | . | . | . | [ 131.747621] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.828146] cloud-init[548]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++ [ 131.891154] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 131.912601] cloud-init[548]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 131.935481] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 131.981100] cloud-init[548]: ci-info: | 0 | 0.0.0.0 | 20.0.0.1 | 0.0.0.0 | ens3 | UG | [ 132.029476] cloud-init[548]: ci-info: | 1 | 20.0.0.0 | 0.0.0.0 | 255.255.255.0 | ens3 | U | [ 132.058772] cloud-init[548]: ci-info: | 2 | 169.254.169.254 | 20.0.0.1 | 255.255.255.255 | ens3 | UGH | [ 132.097319] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 132.126704] cloud-init[548]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 132.154971] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.179131] cloud-init[548]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 132.211508] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.219367] cloud-init[548]: ci-info: | 0 | fe80::/64 | :: | ens3 | U | [ 132.259315] cloud-init[548]: ci-info: | 4 | ff00::/8 | :: | ens3 | U | [ 132.283589] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.311049] cloud-init[548]: Generating public/private rsa key pair. [ 132.318065] cloud-init[548]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key. [ 132.327114] cloud-init[548]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. [ 132.364865] cloud-init[548]: The key fingerprint is: [ 132.378265] cloud-init[548]: SHA256:IhhdlSjJqLvj6PGiwTxA+SX7HbTHc4nSKBkqSIY3ye4 root@test1 [ 132.405438] cloud-init[548]: The key's randomart image is:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.