3

Would anyone know how to associate multiple Elastic IPs to a single instance via the Amazon SDK ? In Ruby, I've tried to use both the aws-sdk and fog gems, which work fine for a single address, but error out trying to assign multiple.

Via the Web UI, this would be done by adding additional private ips and then assigning the public ip to the network interface + private ip however I'm not any private ip parameters in SDK.

  • did you ever find a solution to this? – mikegreiling Apr 7 '15 at 20:48
  • Hey mike, it's been a while since I look at this but, yeah it's doable. I provided a patch to the Fog library to allow for this. edit Mainly, you have to provide multiple private ip addresses and then map the public to private ip individually. In the end, though, I'm no longer using this since it was a bit of a hassle. – jonathanserafini Apr 7 '15 at 21:11
4

Elastic IPs working

By default, instance in the VPC will only get an IP from a private subnet within the VPC, usually somewhere in the 10.0.0.0/8 subnet. This means that while those instances can access other instances within that same subnet, they won’t be able to connect to the wider internet.

One way to give every machine access is to setup a NAT machine which all machines use to route traffic through. We won’t go any deeper into this option.

The alternative is to give every machine it’s own world-routable Elastic IP address. Amazon will route traffic to the elastic ip to your internal IP. This makes this setup simpler from a network topology perspective.

Routing in the VPC works through routing tables which are set up for each subnet. You can view your routing table in the VPC tab of the AWS console. By default, requests within your subnet are routed locally, and requests outside of that range are routed through an ‘internet gateway’.

For example, let’s say that you have an instance with IP 10.0.0.1. It has an Elastic IP associated with it, let’s say 1.2.3.4. Now whenever you access other internal machines on the 10.0.0.0/8 subnet, your request will originate from your private IP and nothing will happen to the traffic.

When you access a machine outside of your subnet, the packets are routed through the internet gateway, a VPC device with a name like igw-9d7534f2. The gateway (which isn’t an actual EC2 instance, just an opaque AWS system) accepts the request and then looks up the internal IP from which the request originates, and checks if this IP is associated with any Elastic IP’s. If so, the request is rewritten to appear as if the origin is the elastic IP, and then send out over the internet. When a package returns to the gateway with an elastic IP as destination, the gateway checks if the elastic IP is associated with an internal IP, and if so rewrites the package to that IP. The package is then forwarded into the private subnet.

This is how the one-to-one NAT of elastic IP’s works. The advantage of this approach is that machines within the subnet need to have no knowledge of their elastic IP’s. This makes it easy to switch elastic IP’s of a running instance, since the only thing that needs to be updated is the mapping table on the internet gateway. The external IP can change many times; since the packages are rewritten to use internal IP’s before they arrive at the instance, the instance will never know this.

Multiple Network Interfaces

In the past, Amazon added an option to add multiple network interfaces to a VPC instance. These interfaces appear as separate ethernet cards on your machine, and will have a separate internal IP. Separate interfaces also means you need to set up your routing between these interfaces correctly: if you send out a packet over the wrong interface, the packet will simply be dropped.

These network interfaces were introduced to give you the option of connecting different subnets together: you can have one interface in each subnet, and then route trafic between them as you like. Since each subnet has its own IP range, it is easy to route traffic to the right network interface.

For each internal IP, it is possible to associate an external Elastic IP. While this is possible, you get into a bit of a tricky routing situation now. Both internal IP’s (on eth0 and eth1) have an associated external IP, and should be able to make requests to the wider internet, by having the internet gateway translate their internal IP to the Elastic IP. However, in order to do this correctly, the requests have to be send out over the correct interface. Amazon itself admits that this is not simple:

By default, Linux has a routing table which routes all traffic outside of your local subnet to a single interface. You can look this up with route:

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.0.1        0.0.0.0         UG    0      0        0 eth0
default         10.0.0.1        0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1

Here, all internet-wide traffic will go out over interface eth0. Even packets with a source ip of the eth1 interface will go out over eth0, and then will be silently discarded instead of being NAT’d to the right elastic IP. While it’s possible to do source-based routing to fix this, this isn’t trivial to set up (note that you need to run dhclient after adding another interface to a machine):

# ifconfig | grep eth\\\|inet\ 
eth0      Link encap:Ethernet  HWaddr 02:86:10:77:7f:fe  
          inet addr:10.0.0.76  Bcast:10.0.0.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 02:86:10:65:fd:31  
          inet addr:10.0.0.226  Bcast:10.0.0.255  Mask:255.255.255.0

# curl --interface 10.0.0.76 ifconfig.me
116.x.x.x
# curl --interface 10.0.0.226 ifconfig.me
<< TIMEOUT >>>

# ip rule add from 10.0.0.226 table out2
# ip route add default via 10.0.0.1 dev eth1 table 2
# ip route flush cache

# curl --interface 10.0.0.226 ifconfig.me
116.x.x.x
# curl --interface 10.0.0.76 ifconfig.me
116.x.x.x

Multiple IP Addresses

This way we can avoid some of the routing nasties from above: all packets will be sent over eth0, regardless of which IP you have. Because of the one-to-one mapping of public and private addresses, you first need to add some new private addresses to one of your instances.

Amazon’s API has been updated to support assigning secondary private IP’s, but for this example it’s easier to just go to the AWS Console. In the EC2 tab, go to Instances. Find the instance you want to update, right-click and choose “Manage Private IP Adresses”.

enter image description here

enter image description here

You’ll now see that the instance has two private IP’s in your subnet (since an interface is locked to specific subnet, all IP’s on that interface have to fall within that same subnet). Nice! However, if you now check your instance:

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:86:10:7b:e4:f5  
          inet addr:10.0.0.34  Bcast:10.0.0.255  Mask:255.255.255.0

It won’t have the new IP yet. Let’s try retrieving it over DHCP!

root@ip-10-0-0-34:/# dhclient -d eth0
Listening on LPF/eth0/02:86:10:7b:e4:f5
Sending on   LPF/eth0/02:86:10:7b:e4:f5
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPREQUEST of 10.0.0.34 on eth0 to 255.255.255.255 port 67
DHCPOFFER of 10.0.0.34 from 10.0.0.1
DHCPACK of 10.0.0.34 from 10.0.0.1

Nope, we only get our primary private ip! With DHCP you can only get a single address. In the future, perhaps Amazon will add support for multiple IP’s using some client identifier or another mechanism, but for now you’ll have to add the address manually. Hopefully Ubuntu’s cloud-init will add support for this in the future, so we don’t have to do it ourselves.

Amazon suggests creating a new virtual interface and then bringing that up. While that will work, it means you store your interface state on your hard drive, and any changes you make on AWS you will have to propagate to these files.

For testing, we can do something simpler:

MAC_ADDR=$(ifconfig eth0 | sed -n 's/.*HWaddr \([a-f0-9:]*\).*/\1/p')
IP=($(curl http://169.x.x.x/latest/meta-data/network/interfaces/macs/$MAC_ADDR/local-ipv4s))
for ip in ${IP[@]:1}; do
    echo "Adding IP: $ip"
    ip addr add dev eth0 $ip/24
done

This will check the EC2 instance meta-data for all internal IPs associated with eth0, and adds all the secondary IP’s.

You can now asociate another elastic ip to the secondary IP in the AWS Console:

enter image description here

And now you have multiple external IP’s!

root@ip-10-0-0-34:/# curl --interface 10.0.0.58 ifconfig.me
116.x.x.x
root@ip-10-0-0-34:/# curl --interface 10.0.0.34 ifconfig.me
116.x.x.x

While this works for testing, it’s not persistent across reboot. You can create an upstart or cloud-init script to do that for you. Even then, ip addresses aren’t automatically added when you add them through EC2. I’m not sure if there’s a good way to do that. Finally, this script also won’t remove any local addresses you may have removed in the meantime.

Hope this will help you.

  • 3
    If you're going to shamelessly copy-paste someone else's work, I'd highly recommend that you at the very least refer back to them :engineering.silk.co/post/31923247961/… – jonathanserafini Jun 27 '14 at 12:56
  • Also, this in no way answers the previous question which is "How can I associate multiple elastic IPs with a single network interface with the SDK" – jonathanserafini Jun 27 '14 at 12:56
  • i didn't say its my own answer,just trying to help,this worked for me before.And however that SDK parameter is confusing – Innovator Jun 27 '14 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.