Running an EC2 server on Ubuntu with 2 simultaneous network interfaces

Question

I have found similar questions relating to this issue but none matching my situation and none of the advice has helped me get this running.

I have a 64bit Ubuntu server from the default AMI running in EC2. This has 2 network instances, each with 2 local IPS and linked EIP's.

The local IP addresses on each instance are intentionally on separate subnets to help overcome the complications of same network IPs on different instances.

The 2 IP's on the first instance work fine, no problem with those.

I have been using the following code in the startup of all my servers for a while

#!/bin/bash
MAC_ADDR=$(ifconfig eth0 | sed -n 's/.*HWaddr \([a-f0-9:]*\).*/\1/p')
IP=($(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDR/local-ipv4s))
for ip in ${IP[@]:1}; do
    echo "Adding IP eth0: $ip"
    ip addr add dev eth0 $ip/24
done

I have added a second part to this script to try and startup the second interfaces IPs

MAC_ADDR=$(ifconfig eth1 | sed -n 's/.*HWaddr \([a-f0-9:]*\).*/\1/p')
IP=($(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDR/local-ipv4s))
for ip in ${IP[@]:1}; do
    echo "Adding IP eth1: $ip"
    ip addr add dev eth1 $ip/24
done

I have modified my /etc/network/interfaces file to start the second interface

# The primary network interface
auto eth0
iface eth0 inet dhcp


auto eth1
iface eth1 inet static
address 10.10.21.22
netmask 255.255.255.0

ifconfig shows both interfaces operating

ifconfig
eth0      Link encap:Ethernet  HWaddr 06:b1:b9:f8:32:af  
          inet addr:10.10.2.20  Bcast:10.10.2.255  Mask:255.255.255.0
          inet6 addr: fe80::4b1:b9ff:fef8:32af/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:114125 errors:0 dropped:0 overruns:0 frame:0
          TX packets:51731 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:106211526 (106.2 MB)  TX bytes:50413300 (50.4 MB)
          Interrupt:26 

eth1      Link encap:Ethernet  HWaddr 06:b1:b9:f5:38:3a  
          inet addr:10.10.21.22  Bcast:10.10.21.255  Mask:255.255.255.0
          inet6 addr: fe80::4b1:b9ff:fef5:383a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11522 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:366172 (366.1 KB)  TX bytes:484140 (484.1 KB)
          Interrupt:27 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:14772 (14.7 KB)  TX bytes:14772 (14.7 KB)

My routing rules seem to be setup correctly for the 2 subnets and 2 interfaces

    Destination     Gateway     Netmask     Interface   
    Default Route   10.10.2.1   
eth0
    1.0.0.0 None    255.255.255.0   eth0
    1.0.0.0 None    255.255.255.0   eth1
    10.10.2.0   None    255.255.255.0   eth0
    10.10.21.0  None    255.255.255.0   eth1

I can ping the local IPs that have issues (10.10.21.22 and 10.10.21.23) from the same server and get a success. I can ping the public IPs for the primary interface from other servers. I cannot ping the IPs on the second interface from other servers at all.

Firewalls have been disabled while testing this.

Any guidance on what may have been missed or wrong?

Edit -- additional info

Main server is Ubuntu 13.04 64 bit from the standard AMI (micro size).

eth0 has 2 ips working, 10.10.2.20 and 10.10.2.21

eth1 has 2 ips not working, 10.10.21.22 and 10.10.21.23

All local IP's also have an external EIP.

Pinging all of these from the same machine works. Pinging the first interface from other machines works. Pinging the second interface from other machines fails.

VPC Route Table says 10.10.0.0/16 is local and 0.0.0.0/0 is igw. Looks fine. Hadn't thought to look here, had only checked the route table inside linux - is there anything special needed in EC2/VPC to get 2 interfaces working? I had assumed that EC2 knew all the IP/instance mapping and would direct it accordingly....

Just tried adding an extra route at the vpc level and got "cannot create a more specific route for 10.10.21.0/24 than local route 10.10.0.0/16 in route table rtb-022ab66b"

New edit - more VPS subnet settings

I mentioned the route table in the previous update but didn't specify the subnets inside VPC

VPC Route Table is 10.10.0.0/16 local, which should encompass both the 10.10.2.0 and 10.10.21.0 subnets

There are 2 separate subnets setup inside the VPC - one interface on each inside EC2. Both subnets are shown inside associations on the VPC route table.

10.10.2.0/24

10.10.21.0/24

I cannot setup more specific routing rules in the EC2 routing table due to the error mentioned at the end of the edit above. The only option is 10.10.0.0/16 and then I cant define 2 and 21 separately, at least not at the PPC level.

Answer 1

I have ported ec2-net-utils from Amazon Linux to Ubuntu. What these scripts do is to configure automatically new interfaces including additional IPs and fix asymmetric routing.

Have a look here:

• https://github.com/ademaria/ubuntu-ec2net
• https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1153626

Answer 2

Thanks to Amazon support for this, but posting here in case it helps anyone else.

In your instance as root type the following commands:

ip route add default via 10.10.2.1 dev eth0 tab 1
ip route add default via 10.10.21.1 dev eth1 tab 2
ip rule add from 10.10.2.0/24 tab 1
ip rule add from 10.10.21.0/24 tab 2

NOTE: To persist through a reboot on an ubuntu instance in EC2; edit this file: /etc/rc.local Within that file, your lines should look just like this:

route add -net XXX.XXX.XXX.XXX netmask XXX.XXX.XXX.XXX gw XXX.XXX.XXX.XXX