The routing table by default will only route traffic to eth0
.
Even though ubuntu detects the other ENI, you still have to route traffic to it.
You'll have to do some advanced routing:
1) Enabling access to 2nd ENI immediately and temporarily.
source: http://www.rjsystems.nl/en/2100-adv-routing.php
# this will show your route table, i'll assume you have eth0 and eth1
# and your default is for eth0 to point to the gateway
# for this example lets assume the following:
# eth0 = 192.168.100.5
# eth1 = 192.168.100.10
# gateway = 192.168.100.1
ip route show ;
# first step is to create a routing table for your new device
cat /etc/iproute2/rt_tables ;
echo 2 eth1_rt >> /etc/iproute2/rt_tables ;
# next add the eth1_rt route table, so by default it will also point to the gateway
ip route add default via 192.168.100.1 dev eth1 table eth1_rt ;
# next take a look at your ip rules
# i'll assume the defaults here, and things flows to default with priority 32767
ip rule;
# let's add a rule, if we see traffic from eth1's IP address,
# use its new routing table we setup, and give it higher priority than default
ip rule add from 192.168.100.10 lookup eth1_rt prio 1000 ;
# done! now check your traffic from both IPs, they should both work.
2) Enabling access to 2nd ENI on reboot but persistently.
source: http://blog.bluemalkin.net/multiple-ips-and-enis-on-ec2-in-a-vpc/
Additionally, if you want this change to persist, you can make all these changes in the interface file and just restart the network service or reboot for it to take effect.
# NOTE: add the eth1_rt routing table to /etc/iproute2/rt_tables as show in previous section
# original config to make dchp, I add mine to /etc/network/interfaces.d/eth1.cfg
auto eth1
iface eth1 inet dchp
# your extra rules for eth1
up ip route add default via 192.168.100.1 dev eth1 table eth1_rt
up ip rule add from 192.168.100.10 lookup eth1_rt prio 1000
For this to take full effect, reboot the system.
NOTE: I tried /etc/init.d/networking restart;
but it didn't pick up the route/rule changes, not sure why, so I had reboot. In the event you want to make it immediate and persistent, do both methods.