2

I have a VPC 10.0.0.0/16 with an Internet gateway and two subnets 10.0.100.0/24 and 10.0.200.0/24 in the same availability zone. A single security group allows tcp/22 inbound from 0.0.0.0/0 and everything outbound. I also have two network interfaces tied to the security group, one in each subnet. Each network interface has its own Elastic IP address. There is a routing table for each subnet pointing 0.0.0.0/0 to the Internet gateway.

Here's the issue I am facing: I have an EC2 instance paired with both network interfaces, but can only SSH from the Internet through the one paired to eth0.

This is the configuration:

$ uname -a
Linux ip-10-0-100-70 5.4.0-1045-aws #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:88:59:6e:78:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.100.70/24 brd 10.0.100.255 scope global dynamic eth0
       valid_lft 1806sec preferred_lft 1806sec
    inet6 fe80::88:59ff:fe6e:78c0/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:7d:b2:d4:b5:ea brd ff:ff:ff:ff:ff:ff
    inet 10.0.200.118/24 brd 10.0.200.255 scope global dynamic eth1
       valid_lft 1807sec preferred_lft 1807sec
    inet6 fe80::7d:b2ff:fed4:b5ea/64 scope link 
       valid_lft forever preferred_lft forever

$ ip route
default via 10.0.100.1 dev eth0 proto dhcp src 10.0.100.70 metric 100 
default via 10.0.200.1 dev eth1 proto dhcp src 10.0.200.118 metric 200 
10.0.200.0/24 dev eth1 proto kernel scope link src 10.0.200.118 
10.0.200.1 dev eth1 proto dhcp scope link src 10.0.200.118 metric 200 
10.0.100.0/24 dev eth0 proto kernel scope link src 10.0.100.70 
10.0.100.1 dev eth0 proto dhcp scope link src 10.0.100.70 metric 100

$ ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

$ sudo ufw status verbose
Status: inactive

$ ss -nlput
Netid   State     Recv-Q    Send-Q             Local Address:Port       Peer Address:Port   Process   
udp     UNCONN    0         0                  127.0.0.53%lo:53              0.0.0.0:*                
udp     UNCONN    0         0              10.0.200.118%eth1:68              0.0.0.0:*                
udp     UNCONN    0         0               10.0.100.70%eth0:68              0.0.0.0:*                
tcp     LISTEN    0         4096               127.0.0.53%lo:53              0.0.0.0:*                
tcp     LISTEN    0         128                      0.0.0.0:22              0.0.0.0:*                
tcp     LISTEN    0         128                         [::]:22                 [::]:*

So it appears that the network interfaces are configured properly and that the SSH daemon is listening on all interfaces. The SSH daemon is functioning properly because I am connected via SSH already to eth0. And both interfaces seem to pass traffic to the Internet just fine:

$ ping -I eth0 -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.0.100.70 eth0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=38 time=11.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=38 time=11.6 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=38 time=11.5 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 11.470/11.515/11.567/0.031 ms

$ ping -I eth1 -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.0.200.118 eth1: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=38 time=11.7 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=38 time=11.8 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=38 time=11.7 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=38 time=11.8 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=38 time=11.8 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
rtt min/avg/max/mdev = 11.705/11.755/11.829/0.042 ms

But cannot connect:

$ ssh -i ~/.ssh/cert.pem ubuntu@3.<redacted>  # EIP for eth0
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1045-aws x86_64)
...

$ ssh -i ~/.ssh/cert.pem ubuntu@52.<redacted> # EIP for eth1
ssh: connect to host 52.<redacted> port 22: Operation timed out

I setup Netflow on the eth1 network adapter in AWS and can see the traffic:

2 840416055907 eni-07cc18b6f1b89378e <redacted> 10.0.200.118 54268 22 6 9 576 1623280724 1623280761 ACCEPT OK

So I feel like the OS is dropping the traffic. But there is no firewall configured and the SSH daemon is listening on all interfaces. I also tried tailing logs /var/log/{syslog,auth.log,kern.log} and dmesg, but nothing appeared in any of them while I attempted to connect.

I hope I'm missing something easy because I'm at a bit of a loss right now. Any help would be very appreciated!

Improve this question
3
  • You tell "There is a routing table for each subnet pointing 0.0.0.0/0 to the Internet gateway." . In your network configuration dumps there aren't any additional routing tables, nor any additional routing rules pointing to these routing tables. So can you clarify what you meant with "There is a routing table for each subnet ..."? A routing table is not a route (entry). And correct multi-homing requires tables and rules
    – A.B
    Commented Jun 10, 2021 at 11:23
  • Thanks for your reply @A.B. The routing tables are defined in AWS so that egress traffic from the EC2 instance can find the Internet gateway.
    – Micah Henning
    Commented Jun 10, 2021 at 15:52
  • 1
    As sshd doesn't have a -I option like ping to lock routing to an interface, I'm afraid you'll have routing issues on your instance. Only the first default route works, the 2nd has no effect for sshd.
    – A.B
    Commented Jun 10, 2021 at 15:54

1 Answer 1

Reset to default
1

Traffic from eth1 needs be configured to route properly:

$ sudo ip rule add from 10.0.200.118 table default
$ sudo ip route add default via 10.0.200.1 dev eth1 table default
$ sudo ip route flush cache

Since we don't want to manually have to SSH into the instance to enter these commands, I created a systemd script to execute on startup.

/home/ubuntu/dual-home.sh

#!/bin/bash

ADDR=$(ip -f inet addr show eth1 | sed -En -e 's/.*inet ([0-9.]+).*/\1/p')
GATEWAY=$(echo $ADDR | sed -En -e 's/(([0-9]+\.){3}).*/\11/p') # assumes /24 mask
sudo ip rule add from $ADDR table default
sudo ip route add default via $GATEWAY dev eth1 table default
sudo ip route flush cache

/etc/systemd/system/dual-home.service

[Unit]
Description=Configure eth1 routing
After=network.target
After=cloud-final.service

[Service]
Type=simple
ExecStart=/bin/bash /home/ubuntu/dual-home.sh

[Install]
WantedBy=cloud-init.target

Then enable the service: $ sudo systemctl enable dual-home

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .