Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a multihoned Ubuntu server with a set of interfaces that includes:

eth2: 10.10.0.131/24
eth3: 10.20.0.2/24

The default interface is eth2, with a gateway of 10.10.0.1. Here's what the routing table looks like:

root@c220-1:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.0.1       0.0.0.0         UG        0 0          0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.20.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth3
10.30.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.40.0.0       0.0.0.0         255.255.0.0     U         0 0          0 eth1

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

If I ping 10.10.0.131 from 192.168.3.5, the packets reach the machine, but it doesn't send any replies:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 0, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 1, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 2, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 3, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 4, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 5, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 6, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 7, length 64

If I ping from the router (10.20.0.1) on the same network, the server replies properly:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80

Note that, as per the answer in this similar question, I have rp_filter off on all of the interfaces, but it doesn't resolve the issue:

$ for i in eth0 eth1 eth2 eth3 all default
> do
> cat /proc/sys/net/ipv4/conf/$i/rp_filter
> done
0
0
0
0
0
0
share|improve this question
    
possible duplicate serverfault.com/q/481654/115189 –  mgorven Mar 14 '13 at 19:03

2 Answers 2

up vote 5 down vote accepted

The problem is that since the default route is through eth2, the ping responses get sent through eth2 even though the requests were received on eth3. (If you tcpdump eth2 you should see the responses being sent.) There's then probably some device which is dropping the packets because they have an invalid source IP for the network they're on. You need some source policy routing in order to get the responses to be sent out the interface they were received on.

  1. Create a new routing table (only needs to be done once):

    echo 13 eth3 >> /etc/iproute2/rt_tables
    
  2. Add a default route to this new table going out eth3:

    ip route add default via 10.20.0.1 table eth3
    
  3. Add a policy rule to use this new table for packets with source address of eth3's IP:

    ip rule add from 10.20.0.2 lookup eth3
    
share|improve this answer
    
D'oh, I didn't think to do the tcpdump on eth2, I do see the responses. The source policy routing worked. –  Lorin Hochstein Mar 14 '13 at 19:34
    
Can the table label "eth3" be arbitrary, or does it need to match the interface name? –  Lorin Hochstein Mar 14 '13 at 20:31
    
@LorinHochstein The name is arbitrary. –  mgorven Mar 14 '13 at 21:02

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

It sounds like you are missing a route for 192.168.3.5/24 from the 10.30.0/24 subnet. You should add a network diagram and traceroutes for each network from each device.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.