Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Linux (Ubuntu 12.04) machine (c220-1) that has multiple interfaces. The two of interest here are:

  • eth2: 10.10.0.131/24
  • eth3: 10.20.0.2/24

The default route is set to 10.20.0.2. However, I'd like to be able to ssh into this machine by connecting to either 10.20.0.2 or 10.10.0.131, from a separate network (192.168.3.0/24)

To be able to ssh into 10.10.0.131 and avoid the symmetric routing problem, I used source routing policy:

root@c220-1:~# ip route list table eth2
default via 10.10.0.1 dev eth2

I added this rule by adding the following line to /etc/iproute2/rt_tables

12 eth2

And then I did

root@c220-1:~# ip route add default via 10.10.0.1 table eth2
root@c220-1:~# ip rule add from 10.10.0.131 lookup eth2

This works, I can ssh to either 10.10.0.131 or 10.20.0.2. But I can no longer connect to 10.10.0.131 from other machines on the 10.10.0.0/24 network. I can arping in both directions, and I can ping from c220-1 (10.10.0.131/24) to c220-2 (10.10.0.132/24). But I can't ping from c220-2 to c220-1.

If I do a tcpdump, I can confirm that c220-1 is receiving the ICMP requests and sending replies:

root@c220-1:~# tcpdump -i eth2 icmp -vvv -n -e
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
23:15:11.455818 d4:8c:b5:4d:a5:3a > 60:73:5c:68:b4:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.0.132 > 10.10.0.131: ICMP echo request, id 8805, seq 1, length 64
23:15:11.455877 60:73:5c:68:b4:b6 > 30:f7:0d:bc:bb:52, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16273, offset 0, flags [none], proto ICMP (1), length 84)
    10.10.0.131 > 10.10.0.132: ICMP echo reply, id 8805, seq 1, length 64
23:15:12.463394 d4:8c:b5:4d:a5:3a > 60:73:5c:68:b4:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.0.132 > 10.10.0.131: ICMP echo request, id 8805, seq 2, length 64
23:15:12.463451 60:73:5c:68:b4:b6 > 30:f7:0d:bc:bb:52, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16274, offset 0, flags [none], proto ICMP (1), length 84)
    10.10.0.131 > 10.10.0.132: ICMP echo reply, id 8805, seq 2, length 64

You can see that the requests are coming from d4:8c:b5:4d:a5:3a, but the replies are going to 30:f7:0d:bc:bb:52.

As expected, the ICMP replies don't show up on c220-2:

root@c220-2:~#  tcpdump -i eth2 icmp -vvv -n -e
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
23:16:19.944225 d4:8c:b5:4d:a5:3a > 60:73:5c:68:b4:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.0.132 > 10.10.0.131: ICMP echo request, id 9380, seq 1, length 64
23:16:20.952497 d4:8c:b5:4d:a5:3a > 60:73:5c:68:b4:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.0.132 > 10.10.0.131: ICMP echo request, id 9380, seq 2, length 64
23:16:21.960458 d4:8c:b5:4d:a5:3a > 60:73:5c:68:b4:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.0.132 > 10.10.0.131: ICMP echo request, id 9380, seq 3, length 64

The destination MAC address for the replies matches the MAC address of the gateway, 10.10.0.1:

root@c220-2:~# arping -i eth2 10.10.0.1
ARPING 10.10.0.1
60 bytes from 30:f7:0d:bc:bb:52 (10.10.0.1): index=0 time=214.100 usec

If I remove the source policy routing rules entirely, then I am able to connect from c220-2, but I can't connect to 10.10.0.131 from 192.168.3.1/24.

How do I specify the rules so that the source policy routing doesn't send packets to 10.10.0.1 when they should go to another machine on the local network?

share|improve this question
    
can you show the packet leaving c220-1 to c220-2 that you see in the tcpdump? –  becomingwisest Apr 22 '13 at 3:12
    
I added tcpdump output from c220-1 and c220-2 when pinging across. That was a helpful suggestion, I can see now that the destination MAC address is for 10.10.0.1 instead of 10.10.0.132, which strongly suggests that it's the source policy routing issue. I just don't know what the correct routing rule to add is to fix this. –  Lorin Hochstein Apr 22 '13 at 3:17
    
Your routing table (route -n) is fine. The packet should hit the longest match and go out via eth2. But you say you have source routing. If it is present that would take precedence. Can you show your source routing settings? –  Benny Apr 22 '13 at 16:17
    
OK, now I saw that. I don't see why you need to use source route. Packets from Host2 will naturally go out through eth2 since they are on the same layer 2 domain. Your requirement should work without source route. Have you tried removing the source route and testing both ping and your original requirement? –  Benny Apr 22 '13 at 16:26
    
I have source routing enabled because I want to be able to ssh to c220-1 at 10.10.0.131 from a different network (192.168.3.0/24), but the default gateway on c220-1 is 10.20.0.1, so I get asymmetric routing problems unless I enable source routing. –  Lorin Hochstein Apr 22 '13 at 19:09

2 Answers 2

up vote 0 down vote accepted

I have just run into the same problem and have found the issue to be in the order of operation in which the route tables are applied.

if you issue the following prior to applying the policy routes:

[root@c220-1 ~]# ip rule show
0:      from all lookup 255
32766:  from all lookup main
32767:  from all lookup default

And then after:

[root@c220-1 ~]# ip rule show
0:      from all lookup 255
32765:  from 10.10.0.131 lookup eth2
32766:  from all lookup main
32767:  from all lookup default

IMHO I would expect any new routing tables to be applied somewhere between the 'main' routing table, which includes all the layer-2 forwarding statements, and the 'default' routing table.

I can see 2 options in resolving the issue

  • Reordering the routing rules so the 'main' table is applied before any custom route tables, or
  • Add layer-2 forwarding statement into the named route table

We went with the 2nd approach, first manually adding routes into the named routing table and verifying connectivity works as expected:

ip route add 10.10.0.0/24 dev eth2 src 10.10.0.131 table eth2

Next, making the above changes persistent across reboots

Create a route file:

[root@c220-1 ~]# vi /etc/sysconfig/network-scripts/route.eth2

10.10.0.0/24 dev eth2 src 10.10.0.131 table eth2
default table eth2 via 10.10.0.1

Create a routing rule file:

[root@c220-1 ~]# vi /etc/sysconfig/network-scripts/rule.eth2

from 10.10.0.131 table eth2

Restart the host and verify the routing policies persist

[root@c220-1 ~]# ip rule show
[root@c220-1 ~]# ip route show table eth2

Additional info I wouldn't normally recommend a bunch of static routes however due to infrastructure limitations our scenario extended upon the above to also require an additional subnet to be routed out via the secondary interface, in this case eth2

Assuming we need to get to 10.10.1.0/24 via eth2, we made the following changes to the above configuration.

Updated route file:

[root@c220-1 ~]# vi /etc/sysconfig/network-scripts/route.eth2

10.10.0.0/24 dev eth2 src 10.10.0.131 table eth2
10.10.1.0/24 dev eth2 via 10.10.0.1 table eth2
default table eth2 via 10.10.0.1

Updated policy routing rule:

[root@c220-1 ~]# vi /etc/sysconfig/network-scripts/rule.eth2

from 10.10.0.131 table eth2
to 10.10.1.0/24 table eth2

Restart networking & verify routing policies:

[root@c220-1 ~]# service networking restart

[root@c220-1 ~]# ip rule show
0:      from all lookup 255
32764:  from all to 10.10.1.0/24 lookup eth2
32765:  from 10.10.0.131 lookup eth2
32766:  from all lookup main
32767:  from all lookup default

[root@c220-1 ~]# ip route show table eth2
10.10.0.0/24 dev eth2 src 10.10.0.131 scope link
10.10.1.0/24 via 10.10.0.1 dev eth2
default via 192.168.152.1 dev eth2

We then restart the host to verify all routing policies were persistent

share|improve this answer

I don't have any linux box to test but this is what I think you should be doing:

root@c220-1:~# ip rule add from 192.168.3.0/24 table c2202
root@c220-1:~# ip route add default via 10.10.0.1 table c2202

I changed the table name from eth2 to avoid confusing with interface name.

share|improve this answer
    
This doesn't fix the issue. If I ping 10.10.0.131 from 192.168.3.5/24, then c220-1 will send the ICMP replies to eth3 (the network with the default gateway) instead of eth2. –  Lorin Hochstein Apr 23 '13 at 21:20
    
Did you flush try to flush the route cache? –  Benny Apr 24 '13 at 5:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.