Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I was trying to determine why a packet forwarded by iptables to my internal lan is returned on the wrong gateway.

Having investigated this further, I determined that, at least in the case of one of the servers, the problem results from a decision by the kernel to forward an Ethernet frame containing the packet reply in the direction of a MAC address that doesn't know how to route the packet.

So, for example, my network cards are:

eth0      Link encap:Ethernet  HWaddr 00:0E:0C:6D:51:8B inet addr:216.125.41.12  
eth1      Link encap:Ethernet  HWaddr 00:0F:1F:FA:54:BD inet addr:192.168.0.1
eth2      Link encap:Ethernet  HWaddr 00:0E:0C:6D:50:D7 inet addr:78.118.48.98

The routing tables are:

# ip route list table all
192.168.0.0/24 via 192.168.0.1 dev eth1  table 2
default via 78.118.48.97 dev eth2  table 2
192.168.20.0/24 via 192.168.0.2 dev eth1
192.168.16.0/24 via 192.168.0.2 dev eth1
192.168.17.0/24 via 192.168.0.2 dev eth1
192.168.0.0/24 dev eth1  proto kernel  scope link  src 192.168.0.1
192.168.15.0/24 via 192.168.0.2 dev eth1
78.118.48.0/24 dev eth2  proto kernel  scope link  src 78.118.48.98
216.125.41.0/24 dev eth0  proto kernel  scope link  src 216.125.41.12
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth1  scope link  metric 1003
169.254.0.0/16 dev eth2  scope link  metric 1004
default equalize
        nexthop via 216.125.41.9  dev eth0 weight 100
        nexthop via 78.118.48.97  dev eth2 weight 1
192.168.0.0/24 via 192.168.0.1 dev eth1  table 1
default via 216.125.41.9 dev eth0  table 1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
broadcast 216.125.41.0 dev eth0  table local  proto kernel  scope link  src 216.125.41.12
broadcast 192.168.0.255 dev eth1  table local  proto kernel  scope link  src 192.168.0.1
broadcast 78.118.48.255 dev eth2  table local  proto kernel  scope link  src 78.118.48.98
broadcast 216.125.41.255 dev eth0  table local  proto kernel  scope link  src 216.125.41.12
local 192.168.0.1 dev eth1  table local  proto kernel  scope host  src 192.168.0.1
broadcast 192.168.0.0 dev eth1  table local  proto kernel  scope link  src 192.168.0.1
broadcast 78.118.48.0 dev eth2  table local  proto kernel  scope link  src 78.118.48.98
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 78.118.48.98 dev eth2  table local  proto kernel  scope host  src 78.118.48.98
local 216.125.41.12 dev eth0  table local  proto kernel  scope host  src 216.125.41.12
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
unreachable ::/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:a00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:7f00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:a9fe::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:ac10::/28 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:c0a8::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:e000::/19 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 3ffe:ffff::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376 hoplimit 0
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev eth1  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev eth2  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
unreachable default dev lo  table unspec  proto kernel  metric -1  error -101 hoplimit 255
local ::1 via :: dev lo  table local  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local fe80::20e:cff:fe6d:50d7 via :: dev lo  table local  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local fe80::20e:cff:fe6d:518b via :: dev lo  table local  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local fe80::20f:1fff:fefa:54bd via :: dev lo  table local  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
ff00::/8 dev eth0  table local  metric 256  mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev eth1  table local  metric 256  mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev eth2  table local  metric 256  mtu 1500 advmss 1440 hoplimit 0
unreachable default dev lo  table unspec  proto kernel  metric -1  error -101 hoplimit 255

#ip rule
0:      from all lookup local
32761:  from 78.118.48.98 lookup 2
32762:  from 216.125.41.12 lookup 1
32766:  from all lookup main
32767:  from all lookup default

# ip route show table local
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 216.125.41.0 dev eth0  proto kernel  scope link  src 216.125.41.12
broadcast 192.168.0.255 dev eth1  proto kernel  scope link  src 192.168.0.1
broadcast 78.118.48.255 dev eth2  proto kernel  scope link  src 78.118.48.98
broadcast 216.125.41.255 dev eth0  proto kernel  scope link  src 216.125.41.12
local 192.168.0.1 dev eth1  proto kernel  scope host  src 192.168.0.1
broadcast 192.168.0.0 dev eth1  proto kernel  scope link  src 192.168.0.1
broadcast 78.118.48.0 dev eth2  proto kernel  scope link  src 78.118.48.98
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 78.118.48.98 dev eth2  proto kernel  scope host  src 78.118.48.98
local 216.125.41.12 dev eth0  proto kernel  scope host  src 216.125.41.12
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1

# ip route show table 2
192.168.0.0/24 via 192.168.0.1 dev eth1
default via 78.118.48.97 dev eth2

# ip route show table 1
192.168.0.0/24 via 192.168.0.1 dev eth1
default via 216.125.41.9 dev eth0

Packets requests from the outside arrives as expected on 00:0E:0C:6D:50:D7, but replies never reach the target apparently because they are routed in the direction of 00:0E:0C:6D:51:8B instead of 0:0E:0C:6D:50:D7.

For example:

When I try to connect from 185.20.66.66 to 78.118.48.98 and the request arrives on the interface eth2 ...

13:38:40.869461  In d4:ca:6d:59:35:63 ethertype IPv4 (0x0800), length 64: (tos 0x20, ttl 116, id 6738, offset 0, flags [DF], proto TCP (6), length 48)
    185.20.66.66.1935 > 78.118.48.98.4444: Flags [S], cksum 0x0241 (correct), seq 974833864, win 65535, options [mss 1440,nop,nop,sackOK], length 0

Then, packet is forwarded to internal host by eth1 with masquerating.

13:38:40.869507 Out 00:0f:1f:fa:54:bd ethertype IPv4 (0x0800), length 64: (tos 0x20, ttl 115, id 6738, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.1.1935 > 192.168.0.3.22: Flags [S], cksum 0x207f (correct), seq 974833864, win 65535, options [mss 1440,nop,nop,sackOK], length 0

Internal host answer to gateway through eth1

13:38:40.870459  In 00:07:e9:c7:10:41 ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.3.22 > 192.168.0.1.1935: Flags [S.], cksum 0x2688 (correct), seq 3669297228, ack 974833865, win 5840, options [mss 1460,nop,nop,sackOK], length 0

but it leaves in the direction of eth0 (00:0e:0c:6d:51:8b), instead of eth2 (00:0E:0C:6D:50:D7) and the ack never arrives to destination ...

13:38:40.870491 Out 00:0e:0c:6d:51:8b ethertype IPv4 (0x0800), length 64: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    78.118.48.98.4444 > 185.20.66.66.1935: Flags [S.], cksum 0x084a (correct), seq 3669297228, ack 974833865, win 5840, options [mss 1460,nop,nop,sackOK], length 0

This is unexpected, because the route rules are configured as follows:

#ip rule
0:      from all lookup local
32761:  from 78.118.48.98 lookup 2
32762:  from 216.125.41.12 lookup 1
32766:  from all lookup main
32767:  from all lookup default

So, my question is: why is the kernel sending the response via the wrong interface?

share|improve this question
    
Why don't you have a route for the whole subnet under each table? Also what is under ip route show table main? –  NickW Nov 7 '13 at 16:17
    
@NickW # ip route show table main 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3 169.254.0.0/16 dev eth0 scope link default via 192.168.0.1 dev eth0 –  EzioF Nov 7 '13 at 16:22
    
The weight for routing is definitely towards the eth0 interface in your equalize statement.. 100 to 1. I'd also have a closer look at your masquerading rules, if you set a particular src address, then it will always hit on that table. –  NickW Nov 7 '13 at 16:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.