33

I'm setting up a Debian box as a router for 4 subnets. For that I have defined 4 virtual interfaces on the NIC where the LAN is connected (eth1).

eth1      Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.1.1  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::960c:6dff:fe82:d98/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6026521 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35331299 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:673201397 (642.0 MiB)  TX bytes:177276932 (169.0 MiB)
          Interrupt:19 Base address:0x6000 

eth1:0    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.2.1  Bcast:10.1.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:19 Base address:0x6000 

eth1:1    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.3.1  Bcast:10.1.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:19 Base address:0x6000 

eth1:2    Link encap:Ethernet  HWaddr 94:0c:6d:82:0d:98  
          inet addr:10.1.4.1  Bcast:10.1.4.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:19 Base address:0x6000 

eth2      Link encap:Ethernet  HWaddr 6c:f0:49:a4:47:38  
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::6ef0:49ff:fea4:4738/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:199809345 errors:0 dropped:0 overruns:0 frame:0
          TX packets:158362936 errors:0 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:1000 
          RX bytes:3656983762 (3.4 GiB)  TX bytes:1715848473 (1.5 GiB)
          Interrupt:27 

eth3      Link encap:Ethernet  HWaddr 94:0c:6d:82:c8:72  
          inet addr:192.168.2.5  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::960c:6dff:fe82:c872/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:110814 errors:0 dropped:0 overruns:0 frame:0
          TX packets:73386 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:16044901 (15.3 MiB)  TX bytes:42125647 (40.1 MiB)
          Interrupt:20 Base address:0x2000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:22351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22351 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2625143 (2.5 MiB)  TX bytes:2625143 (2.5 MiB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:41358924 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23116350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:3065505744 (2.8 GiB)  TX bytes:1324358330 (1.2 GiB)

I have two other computers connected to this network. One has IP 10.1.1.12 (subnet mask 255.255.255.0) and the other one 10.1.2.20 (subnet mask 255.255.255.0). I want to be able to reach 10.1.1.12 from 10.1.2.20.

Since packet forwarding is enabled in the router and the policy of the FORWARD chain is ACCEPT (and there are no other rules), I understand that there should be no problem to ping from 10.1.2.20 to 10.1.1.12 going through the router.

However, this is what I get:

$ ping -c15 10.1.1.12
PING 10.1.1.12 (10.1.1.12): 56 data bytes
Request timeout for icmp_seq 0
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 81d4   0 0000  3f  01 e2b3 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 1
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 899b   0 0000  3f  01 daec 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 2
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 78fe   0 0000  3f  01 eb89 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 3
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 14b8   0 0000  3f  01 4fd0 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 4
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 8ef7   0 0000  3f  01 d590 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 5
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 ec9d   0 0000  3f  01 77ea 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 6
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 70e6   0 0000  3f  01 f3a1 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 7
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 b0d2   0 0000  3f  01 b3b5 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 8
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 f8b4   0 0000  3f  01 6bd3 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 9
Request timeout for icmp_seq 10
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 1c95   0 0000  3f  01 47f3 10.1.2.20  10.1.1.12 

Request timeout for icmp_seq 11
Request timeout for icmp_seq 12
Request timeout for icmp_seq 13
92 bytes from router2.mydomain.com (10.1.2.1): Redirect Host(New addr: 10.1.1.12)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 62bc   0 0000  3f  01 01cc 10.1.2.20  10.1.1.12 

Why does this happen?

From what I've read the Redirect Host response has something to do with the fact that the two hosts are in the same network and there being a shorter route (or so I understood). They are in fact in the same physical network, but why would there be a better route if they are not on the same subnet (they can't see each other)?

What am I missing?

Some extra info you might want to see:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
127.0.0.1       0.0.0.0         255.255.255.255 UH    0      0        0 lo
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth3
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
192.168.1.0     0.0.0.0         255.255.255.0   U     1      0        0 eth2
10.1.4.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.1.3.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth2
0.0.0.0         192.168.2.1     0.0.0.0         UG    100    0        0 eth3

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination   

# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  -- !10.0.0.0/8           10.0.0.0/8          
MASQUERADE  all  --  10.0.0.0/8          !10.0.0.0/8          

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 
0

5 Answers 5

24

At first blush, it looks like Debian is stretching the boundaries for sending an ICMP redirect; quoting RFC 792 (Internet Protocol).

  The gateway sends a redirect message to a host in the following situation.
  A gateway, G1, receives an internet datagram from a host on a network
  to which the gateway is attached.  The gateway, G1, checks its routing
  table and obtains the address of the next gateway, G2, on the route to
  the datagram's internet destination network, X.  If G2 and the host 
  identified by the internet source address of the datagram are on the same
  network, a redirect message is sent to the host.  The redirect message 
  advises the host to send its traffic for network X directly to gateway
  G2 as this is a shorter path to the destination.  The gateway forwards
  the original datagram's data to its internet destination.

In this case, G1 is 10.1.2.1 (eth1:0 above), X is 10.1.1.0/24 and G2 is 10.1.1.12, and the source is 10.1.2.20 (i.e. G2 and the host identified by the internet source address of the datagram are **NOT** on the same network). Maybe this has been historically interpreted differently in the case of interface aliases (or secondary addresses) on the same interface, but strictly speaking I'm not sure we should see Debian send that redirect.

Depending on your requirements, you might be able to solve this by making the subnet for eth1 something like 10.1.0.0/22 (host addresses from 10.1.0.1 - 10.1.3.254) instead of using interface aliases for individual /24 blocks (eth1, eth1:0, eth1:1, eth1:2); if you did this, you'll need to change the netmask of all hosts attached and you wouldn't be able to use 10.1.4.x unless you expanded to a /21.

EDIT

We're venturing a bit outside the scope of the original question, but I'll help work through the design/security issues mentioned in your comment.

If you want to isolate users in your office from each other, let's step back for a second and look at some security issues with what you have now:

You currently have four subnets in one ethernet broadcast domain. All users in one broadcast domain doesn't meet the security requirements you articulated in the comments (all machines will see broadcasts from other machines and could spontaneously send traffic to each other at Layer2, regardless of their default gateway being eth1, eth1:0, eth1:1 or eth1:2). There is nothing your Debian firewall can do to change this (or maybe I should say there is nothing your Debian firewall should do to change this :-).

  • Acquire a managed ethernet switch which supports vlans and dot1q tagging
  • Plug all your users into the ethernet switch
  • Assign users into Vlans (in linux and on the ethernet switch) based on security policy stated in the comments. A properly-configured Vlan will go a long way to fixing the issues mentioned above.
  • With respect to multiple security domains accessing 10.1.1.12, you have a couple of options:
    • Option 1: Given the requirement for all users to access services on 10.1.1.12, you could put all users in one IP subnet and implement security policies with Private Vlans (RFC 5517), assuming your ethernet switch supports this. This option will not require iptables rules to limit intra-office traffic from crossing security boundaries (that is accomplished with private Vlans).
    • Option 2: You could put users into different subnets (corresponding to Vlans) and implement iptables rules to deploy your security policies
  • After you have secured your network at the Vlan level, set up source-based routing policies to send different users out your multiple uplinks.

FYI, if you have a router or Layer3 ethernet switch that supports VRFs, some of this gets even easier; IIRC, you have a Cisco IOS machine onsite. Depending on the model and software image you already have, that Cisco could do a fantastic job isolating your users from each other and implement source-based routing policies.

8
  • Basically what I need is to have 4 subnets for different areas of an office. Some subnets will go out to the internet using one ISP and others will use a different one. Machines from different subnets shouldn't be able to see or connect with each other. EXCEPT for host 10.1.1.12 which offers some services which should be available for all. Currently I haven't setup the appropriate FORWARD rules for this. However, since all forwards are accepted I thought I should be able to ping from 10.1.2.20 to 10.1.1.12.
    – El Barto
    Jun 25, 2012 at 15:37
  • Hmm... ok, thanks Mike. I'll look into VLANs more deeply. I had thought about it before starting all this, and thought I wouldn't need it. The switches we have do support VLANs, though they are unmanaged switches so, if I'm not wrong, I guess I will have to do the tagging on the Debian router, right? The isolation of subnets it's actually not a critical issue in this office, but it's something I think it would be good to have if it doesn't require too much extra work. I'll look into it and see what I can do :)
    – El Barto
    Jun 25, 2012 at 18:28
  • @ElBarto, if your switches don't support Vlan tagging (and that's unlikely if they are unmanaged), then only tagging on Debian wouldn't help. If intra-office subnet isolation isn't a critical issue, then put everyone in two different subnets and make things easy (two subnets ensures you can policy route on the Debian). I would say that the current scheme with four Debian interface aliases offers no real subnet isolation, and it adds much more complication. Jun 25, 2012 at 18:38
  • That's right, from what I understand from the user manual the switches support "keeping the tag" but not "doing the actual tagging". Thanks for the clarification regarding Debian. The thing is that even if I keep two subnets, I'll still need machines from the subnet 10.1.2.0/24 to access to 10.1.1.12.
    – El Barto
    Jun 25, 2012 at 18:46
  • The machines in a different subnet should still be able to access 10.1.1.12. If you block linux from sending ICMP unreachables with iptables, then you will still burn CPU from it sending the ICMP messages, but at least they won't get installed in your host tables. That said, if you add another ethernet interface on the Debian (i.e. dedicate one interface per user 'class'), Debian should not send ICMP unreachables anymore; that would imply that you have two different ethernet switches: one for each user 'class'. Your cabling technicians would dislike it, but it gets the job done Jun 25, 2012 at 18:50
3

It is not really clear what you are trying to do, but I can say the following.

These subnets are connected to the same physical interface. The Linux router will return ICMP redirect message when the received packet should be forwarded over the same physical interface.

2
  • I need to handle these 4 subnets which are all connected through the same NIC. The idea is that hosts from the different subnets shouldn't be able to connect to each other, except for host 10.1.1.12 which should be available for all. I haven't defined the forward rules for this yet, so I thought any host from any of those subnets should be able to reach 10.1.1.12. Is there any way to avoid the ICMP redirect?
    – El Barto
    Jun 25, 2012 at 15:32
  • 1
    @ElBarto, one method is to add an iptables rule that drops redirects going out eth1 Jun 25, 2012 at 15:39
1

I agree with Khaled's comments and would also add to end of his phrase:

"These subnets are connected to the same physical interface. The Linux router will return ICMP redirect message when the received packet should be forwarded over the same physical interface" to the same destination subnet then redirecting the request to the next hop. That happen to me today using a Mikrotik linux router and an F5 bigip LTM device.

root@(primaryadc)(cfg-sync In Sync)(Standby)(/Common)(tmos)# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 192.168.153.20: icmp_seq=1 Redirect Host(New nexthop: 192.168.153.2)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=128 time=82.8 ms
From 192.168.153.20: icmp_seq=2 Redirect Host(New nexthop: 192.168.153.2)
64 bytes from 8.8.8.8: icmp_seq=2 ttl=128 time=123 ms
**routing table**
0.0.0.0  192.168.153.20  0.0.0.0         UG        0 0          0 external
0

Doing this physically sounds like an easier solution - put each LAN on their own switch, and install a 4-port NIC into your one box that wants to serve all of them.

1
  • And if he has 16 'boxes' he wants to serve? Possibly if he is experimenting with Kubernetes?
    – Drakes
    Nov 9, 2020 at 0:48
-1

The basic question here is "why is this happening" in respect of the ping output shown, where there are two things going on:

  1. all the ping requests are timing out
  2. the router (as 10.1.2.1) is indicating to the host, by means of an ICMP redirect, that the destination host is reachable via 10.1.1.12 (in other words, directly on the same interface)

For the avoidance of doubt, the virtual interfaces in the question (eth1:0, eth1:1, and so on) are not VLANs -- VLANs in Linux are not implied by this notation, and the question does not mention VLANs.

As to the basic issue of why the ping is not working, we don't know. There's nothing wrong with the configuration shown (provided those NAT-related rules in the iptables output are not on any of the relevant interfaces -- we can't tell from the output -- but they probably aren't because there was no other discussion of NAT in the question). The IPs and subnet masks the clients mentioned are ok, but otherwise the client details are not given. Perhaps the client routing details are wrong. This could be as simple as having missed either setting a default gateway, or missed creating a static route, depending on what routing configuration on the client side was intended.

As for the ICMP redirect, it is totally normal for this situation, and is not related to the ping failure. A router sends an ICMP redirect message in addition to relaying the original packet, not instead of it -- the ICMP redirect is only an optimization in respect of future packets. Handling of the ICMP redirect is effectively optional. As for the conditions for an ICMP redirect, "there being a shorter route" is overselling the capabilities of the router a bit; the situation is actually very simple: the router is routing a packet out the same interface it came in on, so it would necessarily save one hop if whatever node passed it on to the router instead passed it directly to the next hop. In the general case this can happen when the router is routing the packet to a different gateway in the broadcast domain that it has a routing table entry for but the previous router does not. This is the trivial case where that next hop is the destination host itself because the destination host is in the broadcast domain.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .