Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Linux based router with four interfaces (each with its own private subnet).

When I directly connect a device directly (i.e. no switch, just a patch cable) to one interface and another device directly to another, as below, then the router works perfectly.

   DEVICE1
192.168.8.11 ------- 192.168.8.254 
                         ROUTER
                     10.58.129.254 ------- DEVICE2
                                         10.58.129.1

When I connect the router with our switches in between, as below, then router doesn't work.

   DEVICE1
192.168.8.11 ----------- switch1
                            |
                         switch2
                            |
                         switch3
                            |
                      192.168.8.254 
                         ROUTER
                      10.58.129.254 -------- switch3
                                                |
                                             DEVICE2
                                           10.58.129.1 

All the switches are Layer 3, Switch1 (Dell PowerConnect 3548P) has a fibre connection to Switch2 (Dell PowerConnect 6224F) which is our core switch that handles routing between most VLANs. This is connected via fibre to Switch3 (Dell PowerConnect 6224).

Routing on the core switch isn't enabled either of the two VLANs (192.168.8.11 or 10.58.129.254). The reason for this is because our core switch doesn't support policy based routing, hence the reason behind this Linux box to perform routing on these VLANs.

With the router connected via the switches, from Device1, I can ping the interface 192.168.8.254 on the Linux router, but can't ping the other interface (10.58.129.254).

Switch2 configuration/diagnostics

switch2#show ip route

Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, S - Static
       B - BGP Derived, IA - OSPF Inter Area
       E1 - OSPF External Type 1, E2 - OSPF External Type 2
       N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2

S      0.0.0.0/0 [50/0] via 10.58.3.16,   vlan 3
C      10.58.3.0/24 [0/0] directly connected,   vlan 3
C      10.58.4.0/24 [0/0] directly connected,   vlan 4
C      10.58.5.0/24 [0/0] directly connected,   vlan 5
C      10.58.9.0/24 [0/0] directly connected,   vlan 9
C      10.58.10.0/24 [0/0] directly connected,   vlan 10
C      10.58.11.0/24 [0/0] directly connected,   vlan 11
C      10.58.12.0/24 [0/0] directly connected,   vlan 12
S      10.58.64.0/24 [40/0] via 10.58.3.17,   vlan 3
S      10.58.128.0/24 [40/0] via 10.58.3.254,   vlan 3
S      10.58.129.0/24 [1/0] via 10.58.3.254,   vlan 3 
S      192.168.8.0/24 [1/0] via 10.58.3.254,   vlan 3

switch2#ping 10.58.129.254
Pinging 10.58.129.254 with 64 bytes of data:
----10.58.129.254 PING Statistics----
4 packets transmitted,0 packets received,100% packet loss
round-trip (ms) min/avg/max = 0/NaN/0

switch2#ping 192.168.8.254
Pinging 192.168.8.254 with 64 bytes of data:
----192.168.8.254 PING Statistics----
4 packets transmitted,0 packets received,100% packet loss
round-trip (ms) min/avg/max = 0/NaN/0

Router diagnostics

router# traceroute -d 192.168.8.11
traceroute to 192.168.8.11 (192.168.8.11), 30 hops max, 60 byte packets
 1  192.168.8.11 (192.168.8.11)  0.237 ms  0.222 ms  0.211 ms

router# route -n
Kernel IP routing table
Destination     Gateway      Genmask         Flags Metric Ref    Use Iface
10.58.3.0       0.0.0.0      255.255.255.0   U     0      0        0 eth0
10.58.128.0     0.0.0.0      255.255.255.0   U     0      0        0 eth3
10.58.129.0     0.0.0.0      255.255.255.0   U     0      0        0 eth2
192.168.8.0     0.0.0.0      255.255.255.0   U     0      0        0 eth4

router# ping 192.168.8.11
PING 192.168.8.11 (192.168.8.11) 56(84) bytes of data.
64 bytes from 192.168.8.11: icmp_seq=1 ttl=128 time=2.23 ms
64 bytes from 192.168.8.11: icmp_seq=2 ttl=128 time=0.237 ms

Device1 diagnostics

(device1)c:\>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...bc 30 5b d8 41 c3 ...... Broadcom NetXtreme 57xx Gigabit Controller - Pac
ket Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0    192.168.8.254    192.168.8.11       20
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
      192.168.8.0    255.255.255.0     192.168.8.11    192.168.8.11       20
     192.168.8.11  255.255.255.255        127.0.0.1       127.0.0.1       20
    192.168.8.255  255.255.255.255     192.168.8.11    192.168.8.11       20
        224.0.0.0        240.0.0.0     192.168.8.11    192.168.8.11       20
  255.255.255.255  255.255.255.255     192.168.8.11    192.168.8.11       1
Default Gateway:     192.168.8.254
===========================================================================
Persistent Routes:
  None

(device1)c:\>tracert -d  10.58.129.254
Tracing route to 10.58.129.254 over a maximum of 30 hops
  1     *        *        *     Request timed out.
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
(etc. until 30 hops).

So from device1, running ping 10.58.129.254, and with tcpdump running on the 192.168.8.254 interface of the Linux router, I can see ICMP echo requests and replies

router# tcpdump -i eth4
17:08:08.326221 IP 192.168.8.11 > 10.58.129.254: ICMP echo request, id 512, seq 63746, length 40
17:08:08.326240 IP 10.58.129.254 > 192.168.8.11: ICMP echo reply, id 512, seq 63746, length 40

But the reply never returns to device1.

Does anybody know what the problem might be? tcpdump on eth2,3 & 4 also reveals the following output (I've not seen it on eth0, which is the one VLAN of the above that is routed by the core switch):

19:49:16.246286 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id
8000.a4:ba:db:69:74:91.8014, length 43
19:49:18.257007 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id
8000.a4:ba:db:69:74:91.8014, length 43

I understand this is spanning tree, but I don't know if this is a bad thing or not. Does this offer any clues? For info, the hardware address in the above STP message, is that of switch3.

share|improve this question
2  
When you run traceroute from either end, how far does the trace get? –  Rory Alsop May 9 '11 at 16:59
1  
+1 for the traceroute. Also, care to add the routing table from DEVICE1, ROUTER and DEVICE2 ? Also your problem pinging the remote interface of a multi-homed gateway reminds me of a specific problem with some linux, I'll check it. –  Renik May 9 '11 at 17:08
    
Ditto on the traceroute, but also from the reverse direction as well. On the same vein of thought, what/where are your references for your current ping outcomes? Have you also tried different reference points across your network in order zero in? –  user48838 May 9 '11 at 17:20
    
Thanks for the comments. I left the office a few hours ago, and I can only access the router from here. (I've updated the question with what little I can add at the moment). When I return to work in the morning, I'll add the remainder of the requested info to the question. –  Bryan May 9 '11 at 18:37
    
I'm willing to bet that the icmp requests and replies are coming and leaving from different interfaces. Check with a tcpdump on each interface separately. –  JimB May 9 '11 at 18:43

2 Answers 2

up vote 3 down vote accepted

In your second topology, you appear to have a split subnet. 192.168.8.0/24 spans multiple switches which you state are layer 3. In your output from switch2, you have a static route for this /24 pointing at a single interface:

S      192.168.8.0/24 [1/0] via 10.58.3.254,   vlan 3

This means that traffic hitting switch2 destined for either 192.168.8.254 or 192.168.8.11 will be forwarded to the same next hop. At least one of those destinations

In order for this to work as you intend, you have a couple of options:

  1. Configure switch[123] as layer 2 switches, such that the 192.168.8.0/24 is again a single broadcast domain.
  2. Configure separate networks on each of the links: {device1 -> switch1, switch1 -> switch2, etc}
share|improve this answer
    
+1: Thank Murali. I'm unable to test that now, as I've ended up using a different solution (as per my own answer), but I'm going to accept your answer anyway, as it does seem like that rouge static route would have been the cause of the problem. –  Bryan May 12 '11 at 9:31

Due to not being able to resolve this problem, a different approach is now being utilised which will vastly simplify the configuration. Instead of routing some VLANs by the core switch, and others by a Linux box, I'm now allowing the core switch to perform all routing between VLANs, with ACLs performing the segregation that two routers would have provided.

I'm going to move the Linux box to being the default gateway to the outside world for the entire network, with the help of iproute2, to perform source IP based routing, so the correct systems use the correct gateways.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.