Weird routing issue

Question

I am working on getting networking working correctly on a cluster of 14 virtual servers at a customer site. 11 of them work fine for routing and 3 don't work correctly for their administrative network (172.28.56.0). All are running Windows Web Server 2008R2.

Default gateway is set on the production network (172.28.58.0) and not on the administrative network (handled with persistent static routes).

On a working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX00 0c 29 85 b2 98 ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 15...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.11     266
        10.18.1.22 255.255.255.255      172.28.58.1    172.28.58.11      11
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.201      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
      172.28.34.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.42.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.56.0    255.255.255.0          On-link   172.28.56.201     266
      172.28.56.0    255.255.255.0      172.28.56.1   172.28.56.201      11
    172.28.56.201  255.255.255.255          On-link   172.28.56.201     266
    172.28.56.255  255.255.255.255          On-link   172.28.56.201     266
      172.28.58.0  255.255.255.224          On-link    172.28.58.11     266
      172.28.58.0  255.255.255.224      172.28.58.1    172.28.58.11      11
      172.28.58.1  255.255.255.255      172.28.58.1    172.28.58.11      11
     172.28.58.11  255.255.255.255          On-link    172.28.58.11     266
     172.28.58.31  255.255.255.255          On-link    172.28.58.11     266
      172.28.60.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.63.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      192.168.0.0      255.255.0.0      172.28.56.1   172.28.56.201      11
        224.0.0.0        240.0.0.0          On-link       127.0.0.1     306
        224.0.0.0        240.0.0.0          On-link   172.28.56.201     266
        224.0.0.0        240.0.0.0          On-link    172.28.58.11     266
  255.255.255.255  255.255.255.255          On-link       127.0.0.1     306
  255.255.255.255  255.255.255.255          On-link   172.28.56.201     266
  255.255.255.255  255.255.255.255          On-link    172.28.58.11     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

On one of the non-working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 16...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.21     266
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
       172.28.34.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.42.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.56.0   255.255.255.0      172.28.56.1   172.28.56.211      11
     172.28.56.211 255.255.255.255          On-link   172.28.56.211     266
       172.28.58.0   255.255.255.0      172.28.58.1    172.28.58.21      11
       172.28.58.0 255.255.255.224          On-link    172.28.58.21     266
      172.28.58.21 255.255.255.255          On-link    172.28.58.21     266
      172.28.58.31 255.255.255.255          On-link    172.28.58.21     266
       172.28.60.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.63.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       192.168.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         224.0.0.0       240.0.0.0          On-link       127.0.0.1     306
         224.0.0.0       240.0.0.0          On-link   172.28.56.211     266
         224.0.0.0       240.0.0.0          On-link    172.28.58.21     266
   255.255.255.255 255.255.255.255          On-link       127.0.0.1      06
   255.255.255.255 255.255.255.255          On-link   172.28.56.211     266
   255.255.255.255 255.255.255.255          On-link    172.28.58.21     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
      172.28.58.0    255.255.255.0      172.28.58.1       1
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

I am at a complete loss why the non-working servers have no On-link route for 172.28.56.0. Does anyone have any suggestions on what I should be looking at to figure this out? Also, I do have "physical" access to the console if needed through vSphere Client.

ipconfig /all from non-working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX4
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.211(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.21(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{4E9F1EC4-67C6-4B48-B3D1-4FDAF38452DB}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{7D31E116-0136-4E0F-B120-BF6C06FB9131}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

ipconfig /all from working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX1
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.201(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.11(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{713322C9-3AEA-4517-B8C6-4AAD3BFA4E62}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{2387819F-D4FA-4F52-8EEE-C8BF25A0846F}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 12:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

looks very strange. i would try the following. set the admin card back to dhcp -> deactivate and activate the card after that set the ip parameters -> deactivate and activate the card — user1008764, Sep 14, 2012 at 18:05
Just tried that, no change. Also, all non-working servers are on the same ESXi host. They cannot ping each other as well. Could it be the v-switch? — smccloud, Sep 14, 2012 at 18:15
i tried disabling the production nic on one of the servers & setting a default gw on the admin again, still cannot ping anything (even the esxi host) — smccloud, Sep 14, 2012 at 18:39

August · Accepted Answer · 2012-09-14 18:59:04Z

1

The non-working server has a persistent route for:

172.28.58.0    255.255.255.0      172.28.58.1       1

...that the working server does not have, and the 172.28.58.x network is (apparently) a 255.255.255.224 network according to the IP config so I'd remove that route on the non-working server. I'm not sure what type of traffic is flowing on each NIC/subnets or what specifically "doesn't work correctly for their administrative network (172.28.56.0)"

If all non-working servers are on the ESXi host (and you've confirmed that that is the common thread by vMotioning one of them to another ESXi host and it works), then your issue might be in:

the ESXi host's physical network connections (switch ports not configured correctly for the ESXi host connections)
you have swapped your vSwitch mappings on the VM (NIC1 mapped to Prod vswitch & NIC2 mapped to Admin vswitch or vice versa)
your vSwitch uplinks are swapped between physical NICs on the ESXi host (Prod vswitch uplinks to physical NIC1 & Admin vswitch uplinks to physical NIC2 or vice versa)

If you know the problem is limited to the one ESXi hosts, double-check the vswitch settings and compare to a working host to make sure they are the same and that the physical connections to the network for the host are the same as a working host.

edited Sep 14, 2012 at 18:59

answered Sep 14, 2012 at 18:20

August

3,11416 silver badges17 bronze badges

Just removed that. Still the same issue. Basically, all traffic is attempting to go out over 172.28.58.0 no matter what.
– smccloud
Sep 14, 2012 at 18:35
i tried disabling the production nic on one of the servers & setting a default gw on the admin again, still cannot ping anything (even the esxi host)
– smccloud
Sep 14, 2012 at 18:41
According to Wireshark, I have traffic going across the administrative interface on the non-working servers. But I cannot connect to them VIA RDP, or ping them. I cannot ping between the 3 VMs on the same host, and they all stopped responding at around the same time.
– smccloud
Sep 14, 2012 at 18:59
August, Cannot vMotion the VMs, my user does not have permission for that. I have compared the vSwitch mappings on a working and non-working VM, same. I have also looked at the uplinks between a host w/ working VMs and the one not working, they are the same. I don't have physical access to the systems, nor can my user make any major changes (power on & off VMs & mount ISOs is all I can do).
– smccloud
Sep 14, 2012 at 19:00
Are the non-working servers identifying the proper networks they are connected to? If we have a 2008 R2 server come online and it can't get to a DC, it assumes it is on a public network, the firewall goes up, and nothing can connect to it. Since they are all on the same vswitch on the same host, pinging between VMs should work despite the ESXi host's physical network connection status. Also if they can't ping each other then it isn't a routing issue since they are on the same subnet (traffic won't even hit the gateway).
– August
Sep 14, 2012 at 19:18

| Show 3 more comments

Stack Exchange Network

Weird routing issue

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
gateway
route
.

Hot Network Questions

Weird routing issue

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged gatewayroute.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
gateway
route
.