Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am working on getting networking working correctly on a cluster of 14 virtual servers at a customer site. 11 of them work fine for routing and 3 don't work correctly for their administrative network (172.28.56.0). All are running Windows Web Server 2008R2.

Default gateway is set on the production network (172.28.58.0) and not on the administrative network (handled with persistent static routes).

On a working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX00 0c 29 85 b2 98 ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 15...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.11     266
        10.18.1.22 255.255.255.255      172.28.58.1    172.28.58.11      11
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.201      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
      172.28.34.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.42.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.56.0    255.255.255.0          On-link   172.28.56.201     266
      172.28.56.0    255.255.255.0      172.28.56.1   172.28.56.201      11
    172.28.56.201  255.255.255.255          On-link   172.28.56.201     266
    172.28.56.255  255.255.255.255          On-link   172.28.56.201     266
      172.28.58.0  255.255.255.224          On-link    172.28.58.11     266
      172.28.58.0  255.255.255.224      172.28.58.1    172.28.58.11      11
      172.28.58.1  255.255.255.255      172.28.58.1    172.28.58.11      11
     172.28.58.11  255.255.255.255          On-link    172.28.58.11     266
     172.28.58.31  255.255.255.255          On-link    172.28.58.11     266
      172.28.60.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.63.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      192.168.0.0      255.255.0.0      172.28.56.1   172.28.56.201      11
        224.0.0.0        240.0.0.0          On-link       127.0.0.1     306
        224.0.0.0        240.0.0.0          On-link   172.28.56.201     266
        224.0.0.0        240.0.0.0          On-link    172.28.58.11     266
  255.255.255.255  255.255.255.255          On-link       127.0.0.1     306
  255.255.255.255  255.255.255.255          On-link   172.28.56.201     266
  255.255.255.255  255.255.255.255          On-link    172.28.58.11     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

On one of the non-working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 16...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.21     266
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
       172.28.34.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.42.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.56.0   255.255.255.0      172.28.56.1   172.28.56.211      11
     172.28.56.211 255.255.255.255          On-link   172.28.56.211     266
       172.28.58.0   255.255.255.0      172.28.58.1    172.28.58.21      11
       172.28.58.0 255.255.255.224          On-link    172.28.58.21     266
      172.28.58.21 255.255.255.255          On-link    172.28.58.21     266
      172.28.58.31 255.255.255.255          On-link    172.28.58.21     266
       172.28.60.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.63.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       192.168.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         224.0.0.0       240.0.0.0          On-link       127.0.0.1     306
         224.0.0.0       240.0.0.0          On-link   172.28.56.211     266
         224.0.0.0       240.0.0.0          On-link    172.28.58.21     266
   255.255.255.255 255.255.255.255          On-link       127.0.0.1      06
   255.255.255.255 255.255.255.255          On-link   172.28.56.211     266
   255.255.255.255 255.255.255.255          On-link    172.28.58.21     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
      172.28.58.0    255.255.255.0      172.28.58.1       1
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

I am at a complete loss why the non-working servers have no On-link route for 172.28.56.0. Does anyone have any suggestions on what I should be looking at to figure this out? Also, I do have "physical" access to the console if needed through vSphere Client.

ipconfig /all from non-working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX4
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.211(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.21(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{4E9F1EC4-67C6-4B48-B3D1-4FDAF38452DB}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{7D31E116-0136-4E0F-B120-BF6C06FB9131}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

ipconfig /all from working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX1
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.201(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.11(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{713322C9-3AEA-4517-B8C6-4AAD3BFA4E62}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{2387819F-D4FA-4F52-8EEE-C8BF25A0846F}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 12:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
share|improve this question
    
can you post ipconfig too –  user1008764 Sep 14 '12 at 17:37
    
op edited, at end. –  smccloud Sep 14 '12 at 17:43
    
looks very strange. i would try the following. set the admin card back to dhcp -> deactivate and activate the card after that set the ip parameters -> deactivate and activate the card –  user1008764 Sep 14 '12 at 18:05
    
Just tried that, no change. Also, all non-working servers are on the same ESXi host. They cannot ping each other as well. Could it be the v-switch? –  smccloud Sep 14 '12 at 18:15
    
i tried disabling the production nic on one of the servers & setting a default gw on the admin again, still cannot ping anything (even the esxi host) –  smccloud Sep 14 '12 at 18:39
add comment

1 Answer 1

up vote 1 down vote accepted

The non-working server has a persistent route for:

172.28.58.0    255.255.255.0      172.28.58.1       1

...that the working server does not have, and the 172.28.58.x network is (apparently) a 255.255.255.224 network according to the IP config so I'd remove that route on the non-working server. I'm not sure what type of traffic is flowing on each NIC/subnets or what specifically "doesn't work correctly for their administrative network (172.28.56.0)"

If all non-working servers are on the ESXi host (and you've confirmed that that is the common thread by vMotioning one of them to another ESXi host and it works), then your issue might be in:

  1. the ESXi host's physical network connections (switch ports not configured correctly for the ESXi host connections)
  2. you have swapped your vSwitch mappings on the VM (NIC1 mapped to Prod vswitch & NIC2 mapped to Admin vswitch or vice versa)
  3. your vSwitch uplinks are swapped between physical NICs on the ESXi host (Prod vswitch uplinks to physical NIC1 & Admin vswitch uplinks to physical NIC2 or vice versa)

If you know the problem is limited to the one ESXi hosts, double-check the vswitch settings and compare to a working host to make sure they are the same and that the physical connections to the network for the host are the same as a working host.

share|improve this answer
    
Just removed that. Still the same issue. Basically, all traffic is attempting to go out over 172.28.58.0 no matter what. –  smccloud Sep 14 '12 at 18:35
    
i tried disabling the production nic on one of the servers & setting a default gw on the admin again, still cannot ping anything (even the esxi host) –  smccloud Sep 14 '12 at 18:41
    
According to Wireshark, I have traffic going across the administrative interface on the non-working servers. But I cannot connect to them VIA RDP, or ping them. I cannot ping between the 3 VMs on the same host, and they all stopped responding at around the same time. –  smccloud Sep 14 '12 at 18:59
    
August, Cannot vMotion the VMs, my user does not have permission for that. I have compared the vSwitch mappings on a working and non-working VM, same. I have also looked at the uplinks between a host w/ working VMs and the one not working, they are the same. I don't have physical access to the systems, nor can my user make any major changes (power on & off VMs & mount ISOs is all I can do). –  smccloud Sep 14 '12 at 19:00
    
Are the non-working servers identifying the proper networks they are connected to? If we have a 2008 R2 server come online and it can't get to a DC, it assumes it is on a public network, the firewall goes up, and nothing can connect to it. Since they are all on the same vswitch on the same host, pinging between VMs should work despite the ESXi host's physical network connection status. Also if they can't ping each other then it isn't a routing issue since they are on the same subnet (traffic won't even hit the gateway). –  August Sep 14 '12 at 19:18
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.