0

I'm facing a very strange situation with some virtual machines on my vSphere setup and I can't quite figure out what is happening.

Originally, I am working with a 192.168.9.0/24 network where 192.168.9.254 is the DHCP server, 192.168.9.43 is the gateway, 192.168.9.82 is my workstation (it received its IP from the DHCP server) and 192.168.9.15 is the one for my colleague.
This works just fine and every machine on that network can work with the others, they all are capable of pinging one another as well as the rest of the world via the gateway.

A VSphere 6.5 cluster has been installed, with three hosts which have the 192.168.9.1, 192.168.9.2 and 192.168.9.3 static addresses respectively. Those machines are running ESXi version 6.0.0, 3380124 and each have four NICs connected to a pair of stacked Dell N1524 switches, said switches being connected to the 192.168.9.0/24 network. On that cluster, there is a Production network that is tied to each hosts NICs and so the VMs get their IPs from the 192.168.9.254 DHCP. This also works just fine, but because there has been an increase in VM usage, the IP range served by the DHCP server is now quite crowded, to the point where some regular users are not able to get an IP address if they arrive in the afternoon.

To avoid this, I added a new port group on the vSwitch for each host and gave those port groups the same name (VLAN) and the same VLAN value, being 42.
The Dell physical switches have been reconfigured to allow that VLAN along with the default one on the ports where the NICs from the hosts are connected (trunk mode). I decided that this VLAN would be a 10.10.10.0/24 network so that it is easily recognizable from the regular network and so gave the switch the 10.10.10.252 static IP on that VLAN.

Then, I created a Windows 2012 virtual machine that has two interfaces, one on Production (192.168.9.110), one on VLAN (10.10.10.254) and activated the RRAS role so that this machine now acts as the gateway between 10.10.10.0/24 and the rest of the world.
I created a second Windows 2012 virtual machine that only has one interface, on VLAN with the static 10.10.10.253 address and named it MDC. I activated the Domain Controller, DHCP and DNS roles. DHCP serves leases in the 10.10.10.50 - 10.10.10.200 range while DNS simply forwards to the DNS from the 192.168.9.0/24 network

I then created two virtual machines, one on the first host, alongside MDC and Gateway, and one on the third host by itself, both connected to the VLAN network. As connectivity appeared to work properly, I decided to move existing VMs from the Temporary folder to the VLAN network, using this PowerCLI command:

Get-Folder Temporary | Get-VMs | Get-networkadapater | set-networkadapter -NetworkName VLAN

I also took the opportunity to make sure that all network adapters are vmxnet3 with this command

Get-Folder Temporary | Get-VMs | Get-networkadapater | set-networkadapter -Type vmxnet3

As connectivity was still OK, I created another bunch of virtual machines, also connected to the VLAN network, placed on all three hosts, which gives the following topology:

Host 1
MDC (10.10.10.253)
Gateway (10.10.10.254192.168.9.110)
Machine1_H1 (10.10.10.64)
Machine2_H1 (10.10.10.57)

Host 2
Machine3_H2 (10.10.10.65)

Host 3
Machine4_H3 (10.10.10.50)
Machine5_H3 (10.10.10.51)

And this is where I am getting very strange results when it comes to network connectivity, both inside VLAN and when connecting to the outside world:

  • MDC can ping everyone but the switch (10.10.10.252)
  • Gateway can ping everyone but Machine5_H3
  • Machine1_H1 can ping everyone but Machine3_H2
  • Machine2_H1 can ping everyone but the switch (10.10.10.252)
  • Machine3_H2 can ping everyone except Host 1 and Machine1_H1
  • Machine4_H3 can ping everyone except 192.168.9.43, 192.168.9.15 and google.fr (Name resolution is OK)
  • Machine5_H3 can ping everyone except 192.168.9.254, 192.168.9.82 (my own workstation) and 10.10.10.254
  • My own computer (192.168.9.82) can ping everyone but Machine5_H3 (10.10.10.51)

I made sure that firewalls are off on all machines before doing theses tests, and I also ran arp -a on MDC to see if there was a MAC address conflict and there were no duplicates. The machines in the Temporary folder were also all turned off just in case, but it did not change a thing to the results above. Just to be on the safe side, I used this snippet to force generating a new MAC address for those machines:

foreach ($VM in (Get-Folder Temporary | Get-VM))
{
  $NetworkAdapter = $VM | Get-NetworkAdapter
  $NetworkAdapter | Set-NetworkAdapter -MacAddress 00:50:56:1a:ff:ff -Confirm:$false
  $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
  $spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (1)
  $spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec
  $spec.deviceChange[0].operation = "edit"
  $spec.deviceChange[0].device = $NetworkAdapter.ExtensionData
  $spec.deviceChange[0].device.addressType = "generated"
  $spec.deviceChange[0].device.macAddress = $null
  $VM.ExtensionData.ReconfigVM_Task($spec)
}

That did not change a thing to the situation.

I then installed Wireshark on the gateway, started monitoring the traffic on 10.10.10.254 and I could see every traffic in which that machine is implicated. For instance, if my workstation (192.168.9.82) is pinged by Machine5_H3 (10.10.10.51), I can see the PING request, then the PING reply and yet Machine5_H3 complains it did not receive any reply. If I do it the other way round, I can see the request from 192.168.9.82 but no reply is ever seen by the gateway.

I thus believe that some packets are dropped somewhere, my main suspect being the switch (10.10.10.252) but I'm not sure what I can do to confirm this theory.

Link aggregation was originally activated on the DELL switch stack but it was giving issues connecting from our workstations to the VMs that have IPs in the 192.168.9.0/24 network, so we turned it off.
Changing this setting on the switch stack did not change a thing to the above situation though.

I must have done something wrong, or missed some configuration details, but I can't figure out what it is and would appreciate any suggestion to help solve what is is a mystery to me.

  • How is the NIC teaming configured on the vSwitch? Have you aggregated the host ports on the Dell switches? – Zac67 Feb 28 '18 at 19:44
0

Following comment by Zac67, we verified the NIC teaming configuration on all three hosts, and we discovered that the first two were using the "Route based on IP hash" parameter while the third host used "Route based on originating virtual port".

We then set the third host to the same value as the others and read the warning associated with the first option which says "Link aggregation should be setup on the physical switch".

We thus went back to the switch and reactivated link aggregation for the appropriate ports, but it made the entire connectivity unstable, machines in the 192.168.9.0/24 network becoming partially unreachable while it did not change a thing for those in the 10.10.10.0/24 network.

So we decided to go the opposite way and disabled link aggregation on the switches and used the "Route based on originating virtual port" option on all three hosts.

This allowed to get back normal behavior for the 192.168.9.0/24 network and better connectivity for the 10.10.10.0/24 network. I'm saying better because some machines were still unreachable, namely those on Host3 that could not even reach the DHCP server to retrieve an IP.
Using Wireshark to observe the traffic, we discovered that ARP broadcasts where sometimes filtered, thus explaining why some machines could not talk to each other but still not giving us any clue on a possible solution.

After having been stuck on this for a couple of weeks without any hope of finding an answer, we brought in the consultants who helped install the infrastructure in the first place and they told us two things:

  1. LACP is not compatible with VLANs
  2. VLAN 42 was forbidden on one of the switches port

So, ensuring the configuration did not use LACP at all and removing the restriction on the port allowed to get to a fully working situation.

Now, we are left to wonder how we managed to get VLAN 42 forbidden on only one port on the switch.

As to LACP and VLAN incompatibility, it never occurred to us that this could be the source of our problems, but now that they told us about it, it seems it's a well known issue when stacking DELL switches but I could not find any definitive answer on the subject. But since it works without it, it's all fine by me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.