2

We are currently using Link Status Only for our NIC Teams on our VM hosts, but recently ran into an issue where one of our two switches had a memory error and stopped passing traffic. All of our VM hosts went down, and most of the guests (who had been using that path out already) also stopped responding until we shut off that switch manually.

In the Linux bonding environment, you can use arp_intervals as another way to detect link status, but in VMWare there is only Beacon Probing. BP is not the same as the arp_interval in that you don't choose a host to test connectivity to, as well as you need three or more interfaces to do it.

All of our VM Hosts have four NICs, so the three NIC requirement shouldn't be too much trouble. However, while the documentation only states that at least three separate physical NICs (pNICs) are required, every example also has three separate physical switches, and it doesn't state whether that's also a requirement. As I have looked for the answer to this, I came across this blog which states:

"Don’t use Beacon Probing if more than one pNIC in the vSwitch is connected to the same pSwitch. This could result in the same MAC address being presented on two or more ports on the pSwitch which is “a very bad thing”"

We don't have three switches in our configuration to just add to this problem, and in some of my preliminary testing I was having unexplained link flapping issues that may be related to them being plugged into the same switch.

So are three separate physical switches also a requirement for beacon probing? Am I relegated to link status only for my configuration? And, semi-rhetorically, why don't they have arp_interval as an option in their NIC teaming?

2
  • This seems to be an uncommon configuration. I've never seen this used in practice. What type of switches were they?
    – ewwhite
    May 24, 2013 at 12:09
  • Two Cisco 4948's.
    – Aaron R.
    May 24, 2013 at 15:17

1 Answer 1

4

With beacon probing is is recommended to use at least 3 pNICs because that is the way the beacon works best. ESXi sends out a broadcast packet out of the physical NIC cards. The other pNICs within the same vSwitch then wait to see if they receive the packets from the other pNICs. Whichever pNIC doesn't receive the broadcast, ESXi then assumes it is a down link.

Attaching all 3 pNICs to the same switch and using beacon probing is a waste of resources when link status would work because it's simpler. Is the link on or off? Configuration issues (STP or port blocks) won't show up with link status.

The intent and design of the beacon probing was to have the pNICs attached to different pSwitches because it was used to "test" downstream switches; switches beyond the ones the pNICs were attached to. BP can determine if say the 3rd pSwitch downstream to the iSCSI SAN has failed, link status won't detect it but BP should. Then the ESXi server can determine what it wants to do. Link status would continue to try to send packets to the SAN even though it is not available.

6
  • Right, the pNICs are on the same vSwitch, but do those pNICs also need to be on separate pSwitches? The documentation implies it, but doesn't state it outright.
    – Aaron R.
    May 23, 2013 at 19:49
  • Updated my original answer. In a nutshell, the design of BP was to be complimented with a layer 2, redundant network to determine downstream link failures beyond the first pNIC -> pSwitch connection.
    – Travis
    May 24, 2013 at 2:26
  • I can understand the efficiency may go down a bit, but we are really trying to prevent the kind of failure I outlined above. What do you mean by redundant layer two beyond what we have been talking about here? I am really trying to find an ESXi-version of the arp_interval redundancy. So even if it would mean doubling up two of the NICs on one switch, which would be less efficient, is it still technically possible? Having the second switch is wasted if it doesn't fail over when the first goes down.
    – Aaron R.
    May 24, 2013 at 3:02
  • I re-read my answer. The statement about the MAC addresses are correct. When you have the same pNICs in the same vSwitch they are teamed and to the network they will appear to have 1 MAC address. So, if you have 3 pNICs in 1 vSwitch and the BP sends out the broadcast it will come from one MAC address but 3 different pNICs which will not work. You will need individual pSwitches for each pNIC within the same vSwitch otherwise it will not work correctly.
    – Travis
    May 24, 2013 at 12:11
  • There is no arp_interval redundancy that I know of in ESXi.
    – Travis
    May 24, 2013 at 12:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .