WDS PXE Server occasionally returning bad IP

Question

We have a very complex network boot environment to keep traffic separated for our imaging networks, which process hundred of machines an hour.

We use the WDS Server Selection menu tweak (http://www.stephan-schwarz.net/?p=58) to allow for multiple PXE servers as a manual load balancing solution.

In the process, I found there is a cap of 20 sources for non-UEFI machines and 16 sources for UEFI machines (I posted on Technet regarding this and am content its a BIOS cap)

I've since found the ability of MDT to define a list of server sources with an unknown limit so that's a load balancing solution once booted into the MDT image.

However, very rarely, I see a PXE problem I can't understand.

When I enter into the WDS Server Selection Menu, I see all 16 servers as IP [SERVER NAME (no domain)] as provided by the server that answered first. Policy is to choose a specific server (from 1-16). One of three things happens:

1- The server in question loads (Server 01 loaded the WDS Server Selection Menu, Server 05 was chosen and is booted from). This occurs 90% of the time.

2- The original source server loads instead of the chosen server. This occurs rarely, but may also be hidden by the fact that the server data is almost always the same (up until now)

3- The machine attempts to load an arbitrary IP, and it could be anything- including public IPs. Rebooting and trying again could cause a different IP, over and over. Sometimes the IP is the same. For example, I captured the packet traffic from a machine in Wireshark, and I see normal PXE traffic until the unit in question sends out a DHCP Request for an odd destination IP like 8.128.0.0

Now none of this happens if I let a machine randomly choose a boot server via network boot. I feel (though haven't tested an incredible amount) that it doesn't happen if the server needed is in the first 8 in UEFI (at least on the device in question, UEFI WDS Server Select shows a few devices and then you need to scroll).

Is there any rationale behind this? I have a 2MB pcap file at https://www.dropbox.com/s/qwsnvgda1lhxbcx/pcap_wds_ip_weirdness.pcapng?dl=0 that contains the traffic leading from fresh boot to the wrong IP being requested. Suggested filter would be eth.src == d4:c9:ef:f8:e4:db || eth.dst == d4:c9:ef:f8:e4:db

The environment is mostly HP - ProCurve 2530G and (rarely) 2510G switches with IGMP and STP enabled, Windows Server 2012 R2 servers on an ESXi Cluster, HP Elitebook 810 G2 devices mainly.

I appreciate any insight.

Answer 1

When a PXE client boots it receives N PXE offers from N DHCP/WDS servers. They all offer wdsnbp (or wdsmgfw.efi) as NBP; eventually one offer is accepted and the NBP is TFTP transferred and booted. Now the NBP starts a new PXE DCHP "request" but this time it saves "all" the DHCP ACKs received presenting a menu with all of them. This second time the WDS servers will offer the file bootmgr (or bootmgfw.efi) as the new NBP. Your wireshark shows exactly this.

The different limit (20/16) is not a BIOS/UEFI cap as you think; it's just a limit imposed by the first NBP.

The traffic capture also shows an offer from one DHCP server not offering the second NBP then I consider you either have a plain DHCP (not PXE enabled) or a miss-configured WDS server in your set-up, but this "should" not be the cause of your problem.

What you describe looks like a buggy WDSNBP (or wdsmgfw.efi). Why is this so erratic and difficult to debug? well the first NBP is randomly selected from all the offers then you might be booting different versions of it on each try. You should verify they all are exactly the same version; you might find that an old buggy version might be booted once in a while becoming the source of your problem. The NBPs offered by the PC that forces to boot from its own IP are the first suspicious.