Short: NLB doesn't care.
NLB doesn't give a rat's donkey about your application. User-mode problem are so user-mode.
As long as the network is able to send and receive NLB broadcasts (or multicasts!), NLB will keep on accepting traffic.
Longer: NLB really doesn't care.
NLB runs as a Layer 2 NDIS filter, and simply runs its hash algorithm across all incoming traffic that matches its port rules.
Every node has to see all incoming traffic (multicast/broadcast by the switch), and every node drops packets that don't meet its hash criteria.
NLB is a statistical load balancer.
- It's not important to it that your app is throwing 500s.
- It's oblivious to the fact that your user-mode listener process has crashed.
- It's unaware that your app is running slowly.
It just. Doesn't. Care.
The best-case failure for NLB is for one of:
- the box to die completely from a power failure
- the network cable to be cut or unplugged (possibly by a vacuum cleaner)
- (or the NIC to explode in a shower of sparks)
- the box to bluescreen
Any of these stop the network stack from processing incoming packets, and from sending "I'm alive!" broadcast messages to other NLB nodes, which causes the cluster to get all introspective for a little while, and re-converge. Then, when the remaining nodes are clear on how many of them there are, they'll start their hashing again, and drop any new packets they're not interested in.
If you need health monitoring, careful application of load to underutilized servers, and response time sensitivity, NLB is not the solution for you. If your app is OK with that, or knows enough about NLB to run NLB STOP when there's a problem at the app layer, then it's probably fine. But very few apps do that (ISA/TMG spring to mind).
Looks like I wrote about this a while ago here.
If you're looking for a low-cost Windows-based solution, consider Application Request Routing for IIS 7 - it has most of the health monitoring features it sounds like you're looking for. You wouldn't run it on the same box as the applications, generally, though, and for availability, you'd run NLB underneath the ARR boxes.