From a Windows perspective only:
NTLM
- works with both domain accounts and local user accounts on the IIS box
- works with external (non-domain) clients or internal clients
- for domain accounts, only the server requires direct connectivity to a domain controller
- requires client connectivity only to the IIS server
- can traverse any proxy supporting HTTP Keep-Alive s
- use SSL to work around most others
- requires multiple round-trips to authenticate, with small packets
- (log pattern is 401, 401, 200)
- cannot be used in scenarios where double-hop authentication is required
- i.e. the user's credentials are to be forwarded to a service on another computer
- supports older clients (< Win2000)
- Is susceptible to LM Auth Level discrepancies (mismatched lmcompatibilitylevel)
- is used as a fallback by the Negotiate package if Kerb fails.
- (not "if access is denied with Kerb", it must break for NTLM to be used)
Kerberos
- works with currently domain-joined clients only
- requires client connectivity to an AD DC (tcp/udp 88) AND the server
- might be able to traverse a proxy, but see DC point above; you still need to be on the same network as the DC and the server*
- in reverse proxy scenarios (ISA/TMG), the protocol transition server needs to be on that network
- i.e. not the client, but then the client isn't really doing kerberos is it?
- requires a single round-trip to authenticate, but the payload size is relatively large (commonly 6-16K) (401, {token size} 200)
- can be used with (constrained) delegation to enable Windows authentication of the connecting user to the next service
- for example, to allow UserA to access IIS, and use that same user account when IIS accesses SQL Server (constrained: but not anything else, eg Exchange or another SQL box) - this is "delegation of authentication".
- is currently the primary security package for Negotiate authentication
- meaning Windows domain members prefer it when they can get it
- requires registration of SPNs, which can be tricky. Rules that help.
*- so in theory if you had a domain in which internet-connected clients chatted directly to an internet-connected DC, it's workable. But don't do that unless you already knew that.
While we're at it,
Basic
- can multi-hop. But does so by exposing your username and password directly to the target web app (which can then do anything it wants with them. Anything. Oh, did a Domain Admin just use my app? And did I just read their email? Then reset their password? Awww. Pity)
- needs transport layer security (i.e. TLS/SSL) for any form of security.
- and then, see previous issue
- works with any browser
- requires a single round-trip to authenticate (401, 200)
- can be used in multi-hop scenarios because Windows can perform an interactive logon with basic credentials
- May need the LogonType to be configured to accomplish this (think the default changed to network cleartext between 2000 and 2003, but might be misremembering)
- but again, see first issue. (getting the impression that the first issue is really, really important? It is.)
To sum up:
Kerb can be tricky to set up, but there are loads of guides (my one) out there that try to simplify the process, and the tools have improved vastly from 2003 to 2008 (SetSPN can search for duplicates, which is the most common breaking issue; use SETSPN -S anytime you see guidance to use -A, and life will be happier).
Constrained delegation is worth the cost of admission.