If I'm reading the sysctl/tcp stuff correctly, it's tripped when the number of un-ACKed syn requests exceeds the value of net.ipv4.tcp_max_syn_backlog. Specifically:
The tcp_max_syn_backlog variable tells
your box how many SYN requests to keep
in memory that we have yet to get the
third packet in a 3-way handshake
from. The tcp_max_syn_backlog variable
is overridden by the tcp_syncookies
variable, which needs to be turned on
for this variable to have any effect.
If the server suffers from overloads
at peak times, you may want to
increase this value a little bit.
The reason I think it's that simple is the text from tcp_syncookies:
The tcp_syncookies variable is used to
send out so called syncookies to hosts
when the kernels syn backlog queue for
a specific socket is overflowed. This
means that if our host is flooded with
several SYN packets from different
hosts, the syn backlog queue may
overflow, and hence this function
starts sending out cookies to see if
the SYN packets are really legit.
To me that makes it sound like it really is as simple as the syn queue having > tcp_max_syn_backlog outstanding connections.