In my experience the biggest cause of abnormal latency on otherwise healthy high-speed networks are TCP Windowing (RFC1323, section 2) faults, with a closely related second in faults surrounding TCP Delayed Acks (RFC1122 section 184.108.40.206). Both of these methods are enhancements to TCP for better handling of high speed networks. When they break, speeds drop to very slow levels. Faults in these cases affect large transfers (think backup streams), where extremely transactional small traffic (average data transfer is under the MTU size and there is a LOT of back-n-forth) will be less affected by these.
Again, I've seen the biggest problems with these two issues when two different TCP/IP stacks are talking. Such as Windows/Linux, 2.4-Linux/2.6-Linux, Windows/NetWare, Linux/BSD. Like to like works very, very well. Microsoft rewrote the Windows TCP/IP stack in Server 2008 which introduced Linux interoperability problems that didn't exist with Server 2003 (I believe these are fixed, but I'm not 100% sure of that).
Disagreements on the exact method of Delayed or Selective Acknowledgments can lead to cases like this:
192.168.128.5 -> 192.168.128.20: 1500b payload, SEQ 1562
192.168.128.5 -> 192.168.128.20: 1500b payload, SEQ 9524
192.168.128.20 -> 192.168.128.5: ACK 1562
192.168.128.5 -> 192.168.128.20: 1500b payload, SEQ 12025
192.168.128.5 -> 192.168.128.20: 1500b payload, SEQ 13824
192.168.128.20 -> 192.168.128.5: ACK 12025
Throughput goes through the floor because of all of the 200ms timeouts (Windows defaults it's delayed-ack timer to 200ms). In this case, both sides of the conversation failed to handle TCP Delayed Ack.
TCP Windowing faults are harder to notice because their impact can be less obvious. In extreme cases Windowing fails completely and you get packet->ack->packet->ack->packet->ack which is really slow when transferring anything significantly larger than about 10KB and will magnify any fundamental latency on the link. The harder to detect mode is when both sides are continually renegotiating their Window size and one side (the sender) fails to respect the negotiation which requires a few packets to handle before data can continue to be passed. This kind of fault shows up in red blinking lights in Wireshark traces, but manifests as lower than expected throughput.
As I mentioned, the above tend to plague large transfers. Traffic like streaming video or backup streams can be really nailed by them, as well as simple downloading of very large files (like Linux distro ISO files). As it happens, TCP Windowing was designed as a way to work around fundamental latency problems as it allows pipelining of data; you don't have to wait for round-trip-time for each packet sent you can just send a big block and wait for a single ACK before sending more.
That said, certain network patterns don't benefit from these work-arounds. Highly transactional, small transfers, such as those generated by databases, suffer most from normal latency on the line. If the RTT is high these workloads will suffer greatly, where large streaming workloads will suffer a lot less.