Ultra low latencies? You get to play with the fun toys. The usual platform for this kind of really low latency networking would be InfiniBand (a couple of us, though not me, have gotten to use this stuff). Used in high-performance-computing and other computing areas that require low-latencies, high bandwidth, and scalability. Commonly found in certain scientific computing areas, as well.
Part of how IB gets the low latencies it does is by flattening the network. Similar to Fibre Channel, it is a layer 2 switched network which really helps bring down latency.
You won't find the gear to do this at your local MicroCenter. Using IB for networking is entirely doable, though TCP/IP is perhaps a mediocre choice of protocol if you want the lowest latencies possible.
Because of the low volume in the market, IB network adapters are very expensive compared to 1GB Ethernet, though they do compare favorably with 10GbE adapters. IB switches are similarly expensive, though seem to be near price-parity with 10GbE switches of similar port density.
If IB makes you leery, 10 Gigabit Ethernet can get you most of the way there and is very well supported by everything and is familiar to run of the mill network geeks. If you design your Ethernet so there is one, maybe 2, switches in it, and all ports are in the same subnet, you can get similar low latencies. Depending on exactly how much latency you're willing to tolerate, this may be a more understandable architecture than IB would be.