We've come to the stage of our project where we want to take the software to the next level, generally response times of 20µs or less. Software wise its as good as it gets (though there's room for a little more improvement), however now we're looking to attack the hardware side of things.

I was wondering if you could provide suggestions on how we can accomplish this feat. Do we need any special type of cabling, software stacks, architectures or other technologies?

  • What kind of distances are we talking about here? Networking is normally gauged in ms no µs how fast do you need/want to be? – Zypher Jun 7 '11 at 22:38
  • @Zypher: This would be all LAN related - rack mounted servers, cables no longer than 4-5meters, the application in question does a great deal of calculations in a distributed manner. – Seminar Jun 7 '11 at 23:19

Ultra low latencies? You get to play with the fun toys. The usual platform for this kind of really low latency networking would be InfiniBand (a couple of us, though not me, have gotten to use this stuff). Used in high-performance-computing and other computing areas that require low-latencies, high bandwidth, and scalability. Commonly found in certain scientific computing areas, as well.

Part of how IB gets the low latencies it does is by flattening the network. Similar to Fibre Channel, it is a layer 2 switched network which really helps bring down latency.

You won't find the gear to do this at your local MicroCenter. Using IB for networking is entirely doable, though TCP/IP is perhaps a mediocre choice of protocol if you want the lowest latencies possible.

Because of the low volume in the market, IB network adapters are very expensive compared to 1GB Ethernet, though they do compare favorably with 10GbE adapters. IB switches are similarly expensive, though seem to be near price-parity with 10GbE switches of similar port density.

If IB makes you leery, 10 Gigabit Ethernet can get you most of the way there and is very well supported by everything and is familiar to run of the mill network geeks. If you design your Ethernet so there is one, maybe 2, switches in it, and all ports are in the same subnet, you can get similar low latencies. Depending on exactly how much latency you're willing to tolerate, this may be a more understandable architecture than IB would be.


You are not even telling which network technology you are planning to use.

Assuming Ethernet and dividing the minimum frame length of 520 bytes by your 20 µs RT requirement gets you somewhere into the range of 50 MB/s. Further assuming that your software needs at least half of the time for processing, you are at 100 MB/s and probably looking at 10GE at that point to be able to accommodate for switching and transmission latency.

Then again, you do have plenty of headroom - so I am wondering why you are bothering with things like cabling types which surely will not affect your latency to the order of magnitude where it would get noticeable.

Edit: I'm used to believe that the solution depends on the particular problem. While 10GE latency and transport characteristics might be good for one application, they might present a problem for another. With 10 GE you would not have guaranteed RTTs but just best-effort delivery. This might be enough if your network is sufficiently over-engineered and it is okay for your application to get a response within 20 µs "most of the time". If the response time is a hard limit and a transaction would break because of this, you might want to look into other (mostly cell-based) networking technologies e.g. ATM.

As for Infiniband, you are probably looking into a major rewrite of your application's networking operations for the use of RDMA, if you want efficiency.

Another thing to consider is the distance between components - 10 µs is just enough time for light to travel 1,6 km and back - without any components latency and without considering bit times. So you would have to have short distances and short request/response sequences in order to get there.

All summed up it is not a question of individual components, but of the chosen technology.

  • dj: Thanks for the reply, its currently ethernet, but we're investigating RDMA solutions (inifiniband etc), I was hoping to guage the opinions regarding best of breed solutions, even specific suggestions such as NICs etc. – Seminar Jun 7 '11 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.