I'd like to know if there are any linux distributions that are focused on low latency networking.

The area I'm working in is algorithmic trading, and extremely low latency comms between machines is a must. The current h/w we're using is 10g ethernet, we're looking into things like infiniband RDMA and Voltaire VMA

I've googled around, and have only been able to find tidbtits of kernel patches, command line options and hardware suggestions.

I'm looking for a complete solution, specially built kernel, kernel bypass features, essentially all the goodies rolled up into one package - does such a thing even exist?

I ask as a lot of this stuff seems to be a black art, people keep secret what they know works etc.

closed as not constructive by Ward, HopelessN00b, John Gardeniers, EEAA, Bryan Dec 14 '12 at 12:28

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    I ask as a lot of this stuff seems to be a black art, people keep secret what they know works etc. Yeah, there's a reason for that. Because it's worth huge money - look at how much a coder or SA with this specialty goes for. Good luck in your search and all, but I'm sure wouldn't share something that valuable for free, and I doubt anyone else would either. Best approach is to get a big pile of money together and hire the talent you need to set this up. – HopelessN00b Dec 14 '12 at 3:57
  • 1
    In addition to what has already been said, questions of this type, which we refer to as "shopping questions" are explicitly off topic. – John Gardeniers Dec 14 '12 at 4:36
  • @HopelessN00b: A lot of what is answered on this site is greatly valued - yet people feel like it's 'ok' to share the knowledge and experience. I guess you're just not capable of grok'ing the 'stack exchange' concept. – Zanlor Dec 14 '12 at 5:23
  • Could QNX or another realtime OS be worth investigating? I'm not sure whether small patches are really going to change the latency structure of Linux much. It's been a long time since I worked with QNX but it was originally designed as a low-latency/ realtime OS. – Thomas W Sep 21 '17 at 3:27

If you really want low latency then infiniband verbs & rdma is the way to go. The reason is that the infiniband switches have a constant low latency.
e.g. For FDR (Fourteen Data Rate) infiniband the switches are around 140 nanoseconds.

And because infiniband doesn't use the IP stack, it doesn't have the accociated overhead of IP & TCP and the linux kernel. So moving data around through RDMA doesn't touch the CPU (except to set up the RDMA transfer).

10GBe is fast and good for IP apps but infiniband is really fast for pushing through data at ultra high speed. In any case you can run IPoIB and that is also very fast, but does add ALOT of overhead to your Infiniband network.

FDR Infiniband is here


Programming related: Infiniband verbs RDMA

  • Thanks for your answer, we are heading towards infiniband, however there's a lot that also needs to be done at the OS level, lots non networking specific params that eventually affect n/w performance. was hoping there'd be a package or something that would have all those wrapped up. – Zanlor Dec 14 '12 at 5:24
  • 1
    Funnily enough, we're using a series of kernel patches designed to make destop gamming better, so maybe if i'd reworded the question, left out the trading thing (irks some people) and mentioned gamming instead, perhaps i'd have had better luck... hard to explain the infiniband RDMA thing though :) – Zanlor Dec 14 '12 at 5:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.