Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I will build computing cluster (about 64 nodes) and I'm looking for some informations. I want to connect them with Infiniband and install Linux. I tried to google it a little bit, but I couldn't find any informations how to set up a simple working configuration.

I want to use it to access NAS and for interprocess communication (MPI, etc.)

Is it easy to set up Infiniband on Linux? How to do it? Or maybe there is better solution?

share|improve this question
up vote 4 down vote accepted

The Infiniband itself is not a big deal to set up. You have to load the driver for your device and if you don't want to use anything else then IP the ipoib kernel module.

3 The ib_ipoib driver is an implementation of the IP over InfiniBand 4 protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib 5 working group. It is a "native" implementation in the sense of 6 setting the interface type to ARPHRD_INFINIBAND and the hardware 7 address length to 20 (earlier proprietary implementations 8 masqueraded to the kernel as ethernet interfaces).

After this you are able to use your device as a normal ethernet device.

share|improve this answer

Do you absolutely need Inifiband? Or do you just need high bandwidth links between things? You might be better off bonding several 1GigE links or going with 10GigE.

share|improve this answer
NAS and IPC have different performance characteristics. I would seriously consider FC for NAS, or perhaps FCoE or even iSCSI if you have gigE. Bonding is good if you have sufficient ports available, but do your calculations first, some multi-port NICs can't sustain the aggregate transfer rates required. Personally, I would use separate switches and networks for FCoE (layer 1) and TCP/IP (layer 3) traffic. – CyberFonic May 29 '09 at 23:52

Easy? cheap? not really, but definitely possible. Since you're not going for the budget solution (that's six figures for the metal only), you'd probably want to use support from a paid, enterprise, OS.

share|improve this answer

In our university Scientific Linux,which is prepared by globally known research labs, is used on the cluster. With a wef extra packages Infiniband and IP over Infiniband is possible.

share|improve this answer

Sun HPC Linux Edition is a Linux clustered HPC server with Infiniband. I don't know the details of setting it up, but some good performance numbers are coming.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.