Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I will build computing cluster (about 64 nodes) and I'm looking for some informations. I want to connect them with Infiniband and install Linux. I tried to google it a little bit, but I couldn't find any informations how to set up a simple working configuration.

I want to use it to access NAS and for interprocess communication (MPI, etc.)

Is it easy to set up Infiniband on Linux? How to do it? Or maybe there is better solution?

share|improve this question
add comment

5 Answers

up vote 4 down vote accepted

The Infiniband itself is not a big deal to set up. You have to load the driver for your device and if you don't want to use anything else then IP the ipoib kernel module.

1 IP OVER INFINIBAND 2
3 The ib_ipoib driver is an implementation of the IP over InfiniBand 4 protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib 5 working group. It is a "native" implementation in the sense of 6 setting the interface type to ARPHRD_INFINIBAND and the hardware 7 address length to 20 (earlier proprietary implementations 8 masqueraded to the kernel as ethernet interfaces).

After this you are able to use your device as a normal ethernet device.

share|improve this answer
add comment

Do you absolutely need Inifiband? Or do you just need high bandwidth links between things? You might be better off bonding several 1GigE links or going with 10GigE.

share|improve this answer
    
NAS and IPC have different performance characteristics. I would seriously consider FC for NAS, or perhaps FCoE or even iSCSI if you have gigE. Bonding is good if you have sufficient ports available, but do your calculations first, some multi-port NICs can't sustain the aggregate transfer rates required. Personally, I would use separate switches and networks for FCoE (layer 1) and TCP/IP (layer 3) traffic. –  CyberFonic May 29 '09 at 23:52
add comment

Sun HPC Linux Edition is a Linux clustered HPC server with Infiniband. I don't know the details of setting it up, but some good performance numbers are coming.

share|improve this answer
add comment

In our university Scientific Linux,which is prepared by globally known research labs, is used on the cluster. With a wef extra packages Infiniband and IP over Infiniband is possible.

share|improve this answer
add comment

Easy? cheap? not really, but definitely possible. Since you're not going for the budget solution (that's six figures for the metal only), you'd probably want to use support from a paid, enterprise, OS.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.