Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question.
I got myself a dirt cheap Dell C6100 with 4 Nodes I want to do some number crunching for my master thesis on. I need low latency connection between them but I can not go for IB switch for two reasons, once it would probably double the cost of the hole setup, and I only have 2U of free rack space.
My idea was the same as discussed in the closed question, to use 3 Nodes and get 3 dual port Infiniband cards and connect them in a triangle.
I should point out that I am not primary interested in Ethernet over Infiniband, since they are connected over Ethernet in addition. I only care about the MPI performance.
My question now is
a) Will I be able to perform three times the setup described here Connect two infiniband cards to each other without a switch to create 3 Subnets in this Setup.
and then b) Will this configuration work with the setup described here http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-6.html
[ and c) do you see a setup in which I also can connect the 4th Node to the setup ]
