Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's say I have 3 different servers, each one with an infiniband card. Each card has 2 different ports. (I don't know about the model yet)

Is it possible to create 3 different networks and to allow the 3 servers to communicate with each other without any problems? (and any spof).

I guess I just have to setup the /etc/hosts correctly.

I really don't know about infiniband, so please help me :)

Thanks in advance.

EDIT: Point is to NOT USE a switch!

share|improve this question

closed as not a real question by Ward, Scott Pack, Brent Pabst, mdpc, HopelessN00b Nov 9 '12 at 19:41

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

InfiniBand fabric does not use IP, therefore /etc/hosts thing does not apply. Yet you can run IP on top of InfiniBand. – Dmitri Chubarov Nov 8 '12 at 20:07
Dmitri, IB supports IP encapsulation - in fact it's used quite extensively in at least two very large CGI renderfarms I'm aware of. Though I'll admit it's really not the lowest cost IP solution :) – Chopper3 Nov 8 '12 at 20:33
Why do you want to do this? – ewwhite Nov 8 '12 at 21:05
Buy two cards per system? Build a triangle? – Zoredache Nov 8 '12 at 22:52
Pity the question was closed. The answer is actually yes. You could have done what you ask. Whether it's a good idea is another question. But, I can see why you'd do this if you were trying to save money on an IB switch. – Matt Mar 26 '15 at 1:33

You're just talking about doing a point-to-point connection from each host to each of the other two right?

I've not tried it, but from what I have heard you can do point-to-point connections using IB without using a switch (just as you can with ethernet).

Problems might come in with the subnet manager(s) - I'm not sure if each node's subnet manager can run actively for one of the connected networks and backup for the other. You may need to run two subnet managers per box and bind each to a specific port. Not sure how to do that either but definitely worth researching that side of it more.

And obviously having a couple of switches would make things easier (and more scalable).

share|improve this answer
Yes, you need 1 subnet manager per network. It is also perfectly safe to run 2 subnet managers per node, but you don't need to. – Navin Mar 25 at 9:41

Yes, just buy one or two (for resilience) IB switches. That said you don't mention which protocols you want to use, though it seems like you might want to run IP over IB, which works - and editing hosts (or having a proper DNS environment) will help to make that work but IB is really designed for IB-specific protocols, IP over IB works great, it's not just a mainstream usage case ok.

share|improve this answer
Hi, sorry the point was to NOT buy a switch! – grumpf Nov 8 '12 at 21:00
A point which you failed to include until now - in which case the answer is no, you can't do what you want. – Chopper3 Nov 8 '12 at 21:02
Why that? if I create 3 different network? I'm planning to use IP over infiniband ;) – grumpf Nov 8 '12 at 21:40
But how are they connected between each other? – Chopper3 Nov 8 '12 at 21:48
@grumpf Infiniband isn't IT speak for "magic technology that means I can ignore how things actually work". – RobM Nov 8 '12 at 22:44

Your question basically boils down to this:

I have 3 servers with Inifiniband. Can I create a network between them?

Which is analogous to

I have 3 servers with Inifiniband Ethernet. Can I create a network between them?

So the answer is yes. But then when you throw this into the mix:

Point is to NOT USE a switch!

The answer is no. You need a switch to get 2 or more of anything to work together. Even if you daisy chain them off each other (eww) then you still have a switch in the middle of the daisy chain. I'm not saying you can daisy chain them though, so don't take that as an assumption that you can.

share|improve this answer
Your old answer is actually wrong. A proper installation would use a switch. I guess he was trying to save several thousand on an IB switch. – Matt Mar 26 '15 at 1:34
Not sure what was wrong about my answer? I told him that he has to use a switch in the last paragraph. – Mark Henderson Mar 26 '15 at 1:40
No you could connect them in a ring configuration since each HBA has 2 ports. Becomes like a daisy chain. Since each other server is directly connected to another they would all be able to talk even if one link went down. He'd have to run a subnet manager on each link though. – Matt Mar 26 '15 at 1:44
Which I also mentioned - chaining them off each other. Never having used infiniband I wasn't sure if that was actually possible though, but then the nodes in the middle of the ring become a "switch" anyway. Maybe not a device sitting in a rack, but they still switch packets between nodes on the network. – Mark Henderson Mar 26 '15 at 1:46
@MarkHenderson If you've never used InfiniBand, why are you making up stuff? You do not need any switches. 3 nodes with 3 subnet managers can use 3 wires to connect directly to each other. Did you even read the question? – Navin Mar 25 at 9:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.