Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Let's say I have 3 different servers, each one with an infiniband card. Each card has 2 different ports. (I don't know about the model yet)

Is it possible to create 3 different networks and to allow the 3 servers to communicate with each other without any problems? (and any spof).

I guess I just have to setup the /etc/hosts correctly.

I really don't know about infiniband, so please help me :)

Thanks in advance.

EDIT: Point is to NOT USE a switch!

share|improve this question
1  
InfiniBand fabric does not use IP, therefore /etc/hosts thing does not apply. Yet you can run IP on top of InfiniBand. –  Dmitri Chubarov Nov 8 '12 at 20:07
    
good place to ask these questions openfabrics.org/… –  tony roth Nov 8 '12 at 20:22
1  
Dmitri, IB supports IP encapsulation - in fact it's used quite extensively in at least two very large CGI renderfarms I'm aware of. Though I'll admit it's really not the lowest cost IP solution :) –  Chopper3 Nov 8 '12 at 20:33
1  
Why do you want to do this? –  ewwhite Nov 8 '12 at 21:05
    
Buy two cards per system? Build a triangle? –  Zoredache Nov 8 '12 at 22:52
show 1 more comment

closed as not a real question by Ward, Scott Pack, Brent Pabst, mdpc, HopelessN00b Genius of network Nov 9 '12 at 19:41

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

3 Answers

You're just talking about doing a point-to-point connection from each host to each of the other two right?

I've not tried it, but from what I have heard you can do point-to-point connections using IB without using a switch (just as you can with ethernet).

Problems might come in with the subnet manager(s) - I'm not sure if each node's subnet manager can run actively for one of the connected networks and backup for the other. You may need to run two subnet managers per box and bind each to a specific port. Not sure how to do that either but definitely worth researching that side of it more.

And obviously having a couple of switches would make things easier (and more scalable).

share|improve this answer
add comment

Your question basically boils down to this:

I have 3 servers with Inifiniband. Can I create a network between them?

Which is analogous to

I have 3 servers with Inifiniband Ethernet. Can I create a network between them?

So the answer is yes. But then when you throw this into the mix:

Point is to NOT USE a switch!

The answer is no. You need a switch to get 2 or more of anything to work together. Even if you daisy chain them off each other (eww) then you still have a switch in the middle of the daisy chain. I'm not saying you can daisy chain them though, so don't take that as an assumption that you can.

share|improve this answer
add comment

Yes, just buy one or two (for resilience) IB switches. That said you don't mention which protocols you want to use, though it seems like you might want to run IP over IB, which works - and editing hosts (or having a proper DNS environment) will help to make that work but IB is really designed for IB-specific protocols, IP over IB works great, it's not just a mainstream usage case ok.

share|improve this answer
    
Hi, sorry the point was to NOT buy a switch! –  grumpf Nov 8 '12 at 21:00
6  
A point which you failed to include until now - in which case the answer is no, you can't do what you want. –  Chopper3 Nov 8 '12 at 21:02
    
Why that? if I create 3 different network? I'm planning to use IP over infiniband ;) –  grumpf Nov 8 '12 at 21:40
3  
But how are they connected between each other? –  Chopper3 Nov 8 '12 at 21:48
1  
@grumpf Infiniband isn't IT speak for "magic technology that means I can ignore how things actually work". –  RobM Nov 8 '12 at 22:44
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.