Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards

>lspci | grep -i infini
07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)

I'm interested in connecting them directly to each other bypassing an Infiniband switch (which I don't have). Quick googling showed that at least in some configurations it's possible.

I installed all RedHat Infiniband packages with yum groupinstall "Infiniband Support". However, ibv_devinfo shows that both ports in each card are down, which indicates that cables are not connected. But the cable is connected, although the LEDs are off on the cards (not a good sign). Another source of confusion for me is that according to this, RedHat doesn't come with OFED packages and I'm slightly hesitant to install them from source due to the lack of RedHat support for them... So where am I going with this? The questions I have are:

  1. is it possible to have a switchless/direct Infiniband connection between two servers the way I described above?
  2. If it's possible, do I have to use the OFED packages or can I configure everything with just the packages coming with RHEL.
  3. Why are the LEDs off on my servers even though the cable is connected?

Any additional input/advice/pointers would be appreciated.

P.S. I followed this guide for installation instructions. The Infiniband cards are clearly recognized by my OS and the rdma service is running.

Update: I have opensm installed. When I run it it says:

OpenSM 3.3.13
Command Line Arguments:
 Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.13

Entering DISCOVERING state

Using default GUID 0x1175000076e4c8
SM port is down

and stays at that point.

Update 2: I could not get my configuration to work with the RedHat packages. The OFED packages from openfabrics didn't work either. But I found official drivers from Intel (which aquired QLogic earlier this year) which kind of work. Kind of, because even when I get the cards to work and talk to each other, after reboot they never go back to PORT_ACTIVE state. Well, I'll probably post another question about it.

share|improve this question
    
Do you have a subnet manager (i.e. opensm) installed on either machine? –  rnxrx Nov 22 '12 at 22:33
    
Yes, I do. see update to the question. I don't know how to configure it though or if I have to do that at all. –  exfizik Nov 22 '12 at 22:53
    
Your SM is not running. You should see something like "Entering MASTER state SUBNET UP" at some point if it is working. Many possible config possibilities could be the problem. –  Albert Chu Nov 28 '12 at 17:47
    
SM was running, but it can't do anything when the port is down. –  exfizik Nov 29 '12 at 5:38

4 Answers 4

An old question but here's my answer.

1) Do you have the kernel driver loaded for the card? It should be ib_qib

try:

lsmod | grep ib_qib

If it's not loaded, configure it to load at boot in /etc/modules or whatever the redhat standard uses for this sorted of thing. (it's /etc/modules in debian based systems).

Also if you want to run IPoIB you'll need to load that module. This is my recommended minimum for your case.

ib_qib
rdma_ucm
ib_umad
ib_uverbs
ib_ipoib

2) Ensure the subnet manager is running. You need it running on both ports. Therefore you'll see 2 copies of the subnet manager running in the process list, one for each port.

share|improve this answer

Late response but this actually a known bug in the Intel driver.

Try running this after startup: 'ibportstate -D 0 1 enable'

share|improve this answer
  1. yes it is
  2. Yes the RHEL packages in 6.3 should work
  3. This is a good question

What does the output of "ibstat" show? Is the Physical state of the port down? The answer is probably "yes". I think you are having trouble with the cable or connection in some way. The port state in ibv_devinfo before running the SM should be PORT_INIT. PORT_DOWN is indicating you don't have link.

share|improve this answer
    
See update 2 above. The hardware part is fine. It's a driver issue. –  exfizik Nov 29 '12 at 5:38

You probably should try looking for drivers supplied by card's manufacturer - QLogic.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.