6

I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards

>lspci | grep -i infini
07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)

I'm interested in connecting them directly to each other bypassing an Infiniband switch (which I don't have). Quick googling showed that at least in some configurations it's possible.

I installed all RedHat Infiniband packages with yum groupinstall "Infiniband Support". However, ibv_devinfo shows that both ports in each card are down, which indicates that cables are not connected. But the cable is connected, although the LEDs are off on the cards (not a good sign). Another source of confusion for me is that according to this, RedHat doesn't come with OFED packages and I'm slightly hesitant to install them from source due to the lack of RedHat support for them... So where am I going with this? The questions I have are:

  1. is it possible to have a switchless/direct Infiniband connection between two servers the way I described above?
  2. If it's possible, do I have to use the OFED packages or can I configure everything with just the packages coming with RHEL.
  3. Why are the LEDs off on my servers even though the cable is connected?

Any additional input/advice/pointers would be appreciated.

P.S. I followed this guide for installation instructions. The Infiniband cards are clearly recognized by my OS and the rdma service is running.

Update: I have opensm installed. When I run it it says:

OpenSM 3.3.13
Command Line Arguments:
 Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.13

Entering DISCOVERING state

Using default GUID 0x1175000076e4c8
SM port is down

and stays at that point.

Update 2: I could not get my configuration to work with the RedHat packages. The OFED packages from openfabrics didn't work either. But I found official drivers from Intel (which aquired QLogic earlier this year) which kind of work. Kind of, because even when I get the cards to work and talk to each other, after reboot they never go back to PORT_ACTIVE state. Well, I'll probably post another question about it.

4
  • Do you have a subnet manager (i.e. opensm) installed on either machine?
    – rnxrx
    Nov 22, 2012 at 22:33
  • Yes, I do. see update to the question. I don't know how to configure it though or if I have to do that at all.
    – exfizik
    Nov 22, 2012 at 22:53
  • Your SM is not running. You should see something like "Entering MASTER state SUBNET UP" at some point if it is working. Many possible config possibilities could be the problem.
    – Albert Chu
    Nov 28, 2012 at 17:47
  • SM was running, but it can't do anything when the port is down.
    – exfizik
    Nov 29, 2012 at 5:38

4 Answers 4

4

Late response but this actually a known bug in the Intel driver.

Try running this after startup:

ibportstate -D 0 1 enable
2

You probably should try looking for drivers supplied by card's manufacturer - QLogic.

1
  1. yes it is
  2. Yes the RHEL packages in 6.3 should work
  3. This is a good question

What does the output of "ibstat" show? Is the Physical state of the port down? The answer is probably "yes". I think you are having trouble with the cable or connection in some way. The port state in ibv_devinfo before running the SM should be PORT_INIT. PORT_DOWN is indicating you don't have link.

1
  • See update 2 above. The hardware part is fine. It's a driver issue.
    – exfizik
    Nov 29, 2012 at 5:38
0

An old question but here's my answer.

1) Do you have the kernel driver loaded for the card? It should be ib_qib

try:

lsmod | grep ib_qib

If it's not loaded, configure it to load at boot in /etc/modules or whatever the redhat standard uses for this sorted of thing. (it's /etc/modules in debian based systems).

Also if you want to run IPoIB you'll need to load that module. This is my recommended minimum for your case.

ib_qib
rdma_ucm
ib_umad
ib_uverbs
ib_ipoib

2) Ensure the subnet manager is running. You need it running on both ports. Therefore you'll see 2 copies of the subnet manager running in the process list, one for each port.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .