I've recently upgraded my Infiniband HCAs from SDR to DDR, however upon upgrading, it has not improved the link speed at all. Receiving the following output from 'iblinkinfo':

CA: ip-san-1-au3a mthca0:
  0x0008f104039812b5      3    1[  ] ==( 4X           2.5 Gbps Active/  LinkUp)==>       4    1[  ] "sp-vmh-1a-au3a mthca0" ( Could be 5.0 Gbps)
CA: sp-vmh-1a-au3a mthca0:
  0x0008f10403981435      4    1[  ] ==( 4X           2.5 Gbps Active/  LinkUp)==>       3    1[  ] "ip-san-1-au3a mthca0" ( Could be 5.0 Gbps)

I've searched around for some answers but unable to get any closer to solving this. Has anyone seen this issue before? I am using a Belkin CX4 10M cable to link the two servers together which is apparently capable of SDR/DDR speeds, and running CentOS using the packaged Infiniband drivers.

Any advice or direction on the issue would be great! I'm hoping someone's seen this before and it's an easy answer... cheers!

  • 1
    How many lines does the PCIe slot have? Many PCIe slots are x8 physically and only x4 electrically. What kind of servers are the hosts? – Dmitri Chubarov Jan 4 '14 at 10:37
  • The HCAs are in PCI-E x8 slots for both machines, where both HCAs are connected directly (no switch involved). One machine is a Dual Xeon 5520 with 48GB RAM and the other is a Xeon E3-1230v2 with 16GB RAM. I will have a check shortly to make sure they are actually set to x8 when I can take them offline. – jaseeey Jan 5 '14 at 1:28
  • You do not need to take them offline. It is part of the spec of the motherboard. HCA driver messages in syslog might contain hints as well. – Dmitri Chubarov Jan 5 '14 at 7:27
  • The PCI-E card capacity is x8, but it is only running at x4 - reported by 'lspci', on one of the hosts. After checking the motherboard specifications, they are only PCI-E 2.0 x4 slots. I do have one PCI-E x8 slot, but it's currently being used by a RAID controller card. I might need to upgrade the server hardware before I can see some bandwidth improvements - the other host is OK. I didn't even think of this, thanks for helping answer the question! – jaseeey Jan 5 '14 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.