I have two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables.
I'm really concerned about latency which is higher than on 1Gbit ethernet connection using same hosts.
[root@localhost ~]# ibping -G 0x0008f104039a5589 Pong from host-a.(none) (Lid 3): time 0.238 ms Pong from host-a.(none) (Lid 3): time 0.291 ms Pong from host-a.(none) (Lid 3): time 0.320 ms Pong from host-a.(none) (Lid 3): time 0.290 ms Pong from host-a.(none) (Lid 3): time 0.335 ms Pong from host-a.(none) (Lid 3): time 0.281 ms
Most people are getting 0.040ms - 0.050ms using the same IB cards.
Cisco IB Switch and cards are running latest firmware.
I've tried connecting two hosts back-to-back eliminating the switch but latency is still 0.200ms+
Any ideas?