5

I have several machines with ConnectX-7 Infiniband cards and they're plugged into an Nvidia QM9700 switch. I've confirmed 400 Gbit NDR at both ends (ibstat on the host and in the console on the switch). The machines are running Ubuntu 22.04 and the Mellanox 5.8-3.0.7.0 drivers. I've done a lot of testing with ib_write_bw and the most I can get is ~251 Gbit/s. The actual test commands are:

Server side (host_a):

numactl -N 0 -m 0 ib_write_bw -d mlx5_4 -F --report_gbits

Client side (host_b):

numactl -N 0 -m 0 ib_write_bw -d mlx5_4 -F --report_gbits --run_infinitely host_b

The cards are in the correct numa domains to match numactl, but I've tried other combinations of that with no luck. Output ends up looking something like this:

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_4
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x54 QPN 0x0058 PSN xxx RKey 0x1820e0 VAddr xxx
 remote address: LID 0x53 QPN 0x0058 PSN xxx RKey 0x1820e0 VAddr xxx
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      2353827          0.00               246.81             0.470754
 65536      2339084          0.00               245.27             0.467815
 65536      2338736          0.00               245.23             0.467746
 65536      2338574          0.00               245.22             0.467713
 65536      2338610          0.00               245.22             0.467720

I know this is probably a long shot, but wondering if anyone has actually achieved 400 gbit over infiniband with ib_write_bw that might know something we missed.

4
  • To get full bandwidth at higher speeds, you have to worry about how much PCIe bandwidth you have between the two cards, how many lanes of PCIe, how fast, how fast the Northbridge or PCIe switch can pass data, where the interrupts are going, are the cards both on the same physical CPU, how much other data is on the bus from other cards and the OS, whether your memory bandwidth can sustain this, etc. In my experience, many things start to get in the way. First thing I would do is bigger buffers, in ib_write_bw, maybe more than 1 Mbyte.
    – edt11x
    Sep 14, 2023 at 0:18
  • 1
    This may sound stupid but let me ask. 251Gbps is pretty close to PCI-Ex 3.0 theoretical performance, which if I'm not wrong is 256Gbps. Are you using this cards in which hardware? Does the slot is running at least with PCI-Ex 4.0? Sep 14, 2023 at 3:51
  • can you show lspci result?
    – jshen28
    Sep 18, 2023 at 7:48
  • Can you please share the information about modules and cables you are using? Thanks a lot. I am using the same testing command in your post for two Connectx-7 cards. At first, BW average is about 391Gb/sec, but BW average decreases to 238Gb/sec very quickly. Oct 25, 2023 at 6:54

1 Answer 1

3

So the answer ended up being that we needed to set the PCI parameter MAX_ACC_OUT_READ to 128. Once that was set via mlxconfig -y -d mlx5_4 s MAX_ACC_OUT_READ=128 for each card and then power cycling the machines, throughput jumped from ~250 gbit to ~375 Gbit. Not 400 but I'll take it. To do each card:

apt-get install mlnx-ofed-all
mst start
for i in `mst status -v | grep 'net-ibp' | awk '{print $3}'` ; do mlxconfig -y -d $i s MAX_ACC_OUT_READ=128  ; done

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .