I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband.
Putting those HCAs in FDR-10 or QDR reduces too much the total bandwidth. Ideally I would like to limit each HCA to run between 5 and 5.5 GB/s. I'm not sure if a QoS managed by the IB switch could effectively set a upper limit per HCA (I think it rather uses weights for setting priorities).
Is there other ways to set a upper bandwidth limit for Infiniband HCAs?