Skip to main content
Share Your Experience: Take the 2024 Developer Survey
6 votes
Accepted

IPoIB (IP over InfiniBand) vs. RDMA performance

have you seen these presentations? https://www.linbit.com/en/drbd-9-over-rdma-with-micron-ssds/ http://downloads.openfabrics.org/Media/Monterey_2015/Tuesday/tuesday_09_ma.pdf InfiniBand is just a ...
Tux_DEV_NULL's user avatar
  • 1,103
5 votes
Accepted

File-sharing over Infiniband on FreeBSD

I don't have answers for 1-3, only some additional information about Infiniband(IB) applications that may help you find the answer. I can answer 4 though. 1) What are my connectivity options? The ...
Paul's user avatar
  • 452
5 votes
Accepted

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

QSFP+ stands for Quad (4-channel) Small Form-Factor Pluggable Plus, also known as QSFP10. Now QSFP/QSFP+/QSFP28 all use 4 channels to pass data. QSFP - People mistake this for QSFP+ ALL the time. ...
KrisGT's user avatar
  • 143
4 votes
Accepted

Dual-port RDMA Throughput Issue

I think the bottleneck here is the PCIe link between the ConnectX and the host. The ConnectX-3 has a Gen. 3 x8 PCIe connection, which is bound to a theoretical maximum of 63.04 Gbps (according to this ...
haggai_e's user avatar
  • 272
3 votes
Accepted

Unable to get Mellanox ConnectX-3 to work with transceiver/cable

TL;DR Auto-negotiation was turned off on my switch and had manufacturer's recommend settings for a 40G connection. Turning auto-negotiation on solved the issue. I want to answer my question with ...
ActualExplosives's user avatar
3 votes
Accepted

Latency of memory accesses via interconnectors

Your single node numbers of 90 ns local vs 370 ns other socket seems reasonable. However, I think the 600 ns of Infiniband is supposed to be end to end, through a switch to a different frame. 600 ns ...
John Mahowald's user avatar
3 votes
Accepted

Can you mix 10gb and Infiniband for iScsi storage target

Possible? Yes. Reasonable? Maybe. Safe? Depends. Ethernet and Infiniband are not compatible - but they can both run TCP/IP, so iSCSI on top can work. The easiest solution is a multi-protocol router ...
Zac67's user avatar
  • 11.9k
3 votes

How Infiniband manages to have 100Gbit/s

I don't know much about Infiniband, but the PCIE 3.0 specification is says that a single lane is capable of sending up to 8GT/s. Therefore a 16x PCI 3.0 connection should be capable of 128GT/s. GT/...
Nate Moseman's user avatar
3 votes

CX4 Infiniband to SFP+ Converter

The CX4 ports are "Infiniband style" but still Ethernet (10GBASE-CX4). Media converters for CX4 to SFP+ are (were) available, but they're costly if you can still get them. Swapping the port ...
Zac67's user avatar
  • 11.9k
3 votes
Accepted

Options for an ultra-fast small LAN

"Best" is a matter of requirement and budget. 100GbE switches are readily available but pricey - and as Michael has hinted, your data sources and sinks likely won't be able to keep up with 12 GB/s. ...
Zac67's user avatar
  • 11.9k
3 votes

Ethernet switch with Infiniband uplink

Ethernet and InfiniBand are two completely different and incompatible layer 2 protocols, for one. That's certainly sufficient. Don't confuse the physical media (layer 1) with the wire protocol (layer ...
Michael Hampton's user avatar
3 votes

Infiniband + KVM

NVidia (Mellanox), the main proponent of Infiniband, has good support for Infiniband virtualization. With NVidia ConnectX cards and SR-IOV support, this works with KVM under Redhat, Centos, Debian and ...
edt11x's user avatar
  • 131
3 votes
Accepted

Mellanox ConnectX-7 - Unable to get more than 250 Gbit/s

So the answer ended up being that we needed to set the PCI parameter MAX_ACC_OUT_READ to 128. Once that was set via mlxconfig -y -d mlx5_4 s MAX_ACC_OUT_READ=128 for each card and then power cycling ...
Evan's user avatar
  • 497
2 votes
Accepted

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

You are correct with your assumptions. When running MPI over Infiniband, you normally want to bypass the network stack and use RDMA/Verbs interface to have full performance. All communication sent ...
Thomas's user avatar
  • 4,305
2 votes
Accepted

If I replace an Infiniband switch should I restart opensm?

In short - no, you don't ever have to restart OpenSM. If you have OpenSM running with "normal" configuration (that is, you didn't tweak it with some parameters to disable some of its capabilities), ...
kliteyn's user avatar
  • 154
2 votes
Accepted

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Absolutely, you can have IPoIB and RDMA on the same InfiniBand port. Regarding prioritization, InfiniBand can support multiple traffic classes. You can find information in this document: http://www....
haggai_e's user avatar
  • 272
2 votes

I want to connect to bridge using infiniband

In general, it is not possible to associate an Ethernet bridge with an IPoIB interface. But why do you want the IPoIB interface to be on the bridge in the first place? If you are interested in using ...
haggai_e's user avatar
  • 272
2 votes

Set up Windows 10 SRP initiator

StarWind has Windows iSER and NVMeoF initiators for sure, but I don't know are they public ones, or proprietary user land code using inside their target products. It worth contacting them directly and ...
NISMO1968's user avatar
  • 1,060
2 votes
Accepted

What is a 4x 10gbe QSFP-QSFP cable?

QSFP means, literally, "Quad SFP" connector. In other words, it is a single SFP connector that carries four link. From wikipedia: The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-...
shodanshok's user avatar
  • 50.9k
2 votes

CX4 Infiniband to SFP+ Converter

There are two Dell part numbers for stacking modules (UY108, YY741) plus the CX4 module (GM765) but only the latter is supposed to work in Bay 2. The switch can configure any module as Ethernet or ...
Rampage_Rick's user avatar
2 votes

Infiniband drivers : OFED or distro included?

The "inbox" drivers have gone through the Linux Q&A process and through the distro Q&A. The MOFED drivers have not. There are severe bugs in MOFED that prevent our code from running ...
Robert van Aven's user avatar
2 votes
Accepted

Infiniband drivers : OFED or distro included?

By not using the vendor OFED distribution, in this case Mellanox OFED you should expect not only a performance penalty but lack of features and a lot of stability issues. Infiniband is not rock solid ...
Vinícius Ferrão's user avatar
1 vote
Accepted

How can I use nmap against an IPoIB network?

I've figured out. -sT flag did the trick, but for reasons that I still don't understand it only works with -P0 or -Pn. Since I'm only caring for TCP services, in this case, -sT was sufficient. Also it'...
Vinícius Ferrão's user avatar
1 vote
Accepted

Speeding up SAS data rate from 12Gbps

SAS doesn't use a bus like Parallel SCSI. SAS uses dedicated point-to-point links. Each link negotiates its own link rate. An 8-port 12G SAS RAID controller or HBA has an external, aggregated ...
Zac67's user avatar
  • 11.9k
1 vote

Infiniband OpenSM N-to-N port routing configuration

https://community.mellanox.com/s/question/0D51T00006RVtlU/rdmacm-connection-setup-issues For the record, the issue has been largely resolved by ensuring the ibacm service (Infiniband Assistant ...
Hugo Maxwell's user avatar
1 vote
Accepted

How to access files from a local server with infiniband without using IP

tl;dr: look and stick with NFSoRDMA. That's some points that needs to be clear. Considering that you're using RHEL 7 as basis you may want to serve files over NFS from s1 to s2, s3 and s4. Your right ...
Vinícius Ferrão's user avatar
1 vote

Dual-port RDMA Throughput Issue

I have (2) systems each with a Mellanox FDR MCX354A-FCBT CX354A (ConnectX-3 VPI.) Only having the (2) using InfiniBand, I don't have a switch and just have them directly connected. I'm running dual ...
user1902689's user avatar
1 vote
Accepted

How to set Infiniband port from INIT to ACTIVE

A reboot of gauss7 solved the issue.
Shibumi's user avatar
  • 171
1 vote

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

Additionally, QSFP modules are commonly available in several different types: 4x1G QSFP, 4x10G QSFP+, 4x28G QSFP28. From this side, QSFP/QSFP+/QSFP28 all share the same small form-factor. Literally ...
Ableson's user avatar
  • 31
1 vote

Replaced storage controller, now can't find ESXi ZFS mount point

This is so little information to go on, but I'll try to edit this answer as needed... Please show the output of: zpool list zpool status -v zfs list And possibly http://pastebin.com the zpool ...
ewwhite's user avatar
  • 198k

Only top scored, non community-wiki answers of a minimum length are eligible