8
votes
Accepted
Connect two infiniband cards to each other without a switch
Yes, you can directly connect them. But you must ensure that on at least one of the machines you run the opensm subnet manager.
From the software side, if you were using Ubuntu 14.04, install these ...
- 14.4k
6
votes
Accepted
IPoIB (IP over InfiniBand) vs. RDMA performance
have you seen these presentations?
https://www.linbit.com/en/drbd-9-over-rdma-with-micron-ssds/
http://downloads.openfabrics.org/Media/Monterey_2015/Tuesday/tuesday_09_ma.pdf
InfiniBand is just a ...
- 1,093
6
votes
How can I determine which switch the Infiniband subnet manager is running on?
If you have an OFED installed in your system and the package infiniband-diags you can use the command sminfo to list running subnet managers in the Infiniband network:
~ # sminfo
sminfo: sm ...
- 309
5
votes
Accepted
Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections
QSFP+ stands for Quad (4-channel) Small Form-Factor Pluggable Plus, also known as QSFP10. Now QSFP/QSFP+/QSFP28 all use 4 channels to pass data.
QSFP - People mistake this for QSFP+ ALL the time. ...
- 143
5
votes
Accepted
File-sharing over Infiniband on FreeBSD
I don't have answers for 1-3, only some additional information about Infiniband(IB) applications that may help you find the answer. I can answer 4 though.
1) What are my connectivity options? The ...
- 444
4
votes
Accepted
Dual-port RDMA Throughput Issue
I think the bottleneck here is the PCIe link between the ConnectX and the host. The ConnectX-3 has a Gen. 3 x8 PCIe connection, which is bound to a theoretical maximum of 63.04 Gbps (according to this ...
- 272
3
votes
Ethernet switch with Infiniband uplink
Ethernet and InfiniBand are two completely different and incompatible layer 2 protocols, for one. That's certainly sufficient. Don't confuse the physical media (layer 1) with the wire protocol (layer ...
- 242k
3
votes
Accepted
Can you mix 10gb and Infiniband for iScsi storage target
Possible? Yes. Reasonable? Maybe. Safe? Depends.
Ethernet and Infiniband are not compatible - but they can both run TCP/IP, so iSCSI on top can work.
The easiest solution is a multi-protocol router ...
- 9,477
3
votes
How Infiniband manages to have 100Gbit/s
I don't know much about Infiniband, but the PCIE 3.0 specification is says that a single lane is capable of sending up to 8GT/s. Therefore a 16x PCI 3.0 connection should be capable of 128GT/s. GT/...
- 106
3
votes
CX4 Infiniband to SFP+ Converter
The CX4 ports are "Infiniband style" but still Ethernet (10GBASE-CX4). Media converters for CX4 to SFP+ are (were) available, but they're costly if you can still get them. Swapping the port ...
- 9,477
3
votes
Accepted
Options for an ultra-fast small LAN
"Best" is a matter of requirement and budget.
100GbE switches are readily available but pricey - and as Michael has hinted, your data sources and sinks likely won't be able to keep up with 12 GB/s.
...
- 9,477
3
votes
Accepted
Infiniband cable
What is the difference between an active and a passive cable?
Passive cables only work to lengths of about 5m or under. Active cables boost the signal for increased distance. See this article
...
- 14.4k
3
votes
Accepted
Unable to get Mellanox ConnectX-3 to work with transceiver/cable
TL;DR Auto-negotiation was turned off on my switch and had manufacturer's recommend settings for a 40G connection. Turning auto-negotiation on solved the issue.
I want to answer my question with ...
2
votes
Accepted
ifconfig apparently showing wrong RX/TX values for InfiniBand HCA
You are correct with your assumptions. When running MPI over Infiniband, you normally want to bypass the network stack and use RDMA/Verbs interface to have full performance. All communication sent ...
- 4,205
2
votes
Infiniband Verifying that RDMA is working
It turns out this has been seen before. I don't like the answer because it seems to sweep it under the rug, but it is an answer nontheless:
http://linuxtoolkit.blogspot.com/2013/01/errors-when-...
- 299
2
votes
Accepted
If I replace an Infiniband switch should I restart opensm?
In short - no, you don't ever have to restart OpenSM.
If you have OpenSM running with "normal" configuration (that is, you didn't tweak it with some parameters to disable some of its capabilities), ...
- 154
2
votes
Accepted
In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?
Absolutely, you can have IPoIB and RDMA on the same InfiniBand port.
Regarding prioritization, InfiniBand can support multiple traffic classes. You can find information in this document: http://www....
- 272
2
votes
Set up Windows 10 SRP initiator
StarWind has Windows iSER and NVMeoF initiators for sure, but I don't know are they public ones, or proprietary user land code using inside their target products. It worth contacting them directly and ...
- 778
2
votes
Accepted
What is a 4x 10gbe QSFP-QSFP cable?
QSFP means, literally, "Quad SFP" connector. In other words, it is a single SFP connector that carries four link.
From wikipedia:
The Quad Small Form-factor Pluggable (QSFP) is a compact,
hot-...
- 46.3k
2
votes
CX4 Infiniband to SFP+ Converter
There are two Dell part numbers for stacking modules (UY108, YY741) plus the CX4 module (GM765) but only the latter is supposed to work in Bay 2. The switch can configure any module as Ethernet or ...
- 21
2
votes
I want to connect to bridge using infiniband
In general, it is not possible to associate an Ethernet bridge with an IPoIB interface. But why do you want the IPoIB interface to be on the bridge in the first place? If you are interested in using ...
- 272
2
votes
Accepted
Infiniband drivers : OFED or distro included?
By not using the vendor OFED distribution, in this case Mellanox OFED you should expect not only a performance penalty but lack of features and a lot of stability issues.
Infiniband is not rock solid ...
- 5,480
2
votes
Infiniband drivers : OFED or distro included?
The "inbox" drivers have gone through the Linux Q&A process and through the distro Q&A. The MOFED drivers have not.
There are severe bugs in MOFED that prevent our code from running ...
2
votes
Accepted
Latency of memory accesses via interconnectors
Your single node numbers of 90 ns local vs 370 ns other socket seems reasonable. However, I think the 600 ns of Infiniband is supposed to be end to end, through a switch to a different frame.
600 ns ...
- 31.4k
1
vote
Accepted
1
vote
Dual-port RDMA Throughput Issue
I have (2) systems each with a Mellanox FDR MCX354A-FCBT CX354A (ConnectX-3 VPI.) Only having the (2) using InfiniBand, I don't have a switch and just have them directly connected. I'm running dual ...
- 131
1
vote
Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections
Additionally, QSFP modules are commonly available in several different types: 4x1G QSFP, 4x10G QSFP+, 4x28G QSFP28. From this side, QSFP/QSFP+/QSFP28 all share the same small form-factor. Literally ...
- 21
1
vote
Infiniband port status UP but can't open UMAD port ((null):0)
It maybe just a typo here on SO, but you are specifying LID as 10x22. As LID is supposed to be a hexadecimal number, the 1 is extraneous. It should be just a 0x22.
- 3,947
1
vote
Replaced storage controller, now can't find ESXi ZFS mount point
This is so little information to go on, but I'll try to edit this answer as needed...
Please show the output of:
zpool list
zpool status -v
zfs list
And possibly http://pastebin.com the zpool ...
- 197k
1
vote
How many infiniband adapters should be used in multi socket servers?
That would be the optimal configuration, but it also depends on the applications that make use of the Infiniband connections and you would have to take care that those applications and their processes ...
- 4,205
Only top scored, non community-wiki answers of a minimum length are eligible