Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

Filter by
Sorted by
Tagged with
0 votes
0 answers
76 views

Infiniband adapter down

edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2: I am not able to get my Infiniband adapter working. The output of ibstat states that it is down: CA 'mlx5_0' ...
user avatar
  • 31
2 votes
0 answers
60 views

Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using: Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s Description of ...
user avatar
  • 121
1 vote
1 answer
53 views

Infiniband OpenSM N-to-N port routing configuration

I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch. My RDMA application runs as one process per NUMA ...
user avatar
1 vote
1 answer
2k views

Unable to get Mellanox ConnectX-3 to work with transceiver/cable

I'm attempting to upgrade my local network setup to 40G but I seem to have hit a snag. I have Mellanox ConnectX-3 VPI CX353A network cards I got off ebay, updated their firmware and installed their ...
user avatar
0 votes
1 answer
142 views

Latency of memory accesses via interconnectors

I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency. I have obtained some benchmarks regarding one of the hardware ...
user avatar
  • 23
2 votes
1 answer
79 views

How to access files from a local server with infiniband without using IP

I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me. I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are ...
user avatar
  • 23
1 vote
0 answers
11 views

what is intel omnipath topology error?

I have lustre setup and connected with intel omnipath. the metadata server is getting rebooted often. in logs, i can see the opa topology error. what is this error means?. will it cause the server to ...
user avatar
1 vote
2 answers
265 views

Infiniband drivers : OFED or distro included?

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome ! We are currently using Mellanox OFED drivers, but our infiniband ...
user avatar
1 vote
1 answer
454 views

revive an old QLogic IBA7322 Infiniband card on CentOS8.2

I have a few HP Blades Gen7 equipped with QLogic Infiniband cards IBA7322 which I would like to use with CentOS 8. The problem I have is that I cannot find the right drivers for them. All the ...
user avatar
  • 131
1 vote
0 answers
229 views

Infiniband: ibping works but no ib0 interface

(redirected here from NetworkEngineering) I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
user avatar
0 votes
1 answer
615 views

Infiniband P_Keys and the Linux kernel

I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
user avatar
  • 3,434
0 votes
0 answers
27 views

QLogic 12300 :: Can't use more than 6 ports

I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
user avatar
0 votes
1 answer
798 views

MPI can not find Infiniband interface point-to-point

I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors: [Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
user avatar
0 votes
1 answer
118 views

Can you mix 10gb and Infiniband for iScsi storage target

I'm setting up a Fedora server to act as an iScsi storage target. The primary storage network will be 25gb Infiniband. Is it possible and/or safe to also allow the target to be accessed by 10gb ...
user avatar
2 votes
0 answers
43 views

Can I use Ethernet transceivers on InfiniBand devices?

Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices? If not, what's the cause of such incompatibility?
user avatar
  • 121
-2 votes
1 answer
221 views

How Infiniband manages to have 100Gbit/s

How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
user avatar
0 votes
1 answer
68 views

HP Infiniband HCA on Fujitsu Motherboard

I am building a 2 node cluster using node-to-node Infiniband connection. I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
user avatar
8 votes
1 answer
1k views

File-sharing over Infiniband on FreeBSD

I have a pair of servers, which I'd like to connect to each other directly -- rather than through the switch shared by the rest of the network. I expect to benefit from lower latency and higher ...
user avatar
  • 2,232
1 vote
2 answers
536 views

CX4 Infiniband to SFP+ Converter

It may be a weird question, but I'm facing the following problem at the moment: I've got a DELL PowerConnect 6248 Switch (managed Gigabit-Ethernet) with the YY741-Stacking module in the back. The ...
user avatar
  • 193
0 votes
1 answer
509 views

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
user avatar
  • 1
1 vote
2 answers
134 views

Options for an ultra-fast small LAN [closed]

I need to transfer several terabytes/day between three windows computers. Two computers are employed to acquire medical imaging data (500 Gbytes/sample, ca. 6-7 samples/day), and the third is ...
user avatar
  • 397
4 votes
1 answer
376 views

Set up Windows 10 SRP initiator

How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it. The initiator will be Windows ...
user avatar
0 votes
1 answer
100 views

Any way to segregate ports on a Mellanox Infiniband switch?

I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters. ...
user avatar
  • 3,434
0 votes
1 answer
119 views

What is a 4x 10gbe QSFP-QSFP cable?

I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it ...
user avatar
0 votes
1 answer
120 views

Managed switch for Mellanox MCX354A-FCBT cards?

I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
user avatar
  • 3,434
1 vote
1 answer
581 views

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] ...
user avatar
  • 185
2 votes
2 answers
545 views

Dual-port RDMA Throughput Issue

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
user avatar
0 votes
1 answer
724 views

I want to connect to bridge using infiniband

I want to connect the physical port (ib0) to the bridge (bro). I am constructing kvm, but the physical port of infiniband to the virtual bridge I want to connect, but I could not do it well. Please ...
user avatar
  • 11
0 votes
1 answer
3k views

How to set Infiniband port from INIT to ACTIVE

I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble. # ibnodes Ca : 0x0002c90300f2eef0 ...
user avatar
  • 151
0 votes
1 answer
78 views

Ethernet switch with Infiniband uplink

There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP Are there any such switches with Infiniband uplinks? If not then why?
user avatar
  • 198
1 vote
1 answer
258 views

Get firmware version for InfiniBand adapter from QLogic

How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server? ibstat displays an empty version number like so: Firmware version: Hardware version: 2 lspci -...
user avatar
  • 335
8 votes
1 answer
2k views

IPoIB (IP over InfiniBand) vs. RDMA performance

I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it....
user avatar
  • 553
0 votes
0 answers
22 views

Ideal configuration for a head node? [duplicate]

Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial ...
user avatar
0 votes
1 answer
388 views

If I replace an Infiniband switch should I restart opensm?

I have been using Mellanox ConnectX 2 QDR cards in a point to point link between two machines, with opensm running on one machine. This has worked very well. Recently I got a Mellanox IS5022 switch - ...
user avatar
  • 113
1 vote
2 answers
590 views

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Good day, and my disclaimer that I am no IB expert and have not worked on IB directly yet. My question seems a trivial one, yet I could not find anything in google that answers it. Question: In ...
user avatar
  • 21
4 votes
2 answers
3k views

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, ...
user avatar
  • 143
0 votes
1 answer
900 views

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I ...
user avatar
  • 1
1 vote
1 answer
489 views

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes. Oddly, while Ethernet packets seem to ...
user avatar
  • 133
1 vote
0 answers
207 views

Error while creating an SRP target in container using targetcli

I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
user avatar
  • 131
2 votes
1 answer
2k views

Replaced storage controller, now can't find ESXi ZFS mount point

We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the ...
user avatar
1 vote
0 answers
245 views

Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?

Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)? Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR? Same question goes for 40Gb ...
user avatar
2 votes
1 answer
103 views

How many infiniband adapters should be used in multi socket servers?

Should dual socket motherboards have an infinity band adapter for each CPU? That is, should there be two infiniband band adapters, one in each CPU's PCIe slot. Would this eliminate the signal going ...
user avatar
3 votes
0 answers
1k views

Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
user avatar
2 votes
0 answers
1k views

Connecting 3 Server with Infiniband without a switch

Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question. I got myself a dirt cheap ...
user avatar
1 vote
1 answer
264 views

ibping error: “can't serve class 50 on this port”

I try to run an ibping server on a machine with an infiniband card: ibping -S I get this error message: ibping: iberror: failed: can't serve class 50 on this port Any idea to solve the problem? ...
user avatar
0 votes
1 answer
255 views

Set a upper bandwidth limit for Infiniband HCAs

I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband. Putting those HCAs in FDR-...
user avatar
  • 101
1 vote
1 answer
358 views

Infiniband switch, uplink

Regarding switches like Mellanox Voltaire VLT-30011 (QDR IB, managed). I see a gigabit ethernet port on the switch which in the manual (http://www.mellanox.com/related-docs/prod_ib_switch_systems/...
user avatar
  • 199
1 vote
1 answer
1k views

In Infiniband networks, what is an MGID and how does it relate to multicast group?

I have this error in my Open Subnet manager log. Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ...
user avatar
  • 14k
2 votes
4 answers
1k views

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
user avatar
  • 23
1 vote
0 answers
1k views

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ...
user avatar
  • 11