Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

Infiniband: ibping works but no ib0 interface

(redirected here from NetworkEngineering) I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
0
votes
1answer
158 views

Infiniband P_Keys and the Linux kernel

I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
0
votes
0answers
72 views

What is the use of Mellanox MT27800 Family ConnectX-5 virtual ports?

I have 100 Gb/s EDR two port card (ConnectX-5) in CentOS system. I have installed "InfiniBand Support" group package and have the opensm running in fabric. Driver creates four interfaces ib[0-3] ...
0
votes
0answers
20 views

QLogic 12300 :: Can't use more than 6 ports

I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
0
votes
0answers
17 views

Connecting Mellanox CX353A(QSFP+) to Mellanox MNPA-19-XTR(SFP+)

I Have attached a QSFP+ device to PCI slot in my server and using a QSFP+ to 4 SFP+ fanout cable. Is it possible to create connectivity between QSFP+ to SFP+ without using a switch?
0
votes
0answers
22 views

Are different brands of IB equipment compatible?

Is it possible to mix and match brands of qdr infiniband equipment? If for example I get a intel switch can i use an oracle or netapp qsfp+ cable with it?
0
votes
1answer
229 views

MPI can not find Infiniband interface point-to-point

I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors: [Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
0
votes
1answer
74 views

Can you mix 10gb and Infiniband for iScsi storage target

I'm setting up a Fedora server to act as an iScsi storage target. The primary storage network will be 25gb Infiniband. Is it possible and/or safe to also allow the target to be accessed by 10gb ...
2
votes
0answers
29 views

Can I use Ethernet transceivers on InfiniBand devices?

Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices? If not, what's the cause of such incompatibility?
-3
votes
1answer
126 views

How Infiniband manages to have 100Gbit/s

How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
0
votes
1answer
54 views

HP Infiniband HCA on Fujitsu Motherboard

I am building a 2 node cluster using node-to-node Infiniband connection. I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
7
votes
1answer
543 views

File-sharing over Infiniband on FreeBSD

I have a pair of servers, which I'd like to connect to each other directly -- rather than through the switch shared by the rest of the network. I expect to benefit from lower latency and higher ...
0
votes
2answers
239 views

CX4 Infiniband to SFP+ Converter

It may be a weird question, but I'm facing the following problem at the moment: I've got a DELL PowerConnect 6248 Switch (managed Gigabit-Ethernet) with the YY741-Stacking module in the back. The ...
0
votes
1answer
278 views

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
0
votes
2answers
118 views

Options for an ultra-fast small LAN [closed]

I need to transfer several terabytes/day between three windows computers. Two computers are employed to acquire medical imaging data (500 Gbytes/sample, ca. 6-7 samples/day), and the third is ...
4
votes
1answer
196 views

Set up Windows 10 SRP initiator

How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it. The initiator will be Windows ...
0
votes
1answer
53 views

Any way to segregate ports on a Mellanox Infiniband switch?

I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters. ...
0
votes
1answer
92 views

What is a 4x 10gbe QSFP-QSFP cable?

I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it ...
0
votes
1answer
74 views

Managed switch for Mellanox MCX354A-FCBT cards?

I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
1
vote
1answer
234 views

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] ...
1
vote
2answers
337 views

Dual-port RDMA Throughput Issue

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
0
votes
1answer
465 views

I want to connect to bridge using infiniband

I want to connect the physical port (ib0) to the bridge (bro). I am constructing kvm, but the physical port of infiniband to the virtual bridge I want to connect, but I could not do it well. Please ...
0
votes
1answer
2k views

How to set Infiniband port from INIT to ACTIVE

I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble. # ibnodes Ca : 0x0002c90300f2eef0 ...
0
votes
1answer
49 views

Ethernet switch with Infiniband uplink

There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP Are there any such switches with Infiniband uplinks? If not then why?
1
vote
1answer
168 views

Get firmware version for InfiniBand adapter from QLogic

How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server? ibstat displays an empty version number like so: Firmware version: Hardware version: 2 lspci -...
8
votes
1answer
2k views

IPoIB (IP over InfiniBand) vs. RDMA performance

I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it....
0
votes
0answers
22 views

Ideal configuration for a head node? [duplicate]

Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial ...
0
votes
1answer
265 views

If I replace an Infiniband switch should I restart opensm?

I have been using Mellanox ConnectX 2 QDR cards in a point to point link between two machines, with opensm running on one machine. This has worked very well. Recently I got a Mellanox IS5022 switch - ...
1
vote
2answers
446 views

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Good day, and my disclaimer that I am no IB expert and have not worked on IB directly yet. My question seems a trivial one, yet I could not find anything in google that answers it. Question: In ...
3
votes
2answers
2k views

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, ...
0
votes
1answer
742 views

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I ...
1
vote
1answer
314 views

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes. Oddly, while Ethernet packets seem to ...
1
vote
0answers
159 views

Error while creating an SRP target in container using targetcli

I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
2
votes
1answer
1k views

Replaced storage controller, now can't find ESXi ZFS mount point

We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the ...
1
vote
0answers
241 views

Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?

Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)? Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR? Same question goes for 40Gb ...
2
votes
1answer
87 views

How many infiniband adapters should be used in multi socket servers?

Should dual socket motherboards have an infinity band adapter for each CPU? That is, should there be two infiniband band adapters, one in each CPU's PCIe slot. Would this eliminate the signal going ...
2
votes
0answers
707 views

Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
2
votes
0answers
949 views

Connecting 3 Server with Infiniband without a switch

Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question. I got myself a dirt cheap ...
1
vote
1answer
223 views

ibping error: “can't serve class 50 on this port”

I try to run an ibping server on a machine with an infiniband card: ibping -S I get this error message: ibping: iberror: failed: can't serve class 50 on this port Any idea to solve the problem? ...
0
votes
1answer
159 views

Set a upper bandwidth limit for Infiniband HCAs

I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband. Putting those HCAs in FDR-...
1
vote
1answer
307 views

Infiniband switch, uplink

Regarding switches like Mellanox Voltaire VLT-30011 (QDR IB, managed). I see a gigabit ethernet port on the switch which in the manual (http://www.mellanox.com/related-docs/prod_ib_switch_systems/...
1
vote
1answer
895 views

In Infiniband networks, what is an MGID and how does it relate to multicast group?

I have this error in my Open Subnet manager log. Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ...
2
votes
4answers
866 views

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
1
vote
0answers
1k views

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ...
2
votes
2answers
784 views

Mellanox and Infiniband latency

I got two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables. I'm really concern about latency which is higher than on ...
0
votes
0answers
104 views

forwarding traffic between the ports

How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology. PC Listing GUIDs: Port 0: 0x2c903000f6d27 Port 1: 0x2c903000f6d28 PC Listing GUIDs: Port 0: ...
0
votes
1answer
1k views

VMware infiniband configuration

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup: The first card should work ...
4
votes
2answers
5k views

Infiniband Verifying that RDMA is working

I have two identical computers with Mellanox cards connected to each other through a cable. No switch. Using opensm. I have run several tests, including ping_pong tests, ibping, etc. They all seem to ...
0
votes
0answers
2k views

opensm CentOS 7 software switch

I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports. I have read the ...
7
votes
1answer
7k views

Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $...