Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

0
votes
1answer
20 views

HP Infiniband HCA on Fujitsu Motherboard

I am building a 2 node cluster using node-to-node Infiniband connection. I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
4
votes
1answer
82 views

File-sharing over Infiniband on FreeBSD

I have a pair of servers, which I'd like to connect to each other directly -- rather than through the switch shared by the rest of the network. I expect to benefit from lower latency and higher ...
0
votes
1answer
19 views

CX4 Infiniband to SFP+ Converter

It may be a weird question, but I'm facing the following problem at the moment: I've got a DELL PowerConnect 6248 Switch (managed Gigabit-Ethernet) with the YY741-Stacking module in the back. The ...
0
votes
1answer
37 views

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
0
votes
2answers
94 views

Options for an ultra-fast small LAN [closed]

I need to transfer several terabytes/day between three windows computers. Two computers are employed to acquire medical imaging data (500 Gbytes/sample, ca. 6-7 samples/day), and the third is ...
3
votes
1answer
47 views

Set up Windows 10 SRP initiator

How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it. The initiator will be Windows ...
0
votes
1answer
34 views

Any way to segregate ports on a Mellanox Infiniband switch?

I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters. ...
0
votes
1answer
43 views

What is a 4x 10gbe QSFP-QSFP cable?

I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it ...
0
votes
1answer
20 views

Managed switch for Mellanox MCX354A-FCBT cards?

I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
1
vote
1answer
107 views

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] ...
1
vote
2answers
127 views

Dual-port RDMA Throughput Issue

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
0
votes
1answer
170 views

I want to connect to bridge using infiniband

I want to connect the physical port (ib0) to the bridge (bro). I am constructing kvm, but the physical port of infiniband to the virtual bridge I want to connect, but I could not do it well. Please ...
0
votes
1answer
398 views

How to set Infiniband port from INIT to ACTIVE

I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble. # ibnodes Ca : 0x0002c90300f2eef0 ...
0
votes
1answer
38 views

Ethernet switch with Infiniband uplink

There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP Are there any such switches with Infiniband uplinks? If not then why?
1
vote
1answer
81 views

Get firmware version for InfiniBand adapter from QLogic

How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server? ibstat displays an empty version number like so: Firmware version: Hardware version: 2 lspci -...
7
votes
1answer
765 views

IPoIB (IP over InfiniBand) vs. RDMA performance

I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it....
0
votes
0answers
21 views

Ideal configuration for a head node? [duplicate]

Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial ...
0
votes
1answer
133 views

If I replace an Infiniband switch should I restart opensm?

I have been using Mellanox ConnectX 2 QDR cards in a point to point link between two machines, with opensm running on one machine. This has worked very well. Recently I got a Mellanox IS5022 switch - ...
1
vote
2answers
224 views

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Good day, and my disclaimer that I am no IB expert and have not worked on IB directly yet. My question seems a trivial one, yet I could not find anything in google that answers it. Question: In ...
3
votes
2answers
736 views

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, ...
0
votes
1answer
502 views

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I ...
1
vote
1answer
178 views

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes. Oddly, while Ethernet packets seem to ...
1
vote
0answers
104 views

Error while creating an SRP target in container using targetcli

I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
2
votes
1answer
1k views

Replaced storage controller, now can't find ESXi ZFS mount point

We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the ...
1
vote
0answers
232 views

Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?

Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)? Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR? Same question goes for 40Gb ...
2
votes
1answer
78 views

How many infiniband adapters should be used in multi socket servers?

Should dual socket motherboards have an infinity band adapter for each CPU? That is, should there be two infiniband band adapters, one in each CPU's PCIe slot. Would this eliminate the signal going ...
1
vote
0answers
467 views

Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
0
votes
0answers
251 views

ibportstate: How do I reset/ensure HCA LinkUp rates operating for card specs?

How do I ensure the rate on my HCAs via ibportstate (or other)? I believe I've inadvertently 'mangled' one of my HCAs from achieving 20 Gbps/4x DDR. I have two machines, HPV00 & HPV01, ...
1
vote
0answers
670 views

Connecting 3 Server with Infiniband without a switch

Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question. I got myself a dirt cheap ...
1
vote
1answer
160 views

ibping error: “can't serve class 50 on this port”

I try to run an ibping server on a machine with an infiniband card: ibping -S I get this error message: ibping: iberror: failed: can't serve class 50 on this port Any idea to solve the problem? ...
0
votes
1answer
117 views

Set a upper bandwidth limit for Infiniband HCAs

I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband. Putting those HCAs in FDR-...
1
vote
1answer
258 views

Infiniband switch, uplink

Regarding switches like Mellanox Voltaire VLT-30011 (QDR IB, managed). I see a gigabit ethernet port on the switch which in the manual (http://www.mellanox.com/related-docs/prod_ib_switch_systems/...
1
vote
1answer
734 views

In Infiniband networks, what is an MGID and how does it relate to multicast group?

I have this error in my Open Subnet manager log. Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ...
2
votes
4answers
552 views

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
1
vote
0answers
917 views

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ...
2
votes
2answers
622 views

Mellanox and Infiniband latency

I got two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables. I'm really concern about latency which is higher than on ...
0
votes
0answers
97 views

forwarding traffic between the ports

How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology. PC Listing GUIDs: Port 0: 0x2c903000f6d27 Port 1: 0x2c903000f6d28 PC Listing GUIDs: Port 0: ...
0
votes
1answer
1k views

VMware infiniband configuration

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup: The first card should work ...
4
votes
2answers
4k views

Infiniband Verifying that RDMA is working

I have two identical computers with Mellanox cards connected to each other through a cable. No switch. Using opensm. I have run several tests, including ping_pong tests, ibping, etc. They all seem to ...
0
votes
0answers
1k views

opensm CentOS 7 software switch

I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports. I have read the ...
7
votes
1answer
5k views

Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $...
1
vote
1answer
251 views

Infiniband cable

What is the difference between an active and a passive cable? What is the difference between QSFP and QSFP+? Given the choice between copper and fiber, for a very short distance like 3 feet max, is ...
1
vote
0answers
2k views

CentOS 7 - Systemctl dependency problems with network mount and OpenSM

I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and ...
1
vote
0answers
562 views

Setting up Infiniband network - wrong route

So I'm trying to set up an Infiniband network alongside my Ethernet network. I have 10 compute nodes and one conductor node. All 11 machines are running CentOS and are plugged in to an Infiniband ...
0
votes
1answer
364 views

Debian-based Infiniband point-to-point Voltaire dual port cards

I have the following configuration: I------------------I I------------------I I I(P0)---------(P0)I I I Voltaire HCA 0 I I ...
0
votes
0answers
147 views

Getting eth_ipoib in Ubuntu

I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with. However, there appears to be no kernel module for it in the mainline. http://lwn.net/Articles/507258/ How ...
5
votes
1answer
3k views

Passing through RDMA network devices to docker containers

I'm wanting to passthrough infiniband to a docker container so that I can run some high performance apps over ipoib and use rdma. Currently, I'm doing this with xen virtual machines. Now I'm looking ...
2
votes
0answers
523 views

Infiniband + KVM

this question is a copy from stackoverflow, as I was told, my question would fit here better. Despite quite a long search for days, I could not find any good information about the following - though ...
0
votes
0answers
402 views

Possible to connect multiple InfiniBand switches?

Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need ...
0
votes
2answers
754 views

Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?

I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat. Now I'm running MPI tests to verify that InfiniBand is ...