Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

38 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
1 answer
309 views

highly available storage over infiniband: what other than mdraid?

is mdadm over infiniband a bad idea? what is the real trick to get reasonable performing storage to survive a single machine failure? We have been running ceph for a few years now and its great for ...
aep's user avatar
  • 187
3 votes
0 answers
1k views

Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
Ryan Babchishin's user avatar
3 votes
0 answers
2k views

Connecting 3 Server with Infiniband without a switch

Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question. I got myself a dirt cheap ...
the_architect's user avatar
2 votes
0 answers
392 views

Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using: Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s Description of ...
theenemy's user avatar
  • 121
2 votes
0 answers
705 views

Infiniband: ibping works but no ib0 interface

(redirected here from NetworkEngineering) I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
user407898's user avatar
2 votes
0 answers
109 views

Can I use Ethernet transceivers on InfiniBand devices?

Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices? If not, what's the cause of such incompatibility?
Proton's user avatar
  • 121
2 votes
0 answers
2k views

CentOS 7 - Systemctl dependency problems with network mount and OpenSM

I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and ...
M.K. aka Grisu's user avatar
2 votes
0 answers
977 views

Can two infiniband ports be bridged together to make a switch?

I have 3 machines with infiniband cards, two with single ports and one with two ports. I do not own any switches. I have my network setup such that the computer with 2 ports, running Linux, routes ...
Dougvj's user avatar
  • 121
2 votes
0 answers
252 views

Infiniband link is at 4X SDR with DDR HCAs; has anyone seen this before?

I've recently upgraded my Infiniband HCAs from SDR to DDR, however upon upgrading, it has not improved the link speed at all. Receiving the following output from 'iblinkinfo': CA: ip-san-1-au3a ...
jaseeey's user avatar
  • 1,472
2 votes
2 answers
920 views

Infiniband, SRP and failover

I am considering an linux-based infiniband storage setup, but I am tripping over whether or not I can do active/passive or active/active failover with DRBD and SRP targets. I am led to believe that ...
nrb's user avatar
  • 93
2 votes
0 answers
472 views

How do I make dnsmasq serve IP addresses via IPoIB?

I have a cluster farm that I'm setting up. The nodes (computers in the farm) are connected via ethernet & IP over Infiniband. I'm needing to netboot the nodes and thought dnsmasq would fit well ...
hookenz's user avatar
  • 14.6k
1 vote
1 answer
234 views

HPC master node no infiniband network influence on compute nodes - Slurm management

I'm writing because I'm facing an issue that I cannot solve trying to configure a cluster with a master node ( or Frontend node ) as a Virtual machine managing nodes with infiniband network. I use ...
SimoneM's user avatar
  • 121
1 vote
0 answers
15 views

what is intel omnipath topology error?

I have lustre setup and connected with intel omnipath. the metadata server is getting rebooted often. in logs, i can see the opa topology error. what is this error means?. will it cause the server to ...
heyloitsmeraj's user avatar
1 vote
2 answers
980 views

revive an old QLogic IBA7322 Infiniband card on CentOS8.2

I have a few HP Blades Gen7 equipped with QLogic Infiniband cards IBA7322 which I would like to use with CentOS 8. The problem I have is that I cannot find the right drivers for them. All the ...
lucian's user avatar
  • 131
1 vote
1 answer
975 views

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] ...
rg6's user avatar
  • 195
1 vote
0 answers
292 views

Error while creating an SRP target in container using targetcli

I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
geoffjay's user avatar
  • 131
1 vote
0 answers
251 views

Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?

Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)? Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR? Same question goes for 40Gb ...
Hull2356's user avatar
1 vote
1 answer
403 views

Set a upper bandwidth limit for Infiniband HCAs

I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband. Putting those HCAs in FDR-...
jyvet's user avatar
  • 111
1 vote
0 answers
2k views

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ...
jmlero's user avatar
  • 11
1 vote
0 answers
1k views

Setting up Infiniband network - wrong route

So I'm trying to set up an Infiniband network alongside my Ethernet network. I have 10 compute nodes and one conductor node. All 11 machines are running CentOS and are plugged in to an Infiniband ...
ostrich's user avatar
  • 71
1 vote
0 answers
643 views

HP ConnectX cards identifying as ConnectX EN

I bought a lot of four HP branded ConnectX card on Ebay. The cards are : HP InfiniBand 4X DDR ConnectX PCI-E Gen2 Dual-Port HCA 483514-B2 My problem is that two out of four cards identify themselves ...
user228840's user avatar
0 votes
1 answer
188 views

Mellanox ConnectX-7 - BW average drops from 395Gb/s to 248Gb/s in ib_write_bw test

I have two machines connected by two ConnectX-7 to each other. When running ib_write_bw test, the BW average starts from 395Gb/s, which is very good. But the speed drops very fast to less than 250Gb/s....
Xiaoquan Li's user avatar
0 votes
1 answer
368 views

Ethernet connection is dropped when restarting openibd ( Infiniband ) service

I have a multiple servers with Ethernet controller on board and InfiniBand controller installed in a PCI slot. The problem is when im restarting openibd.service which should control only the ...
Oren's user avatar
  • 21
0 votes
0 answers
196 views

HPC node, Infiniband is DOWN

I have an HPC with 17 nodes running CentOS 7 and a dedicated Mellanox SX6036 Infiniband switch, each node has an Infiniband FDR interface. Recently one node started giving errors and a quick look ...
Chris Woelkers's user avatar
0 votes
0 answers
2k views

Infiniband adapter down

edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2: I am not able to get my Infiniband adapter working. The output of ibstat states that it is down: CA 'mlx5_0' ...
Holger's user avatar
  • 51
0 votes
1 answer
1k views

Infiniband P_Keys and the Linux kernel

I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
Server Fault's user avatar
  • 3,774
0 votes
0 answers
37 views

QLogic 12300 :: Can't use more than 6 ports

I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
George Vardikos's user avatar
0 votes
1 answer
2k views

MPI can not find Infiniband interface point-to-point

I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors: [Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
Bryan Carroll's user avatar
0 votes
1 answer
88 views

HP Infiniband HCA on Fujitsu Motherboard

I am building a 2 node cluster using node-to-node Infiniband connection. I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
MaveriQ's user avatar
0 votes
1 answer
719 views

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
MDK's user avatar
  • 101
0 votes
0 answers
114 views

forwarding traffic between the ports

How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology. PC Listing GUIDs: Port 0: 0x2c903000f6d27 Port 1: 0x2c903000f6d28 PC Listing GUIDs: Port 0: ...
Roman Ostrovsky's user avatar
0 votes
0 answers
2k views

opensm CentOS 7 software switch

I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports. I have read the ...
Ivan's user avatar
  • 299
0 votes
1 answer
558 views

Debian-based Infiniband point-to-point Voltaire dual port cards

I have the following configuration: I------------------I I------------------I I I(P0)---------(P0)I I I Voltaire HCA 0 I I ...
user186658's user avatar
0 votes
0 answers
237 views

Getting eth_ipoib in Ubuntu

I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with. However, there appears to be no kernel module for it in the mainline. http://lwn.net/Articles/507258/ How ...
hookenz's user avatar
  • 14.6k
0 votes
0 answers
593 views

Possible to connect multiple InfiniBand switches?

Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need ...
cmorris14's user avatar
  • 181
0 votes
1 answer
633 views

Cannot use SDP in Debian Squeezy

I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this: mlx4_ib # Mellanox ConnectX cards #ib_mthca # some mellanox ...
Ulf's user avatar
  • 387
0 votes
1 answer
1k views

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I ...
catbus's user avatar
  • 1
-1 votes
1 answer
338 views

2 servers with infiniband cards and one IB switch i want to use the Ip Over IB

I have 2 super Micro servers with XCP installed i have 2 Infiniband cards on them, i have IB switch also, Now i want to connect Both servers with IP O IB, How can i do that, if i am using Infiniband i ...
Jagan Thoutam's user avatar