Questions tagged [infiniband]
InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.
38
questions with no upvoted or accepted answers
3
votes
1
answer
309
views
highly available storage over infiniband: what other than mdraid?
is mdadm over infiniband a bad idea?
what is the real trick to get reasonable performing storage to survive a single machine failure?
We have been running ceph for a few years now and its great for ...
3
votes
0
answers
1k
views
Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software
I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
3
votes
0
answers
2k
views
Connecting 3 Server with Infiniband without a switch
Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question.
I got myself a dirt cheap ...
2
votes
0
answers
392
views
Infiniband fabric with 3 nodes - newbie
I am trying to connect 3 HP z840 workstations using:
Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s
Description of ...
2
votes
0
answers
705
views
Infiniband: ibping works but no ib0 interface
(redirected here from NetworkEngineering)
I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
2
votes
0
answers
109
views
Can I use Ethernet transceivers on InfiniBand devices?
Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices?
If not, what's the cause of such incompatibility?
2
votes
0
answers
2k
views
CentOS 7 - Systemctl dependency problems with network mount and OpenSM
I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and ...
2
votes
0
answers
977
views
Can two infiniband ports be bridged together to make a switch?
I have 3 machines with infiniband cards, two with single ports and one with two ports. I do not own any switches. I have my network setup such that the computer with 2 ports, running Linux, routes ...
2
votes
0
answers
252
views
Infiniband link is at 4X SDR with DDR HCAs; has anyone seen this before?
I've recently upgraded my Infiniband HCAs from SDR to DDR, however upon upgrading, it has not improved the link speed at all. Receiving the following output from 'iblinkinfo':
CA: ip-san-1-au3a ...
2
votes
2
answers
920
views
Infiniband, SRP and failover
I am considering an linux-based infiniband storage setup, but I am tripping over whether or not I can do active/passive or active/active failover with DRBD and SRP targets.
I am led to believe that ...
2
votes
0
answers
472
views
How do I make dnsmasq serve IP addresses via IPoIB?
I have a cluster farm that I'm setting up.
The nodes (computers in the farm) are connected via ethernet & IP over Infiniband. I'm needing to netboot the nodes and thought dnsmasq would fit well ...
1
vote
1
answer
234
views
HPC master node no infiniband network influence on compute nodes - Slurm management
I'm writing because I'm facing an issue that I cannot solve trying to configure a cluster with a master node ( or Frontend node ) as a Virtual machine managing nodes with infiniband network.
I use ...
1
vote
0
answers
15
views
what is intel omnipath topology error?
I have lustre setup and connected with intel omnipath. the metadata server is getting rebooted often. in logs, i can see the opa topology error. what is this error means?. will it cause the server to ...
1
vote
2
answers
980
views
revive an old QLogic IBA7322 Infiniband card on CentOS8.2
I have a few HP Blades Gen7 equipped with QLogic Infiniband cards IBA7322 which I would like to use with CentOS 8. The problem I have is that I cannot find the right drivers for them. All the ...
1
vote
1
answer
975
views
Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS
I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9,
server1>lspci
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
...
1
vote
0
answers
292
views
Error while creating an SRP target in container using targetcli
I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
1
vote
0
answers
251
views
Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?
Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)?
Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR?
Same question goes for 40Gb ...
1
vote
1
answer
403
views
Set a upper bandwidth limit for Infiniband HCAs
I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband.
Putting those HCAs in FDR-...
1
vote
0
answers
2k
views
Centos. infiniband multicast join failed for status -22
I have a cluster with Linux Centos 6.5 as OS.
All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message
...
1
vote
0
answers
1k
views
Setting up Infiniband network - wrong route
So I'm trying to set up an Infiniband network alongside my Ethernet network.
I have 10 compute nodes and one conductor node. All 11 machines are running CentOS and are plugged in to an Infiniband ...
1
vote
0
answers
643
views
HP ConnectX cards identifying as ConnectX EN
I bought a lot of four HP branded ConnectX card on Ebay. The cards are :
HP InfiniBand 4X DDR ConnectX PCI-E Gen2 Dual-Port HCA 483514-B2
My problem is that two out of four cards identify themselves ...
0
votes
1
answer
188
views
Mellanox ConnectX-7 - BW average drops from 395Gb/s to 248Gb/s in ib_write_bw test
I have two machines connected by two ConnectX-7 to each other.
When running ib_write_bw test, the BW average starts from 395Gb/s, which is very good. But the speed drops very fast to less than 250Gb/s....
0
votes
1
answer
368
views
Ethernet connection is dropped when restarting openibd ( Infiniband ) service
I have a multiple servers with Ethernet controller on board and InfiniBand controller installed in a PCI slot.
The problem is when im restarting openibd.service which should control only the ...
0
votes
0
answers
196
views
HPC node, Infiniband is DOWN
I have an HPC with 17 nodes running CentOS 7 and a dedicated Mellanox SX6036 Infiniband switch, each node has an Infiniband FDR interface.
Recently one node started giving errors and a quick look ...
0
votes
0
answers
2k
views
Infiniband adapter down
edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2:
I am not able to get my Infiniband adapter working.
The output of ibstat states that it is down:
CA 'mlx5_0'
...
0
votes
1
answer
1k
views
Infiniband P_Keys and the Linux kernel
I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
0
votes
0
answers
37
views
QLogic 12300 :: Can't use more than 6 ports
I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
0
votes
1
answer
2k
views
MPI can not find Infiniband interface point-to-point
I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors:
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
0
votes
1
answer
88
views
HP Infiniband HCA on Fujitsu Motherboard
I am building a 2 node cluster using node-to-node Infiniband connection.
I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
0
votes
1
answer
719
views
Can not setup Mellanox connectx-2 in 40g speed
I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
0
votes
0
answers
114
views
forwarding traffic between the ports
How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology.
PC
Listing GUIDs:
Port 0: 0x2c903000f6d27
Port 1: 0x2c903000f6d28
PC
Listing GUIDs:
Port 0: ...
0
votes
0
answers
2k
views
opensm CentOS 7 software switch
I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports.
I have read the ...
0
votes
1
answer
558
views
Debian-based Infiniband point-to-point Voltaire dual port cards
I have the following configuration:
I------------------I I------------------I
I I(P0)---------(P0)I I
I Voltaire HCA 0 I I ...
0
votes
0
answers
237
views
Getting eth_ipoib in Ubuntu
I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with.
However, there appears to be no kernel module for it in the mainline.
http://lwn.net/Articles/507258/
How ...
0
votes
0
answers
593
views
Possible to connect multiple InfiniBand switches?
Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need ...
0
votes
1
answer
633
views
Cannot use SDP in Debian Squeezy
I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this:
mlx4_ib
# Mellanox ConnectX cards
#ib_mthca # some mellanox ...
0
votes
1
answer
1k
views
Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA
SOLVED
I got troubles setting up infiniband software from OpenSUSE repository.
The same card works fine with SLES12 + Mellanox OFED stack.
Installing everything containing 'infiniband' from yast I ...
-1
votes
1
answer
338
views
2 servers with infiniband cards and one IB switch i want to use the Ip Over IB
I have 2 super Micro servers with XCP installed i have 2 Infiniband cards on them, i have IB switch also, Now i want to connect Both servers with IP O IB, How can i do that, if i am using Infiniband i ...