Questions tagged [infiniband]
InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.
112
questions
0
votes
0
answers
14
views
HPC node, Infiniband is DOWN
I have an HPC with 17 nodes running CentOS 7 and a dedicated Mellanox SX6036 Infiniband switch, each node has an Infiniband FDR interface.
Recently one node started giving errors and a quick look ...
- 190
2
votes
0
answers
51
views
highly available storage over infiniband: what other than mdraid?
is mdadm over infiniband a bad idea?
what is the real trick to get reasonable performing storage to survive a single machine failure?
We have been running ceph for a few years now and its great for ...
- 177
1
vote
1
answer
57
views
How can I use nmap against an IPoIB network?
I'm trying to use nmap to check which services are running on a given network but it cannot start just because nmap issues ARP Ping Scans on IPv4 network, and on a Infiniband Network, as far as I know,...
- 5,480
0
votes
0
answers
15
views
How can i set up my SunOracle Switch?
I am posting this question to see if anyone can help me. What I am trying to do is network my 2 hosts one being my proxmox machine and the other being my truenas scale machine into a sun oracle ...
-1
votes
1
answer
118
views
Speeding up SAS data rate from 12Gbps
I'm curious about SAS data transfer speed.
Maximum is 12Gbps in the whole bus (not per drive) as far as I understand, but I have a scenario where I would like to have a faster data rate (hopefully ...
- 11
0
votes
0
answers
41
views
`failed to found root port` when using `ibdiagnet`
I am trying to run ibdiagnet tool to test the InfiniBand connections as instructed by NVIDIA Sharp.
Yet I got the following error:
$ sudo ibdiagnet
Running: ibdiagnet
----------
Load Plugins from:
/...
- 101
1
vote
1
answer
104
views
HPC master node no infiniband network influence on compute nodes - Slurm management
I'm writing because I'm facing an issue that I cannot solve trying to configure a cluster with a master node ( or Frontend node ) as a Virtual machine managing nodes with infiniband network.
I use ...
- 121
0
votes
0
answers
923
views
Infiniband adapter down
edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2:
I am not able to get my Infiniband adapter working.
The output of ibstat states that it is down:
CA 'mlx5_0'
...
- 51
2
votes
0
answers
269
views
Infiniband fabric with 3 nodes - newbie
I am trying to connect 3 HP z840 workstations using:
Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s
Description of ...
- 121
1
vote
1
answer
159
views
Infiniband OpenSM N-to-N port routing configuration
I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch.
My RDMA application runs as one process per NUMA ...
- 121
2
votes
1
answer
4k
views
Unable to get Mellanox ConnectX-3 to work with transceiver/cable
I'm attempting to upgrade my local network setup to 40G but I seem to have hit a snag. I have Mellanox ConnectX-3 VPI CX353A network cards I got off ebay, updated their firmware and installed their ...
1
vote
1
answer
424
views
Latency of memory accesses via interconnectors
I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency.
I have obtained some benchmarks regarding one of the hardware ...
- 31
2
votes
1
answer
250
views
How to access files from a local server with infiniband without using IP
I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me.
I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are ...
- 23
1
vote
0
answers
11
views
what is intel omnipath topology error?
I have lustre setup and connected with intel omnipath. the metadata server is getting rebooted often. in logs, i can see the opa topology error. what is this error means?. will it cause the server to ...
2
votes
2
answers
773
views
Infiniband drivers : OFED or distro included?
I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome !
We are currently using Mellanox OFED drivers, but our infiniband ...
- 33
1
vote
2
answers
739
views
revive an old QLogic IBA7322 Infiniband card on CentOS8.2
I have a few HP Blades Gen7 equipped with QLogic Infiniband cards IBA7322 which I would like to use with CentOS 8. The problem I have is that I cannot find the right drivers for them. All the ...
- 131
2
votes
0
answers
480
views
Infiniband: ibping works but no ib0 interface
(redirected here from NetworkEngineering)
I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
- 21
0
votes
1
answer
845
views
Infiniband P_Keys and the Linux kernel
I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
- 3,634
0
votes
0
answers
29
views
QLogic 12300 :: Can't use more than 6 ports
I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
0
votes
1
answer
1k
views
MPI can not find Infiniband interface point-to-point
I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors:
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
0
votes
1
answer
162
views
Can you mix 10gb and Infiniband for iScsi storage target
I'm setting up a Fedora server to act as an iScsi storage target. The primary storage network will be 25gb Infiniband. Is it possible and/or safe to also allow the target to be accessed by 10gb ...
- 443
2
votes
0
answers
58
views
Can I use Ethernet transceivers on InfiniBand devices?
Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices?
If not, what's the cause of such incompatibility?
- 121
-2
votes
1
answer
289
views
How Infiniband manages to have 100Gbit/s
How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
0
votes
1
answer
77
views
HP Infiniband HCA on Fujitsu Motherboard
I am building a 2 node cluster using node-to-node Infiniband connection.
I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
- 1
8
votes
1
answer
1k
views
File-sharing over Infiniband on FreeBSD
I have a pair of servers, which I'd like to connect to each other
directly -- rather than through the switch shared by the rest of
the network. I expect to benefit from lower latency and higher
...
- 2,282
1
vote
2
answers
797
views
CX4 Infiniband to SFP+ Converter
It may be a weird question, but I'm facing the following problem at the moment:
I've got a DELL PowerConnect 6248 Switch (managed Gigabit-Ethernet) with the YY741-Stacking module in the back. The ...
- 193
0
votes
1
answer
612
views
Can not setup Mellanox connectx-2 in 40g speed
I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
- 101
1
vote
2
answers
135
views
Options for an ultra-fast small LAN [closed]
I need to transfer several terabytes/day between three windows computers. Two computers are employed to acquire medical imaging data (500 Gbytes/sample, ca. 6-7 samples/day), and the third is ...
- 407
4
votes
2
answers
469
views
Set up Windows 10 SRP initiator
How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it.
The initiator will be Windows ...
- 212
0
votes
1
answer
152
views
Any way to segregate ports on a Mellanox Infiniband switch?
I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters.
...
- 3,634
0
votes
1
answer
134
views
What is a 4x 10gbe QSFP-QSFP cable?
I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it ...
- 212
0
votes
1
answer
161
views
Managed switch for Mellanox MCX354A-FCBT cards?
I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
- 3,634
1
vote
1
answer
808
views
Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS
I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9,
server1>lspci
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
...
- 185
2
votes
2
answers
671
views
Dual-port RDMA Throughput Issue
I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
0
votes
1
answer
925
views
I want to connect to bridge using infiniband
I want to connect the physical port (ib0) to the bridge (bro).
I am constructing kvm, but the physical port of infiniband to the virtual bridge
I want to connect, but I could not do it well.
Please ...
- 11
0
votes
1
answer
4k
views
How to set Infiniband port from INIT to ACTIVE
I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble.
# ibnodes
Ca : 0x0002c90300f2eef0 ...
- 151
0
votes
1
answer
190
views
Ethernet switch with Infiniband uplink
There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP
Are there any such switches with Infiniband uplinks? If not then why?
- 198
1
vote
1
answer
306
views
Get firmware version for InfiniBand adapter from QLogic
How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server?
ibstat displays an empty version number like so:
Firmware version:
Hardware version: 2
lspci -...
- 365
8
votes
1
answer
3k
views
IPoIB (IP over InfiniBand) vs. RDMA performance
I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it....
- 623
0
votes
0
answers
23
views
Ideal configuration for a head node? [duplicate]
Which hardware should I concentrate on, when assembling a head node for an HPC cluster? The main task for the head node is to relay instructions to the compute nodes which will be running artificial ...
- 111
0
votes
1
answer
428
views
If I replace an Infiniband switch should I restart opensm?
I have been using Mellanox ConnectX 2 QDR cards in a point to point link between two machines, with opensm running on one machine. This has worked very well.
Recently I got a Mellanox IS5022 switch - ...
- 113
1
vote
2
answers
704
views
In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?
Good day,
and my disclaimer that I am no IB expert and have not worked on IB directly yet.
My question seems a trivial one, yet I could not find anything in google that answers it.
Question: In ...
- 21
6
votes
2
answers
3k
views
Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections
I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, ...
- 163
0
votes
1
answer
978
views
Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA
SOLVED
I got troubles setting up infiniband software from OpenSUSE repository.
The same card works fine with SLES12 + Mellanox OFED stack.
Installing everything containing 'infiniband' from yast I ...
- 1
1
vote
1
answer
611
views
ifconfig apparently showing wrong RX/TX values for InfiniBand HCA
Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes.
Oddly, while Ethernet packets seem to ...
- 133
1
vote
0
answers
246
views
Error while creating an SRP target in container using targetcli
I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to ...
- 131
2
votes
1
answer
2k
views
Replaced storage controller, now can't find ESXi ZFS mount point
We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the ...
1
vote
0
answers
250
views
Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?
Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)?
Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR?
Same question goes for 40Gb ...
- 11
2
votes
1
answer
116
views
How many infiniband adapters should be used in multi socket servers?
Should dual socket motherboards have an infinity band adapter for each CPU?
That is, should there be two infiniband band adapters, one in each CPU's PCIe slot.
Would this eliminate the signal going ...
- 311
3
votes
0
answers
1k
views
Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software
I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE ...
- 6,240