Questions tagged [infiniband]
InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.
114
questions
1
vote
1
answer
130
views
3 Servers, Infiniband, without IB switch
I took a look every post related similar questions but they didn't give me clear answer.
Our planned system consists of three machines named n1, n2, and n3. The connection is set up as follows:
n1 ----...
- 102k
0
votes
1
answer
199
views
Mellanox ConnectX-7 - BW average drops from 395Gb/s to 248Gb/s in ib_write_bw test
I have two machines connected by two ConnectX-7 to each other.
When running ib_write_bw test, the BW average starts from 395Gb/s, which is very good. But the speed drops very fast to less than 250Gb/s....
- 1
5
votes
1
answer
548
views
Mellanox ConnectX-7 - Unable to get more than 250 Gbit/s
I have several machines with ConnectX-7 Infiniband cards and they're plugged into an Nvidia QM9700 switch. I've confirmed 400 Gbit NDR at both ends (ibstat on the host and in the console on the ...
3
votes
1
answer
328
views
highly available storage over infiniband: what other than mdraid?
is mdadm over infiniband a bad idea?
what is the real trick to get reasonable performing storage to survive a single machine failure?
We have been running ceph for a few years now and its great for ...
- 508
0
votes
1
answer
384
views
Ethernet connection is dropped when restarting openibd ( Infiniband ) service
I have a multiple servers with Ethernet controller on board and InfiniBand controller installed in a PCI slot.
The problem is when im restarting openibd.service which should control only the ...
- 272
0
votes
0
answers
209
views
HPC node, Infiniband is DOWN
I have an HPC with 17 nodes running CentOS 7 and a dedicated Mellanox SX6036 Infiniband switch, each node has an Infiniband FDR interface.
Recently one node started giving errors and a quick look ...
- 308
1
vote
1
answer
101
views
How can I use nmap against an IPoIB network?
I'm trying to use nmap to check which services are running on a given network but it cannot start just because nmap issues ARP Ping Scans on IPv4 network, and on a Infiniband Network, as far as I know,...
- 5,620
1
vote
2
answers
1k
views
revive an old QLogic IBA7322 Infiniband card on CentOS8.2
I have a few HP Blades Gen7 equipped with QLogic Infiniband cards IBA7322 which I would like to use with CentOS 8. The problem I have is that I cannot find the right drivers for them. All the ...
- 131
14
votes
4
answers
11k
views
How can I determine which switch the Infiniband subnet manager is running on?
I recently inherited an Infiniband network containing multiple switches, and I know that one of these switches is running the subnet manager. The rest supposedly have that feature turned off, or were ...
2
votes
5
answers
2k
views
set the Ca name on unmanaged mellanox switch
How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch?
I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
- 1
-1
votes
1
answer
352
views
Speeding up SAS data rate from 12Gbps
I'm curious about SAS data transfer speed.
Maximum is 12Gbps in the whole bus (not per drive) as far as I understand, but I have a scenario where I would like to have a faster data rate (hopefully ...
- 11.7k
1
vote
1
answer
242
views
HPC master node no infiniband network influence on compute nodes - Slurm management
I'm writing because I'm facing an issue that I cannot solve trying to configure a cluster with a master node ( or Frontend node ) as a Virtual machine managing nodes with infiniband network.
I use ...
- 13
4
votes
2
answers
608
views
Set up Windows 10 SRP initiator
How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it.
The initiator will be Windows ...
- 161
4
votes
1
answer
1k
views
Infiniband + KVM
this question is a copy from stackoverflow, as I was told, my question would fit here better.
Despite quite a long search for days, I could not find any good information about the following - though ...
- 131
2
votes
2
answers
1k
views
Infiniband drivers : OFED or distro included?
I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome !
We are currently using Mellanox OFED drivers, but our infiniband ...
0
votes
0
answers
2k
views
Infiniband adapter down
edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2:
I am not able to get my Infiniband adapter working.
The output of ibstat states that it is down:
CA 'mlx5_0'
...
- 51
2
votes
0
answers
398
views
Infiniband fabric with 3 nodes - newbie
I am trying to connect 3 HP z840 workstations using:
Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s
Description of ...
- 5,620
2
votes
1
answer
454
views
How to access files from a local server with infiniband without using IP
I want to start of by saying that I'm very inexperienced with everything I'm doing so please take it easy on me.
I have 4 computers each with Red hat Enterprise Linux server 7.8 installed, they are ...
- 5,620
3
votes
2
answers
1k
views
Mellanox and Infiniband latency
I have two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables.
I'm really concerned about latency which is higher than on ...
- 4,514
1
vote
1
answer
237
views
Infiniband OpenSM N-to-N port routing configuration
I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch.
My RDMA application runs as one process per NUMA ...
- 121
2
votes
1
answer
6k
views
Unable to get Mellanox ConnectX-3 to work with transceiver/cable
I'm attempting to upgrade my local network setup to 40G but I seem to have hit a snag. I have Mellanox ConnectX-3 VPI CX353A network cards I got off ebay, updated their firmware and installed their ...
1
vote
1
answer
840
views
Latency of memory accesses via interconnectors
I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency.
I have obtained some benchmarks regarding one of the hardware ...
- 33.3k
1
vote
0
answers
15
views
what is intel omnipath topology error?
I have lustre setup and connected with intel omnipath. the metadata server is getting rebooted often. in logs, i can see the opa topology error. what is this error means?. will it cause the server to ...
1
vote
2
answers
920
views
CX4 Infiniband to SFP+ Converter
It may be a weird question, but I'm facing the following problem at the moment:
I've got a DELL PowerConnect 6248 Switch (managed Gigabit-Ethernet) with the YY741-Stacking module in the back. The ...
- 11.7k
4
votes
1
answer
4k
views
How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?
I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server.
What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines.
...
CommunityBot
- 1
2
votes
0
answers
725
views
Infiniband: ibping works but no ib0 interface
(redirected here from NetworkEngineering)
I have a number of Infiniband-connected servers that all report their ib0 network connection disappeared from ip and ifcfg within a few hours of each other. ...
- 21
0
votes
1
answer
1k
views
Infiniband P_Keys and the Linux kernel
I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
- 3,784
0
votes
1
answer
2k
views
MPI can not find Infiniband interface point-to-point
I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors:
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/...
2
votes
1
answer
2k
views
Replaced storage controller, now can't find ESXi ZFS mount point
We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the ...
CommunityBot
- 1
0
votes
0
answers
37
views
QLogic 12300 :: Can't use more than 6 ports
I have 2 QLogic 12300 InfiniBand switches (12300-BS01) with 36 ports. For some strange reason I am unable to connect anything to ports 7-36, it seems that only ports 1-6 are working. I Googled and ...
3
votes
1
answer
763
views
back to back infiniband
I've been given two hpc cluster nodes, each with an infiniband hca.
Is it possible to connect them back-to-back, without using a switch ?
0
votes
1
answer
204
views
Can you mix 10gb and Infiniband for iScsi storage target
I'm setting up a Fedora server to act as an iScsi storage target. The primary storage network will be 25gb Infiniband. Is it possible and/or safe to also allow the target to be accessed by 10gb ...
- 11.7k
2
votes
0
answers
111
views
Can I use Ethernet transceivers on InfiniBand devices?
Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices?
If not, what's the cause of such incompatibility?
- 121
-2
votes
1
answer
407
views
How Infiniband manages to have 100Gbit/s
How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
- 106
1
vote
1
answer
986
views
Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS
I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9,
server1>lspci
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
...
- 452
0
votes
1
answer
209
views
Managed switch for Mellanox MCX354A-FCBT cards?
I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
- 452
0
votes
1
answer
729
views
Can not setup Mellanox connectx-2 in 40g speed
I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
- 452
8
votes
1
answer
2k
views
File-sharing over Infiniband on FreeBSD
I have a pair of servers, which I'd like to connect to each other
directly -- rather than through the switch shared by the rest of
the network. I expect to benefit from lower latency and higher
...
- 452
2
votes
2
answers
864
views
Dual-port RDMA Throughput Issue
I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
- 181
0
votes
1
answer
89
views
HP Infiniband HCA on Fujitsu Motherboard
I am building a 2 node cluster using node-to-node Infiniband connection.
I have HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP (P/N : 649282-B21) and Fujitsu D3118-A22 GS-2 Mainboard with dual ...
- 531
0
votes
4
answers
2k
views
allow infiniband for non root users
I got Infiniband running on RHEL 6.3
[root@master ~]# ibv_devinfo
hca_id: mthca0
transport: InfiniBand (0)
fw_ver: 4.7.927
node_guid: 0017:08ff:ffd0:6f1c
sys_image_guid:...
- 130
1
vote
2
answers
137
views
Options for an ultra-fast small LAN [closed]
I need to transfer several terabytes/day between three windows computers. Two computers are employed to acquire medical imaging data (500 Gbytes/sample, ca. 6-7 samples/day), and the third is ...
- 50.7k
4
votes
4
answers
13k
views
Set up simple Infiniband Block Storage (SRP or iSER)
I'm trying to figure out how to set up a simple storage system which exports block storage over Infiniband, using either SRP or iSER.
I'm very early in the process, and at the moment I'm basically ...
- 175
0
votes
1
answer
214
views
Any way to segregate ports on a Mellanox Infiniband switch?
I currently have two infiniband IPoIB networks across multiple unmanaged switches (along with some PCs running as the subnet managers) with multiple Linux nodes attached with Infiniband adapters.
...
- 272
0
votes
1
answer
158
views
What is a 4x 10gbe QSFP-QSFP cable?
I was looking for QSFP cables on Ebay and came across a Brocade 58-0000033-01 marked as 4x 10GE even though it has one QSFP connector on each end. I don't understand what this cable is for, will it ...
- 50.7k
4
votes
2
answers
4k
views
Virtual infiniband on kvm/qemu or another open source platform?
Is there any way to set up a virtual infiniband network within kvm/qemu? If not, is there a viable open source alternative which would allow virtual infiniband networking?
(Note that I'm not looking ...
- 161
6
votes
2
answers
4k
views
Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections
I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, ...
- 31
0
votes
1
answer
1k
views
I want to connect to bridge using infiniband
I want to connect the physical port (ib0) to the bridge (bro).
I am constructing kvm, but the physical port of infiniband to the virtual bridge
I want to connect, but I could not do it well.
Please ...
- 272
0
votes
1
answer
4k
views
How to set Infiniband port from INIT to ACTIVE
I have the following setup. 7 Nodes, let's say they are called gauss1 to gauss7. I have a stable connection between gauss1 to gauss6. Just gauss7 is making trouble.
# ibnodes
Ca : 0x0002c90300f2eef0 ...
- 171
0
votes
1
answer
312
views
Ethernet switch with Infiniband uplink
There are Ethernet switches (copper based, i.e. with RJ45 sockets) that have Fibre uplink. Example: Netgear GS110TP
Are there any such switches with Infiniband uplinks? If not then why?
- 4,852