InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

learn more… | top users | synonyms

0
votes
1answer
22 views

Infiniband switch, uplink

Regarding switches like Mellanox Voltaire VLT-30011 (QDR IB, managed). I see a gigabit ethernet port on the switch which in the manual ...
0
votes
0answers
99 views

Can't use hostnames to connect to SMB shares, only IP addresses

Right now I have a cluster of servers connected to a single storage server. 3 Hypervisors, 1 Storage Server The ethernet interfaces on the hypervisors are on a different subnet than the ethernet ...
1
vote
1answer
103 views

In Infiniband networks, what is an MGID and how does it relate to multicast group?

I have this error in my Open Subnet manager log. Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ...
0
votes
0answers
33 views

Enable flexboot on Mellanbox based OEM firmware?

I'm updating some firmware on my Infiniband Mellanox ConnectX-3 adapters. They are not Mellanox branded cards, but are mellanox based and onboard. The firmware is not available on the manufacturers ...
0
votes
0answers
30 views

Performance impact of blocking Infiniband

How does a small blocking in an Infiniband network affect the latency? More specifically, if I compare a 4xQDR non-blocking setup with an FDR setup with a blocking factor of 1:1.6, how would they ...
0
votes
0answers
31 views

VL15Dropped Infiniband Errors

Has anyone seen VL15Dropped errors before? The only thing I can find about it is a definition... "VL15Dropped: The number of incoming VL15 packets dropped due to resource limitations (for example, ...
1
vote
2answers
69 views

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or ...
0
votes
0answers
172 views

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ...
0
votes
0answers
25 views

How to disconnect a SRP (SCSI RDMA Protocol) drive after unmounting

I have an InfiniBand network working with SRP sharing. On the target system, I am running targetcli (the -fb free branch version) and OpenFabric's srptools. On the initiator system, I am running ...
2
votes
2answers
74 views

Mellanox and Infiniband latency

I got two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables. I'm really concern about latency which is higher than on ...
0
votes
0answers
28 views

forwarding traffic between the ports

How to set up forwarding traffic between interfaces "HCA"? I want to implement a ring topology. PC Listing GUIDs: Port 0: 0x2c903000f6d27 Port 1: 0x2c903000f6d28 PC Listing GUIDs: Port 0: ...
0
votes
1answer
377 views

VMware infiniband configuration

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup: The first card should work ...
2
votes
2answers
496 views

Infiniband Verifying that RDMA is working

I have two identical computers with Mellanox cards connected to each other through a cable. No switch. Using opensm. I have run several tests, including ping_pong tests, ibping, etc. They all seem to ...
0
votes
0answers
312 views

opensm CentOS 7 software switch

I have two computers both with infiniband cards in them that I would like to connect without a switch. I have a cable connecting the two computers through their QSFP ports. I have read the ...
3
votes
1answer
550 views

Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend ...
1
vote
1answer
86 views

Infiniband cable

What is the difference between an active and a passive cable? What is the difference between QSFP and QSFP+? Given the choice between copper and fiber, for a very short distance like 3 feet max, is ...
1
vote
0answers
607 views

CentOS 7 - Systemctl dependency problems with network mount and OpenSM

I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and ...
1
vote
0answers
173 views

Setting up Infiniband network - wrong route

So I'm trying to set up an Infiniband network alongside my Ethernet network. I have 10 compute nodes and one conductor node. All 11 machines are running CentOS and are plugged in to an Infiniband ...
0
votes
1answer
142 views

Debian-based Infiniband point-to-point Voltaire dual port cards

I have the following configuration: I------------------I I------------------I I I(P0)---------(P0)I I I Voltaire HCA 0 I I ...
0
votes
0answers
51 views

Getting eth_ipoib in Ubuntu

I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with. However, there appears to be no kernel module for it in the mainline. http://lwn.net/Articles/507258/ How ...
4
votes
1answer
1k views

Passing through RDMA network devices to docker containers

I'm wanting to passthrough infiniband to a docker container so that I can run some high performance apps over ipoib and use rdma. Currently, I'm doing this with xen virtual machines. Now I'm looking ...
1
vote
0answers
155 views

Infiniband + KVM

this question is a copy from stackoverflow, as I was told, my question would fit here better. Despite quite a long search for days, I could not find any good information about the following - though ...
0
votes
0answers
153 views

Possible to connect multiple InfiniBand switches?

Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need ...
0
votes
2answers
443 views

Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?

I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat. Now I'm running MPI tests to verify that InfiniBand is ...
1
vote
1answer
220 views

InfiniBand RDMA latency test on Xen's dom0 crashes

The short story: while setting up InfiniBand connection between two servers, I cannot complete the RDMA latency test. It crashes even with breaking the ssh connection. The long story. The first ...
1
vote
0answers
195 views

HP ConnectX cards identifying as ConnectX EN

I bought a lot of four HP branded ConnectX card on Ebay. The cards are : HP InfiniBand 4X DDR ConnectX PCI-E Gen2 Dual-Port HCA 483514-B2 My problem is that two out of four cards identify themselves ...
2
votes
3answers
85 views

Is it possible to get any kind of unique identifier for the Infiniband switch that a particular Linux box is connected to?

I am trying to identify which Infiniband NIC's are connected to which switches, in a manner that is very similar to this question. However, the solution proposed there is not working for Mellanox ...
2
votes
0answers
178 views

Can two infiniband ports be bridged together to make a switch?

I have 3 machines with infiniband cards, two with single ports and one with two ports. I do not own any switches. I have my network setup such that the computer with 2 ports, running Linux, routes ...
4
votes
2answers
1k views

what does `pipe <number>` mean in `ping` output?

While running a test to diagnose some alleged network problem, I got this output from ping: --- r01c01b11n02 ping statistics --- 30000 packets transmitted, 29995 received, 0% packet loss, time 6309ms ...
0
votes
1answer
102 views

How much throughput increase to expect from link aggregation (LACP/802.3ad)?

I identified that data transfer within my cluster will be bottlenecked by the network interconnect (I'm saturating the dual SAS connection to my storage with sequential workloads), and I'm undecided ...
2
votes
0answers
186 views

Infiniband link is at 4X SDR with DDR HCAs; has anyone seen this before?

I've recently upgraded my Infiniband HCAs from SDR to DDR, however upon upgrading, it has not improved the link speed at all. Receiving the following output from 'iblinkinfo': CA: ip-san-1-au3a ...
4
votes
1answer
549 views

iSCSI TGTD over InfiniBand Performance

I have just configured a server-to-server 4x SDR InfiniBand link to increase bandwidth for the backend storage of my virtual machines, specifically a database server. I am using iSCSI with TGTD on ...
2
votes
1answer
974 views

How to configure hosts to show host name in ibnetdiscover (infiniband)?

Windows machines with the OFED stack installed show the host names in ibnetdiscover. Under Ubuntu Linux all I see is "MT25408 ConnectX Mellanox Technologies". Is there a way of configuring the ...
3
votes
1answer
945 views

Is it possible to update Supermicro ConnectX-3 Firmware with newer firmware from Mellanox?

I have a Supermicro server with a X9DRT-HIBFF motherboard installed. It has on-board Infiniband based on the ConnectX-3 chipset from Mellanox. Supermicro supply firmware version 2.10.0700 and ...
3
votes
1answer
2k views

How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?

I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server. What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines. ...
0
votes
1answer
1k views

How do I get IP packats forwarded/routed to/from my Infiniband network

I have two networks. One is a standard Ethernet network running IP. The second network is Infiniband which in addition to some custom protocols can talks IPoIB (IP over Infiniband). The router ...
1
vote
2answers
984 views

Alternatives to bridging infiniband ipoib within xen domains?

I've installed xen-hypervisor-4.1 package onto an Ubuntu 12.04.3 server. It has an infiniband adaptor installed and I run IPoIB with it. What I'm wanting to do is provide the ipoib to one or more ...
1
vote
2answers
2k views

Infiniband port status UP but can't open UMAD port ((null):0)

My system has 2 infiniband devices, one of which has both the ports up. $> ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:0002:c903:000f:0a9f ...
-1
votes
1answer
139 views

2 servers with infiniband cards and one IB switch i want to use the Ip Over IB

I have 2 super Micro servers with XCP installed i have 2 Infiniband cards on them, i have IB switch also, Now i want to connect Both servers with IP O IB, How can i do that, if i am using Infiniband i ...
2
votes
2answers
2k views

Mellanox dual-port HCA, can ping if ib0 pair connected but not if only ib1 pair connected

I have installed 2 mellanox FDR dual-port ConnectX-3 HCA cards (CX354A), each to a separate machines. The machines are connected directly to each other (switchless configuration). Both ports on the ...
1
vote
2answers
494 views

Infiniband, SRP and failover

I am considering an linux-based infiniband storage setup, but I am tripping over whether or not I can do active/passive or active/active failover with DRBD and SRP targets. I am led to believe that ...
0
votes
0answers
504 views

Setting up computational cluster using infiniband fails

My task: I need to assemble computational cluster, consists of two nodes using infiniband technology. When I try to start mpirun ... for two nodes, the process hangs. If i try to use flag ...
1
vote
3answers
1k views

are DDR infiniband and 10Gbase-CX4 cables the same?

They looks the same. It's SFF-8470 Is it ok to buy a "10Gbase-CX4" cable for an infiniband network ?
10
votes
3answers
2k views

How can I determine which switch the Infiniband subnet manager is running on?

I recently inherited an Infiniband network containing multiple switches, and I know that one of these switches is running the subnet manager. The rest supposedly have that feature turned off, or were ...
0
votes
1answer
444 views

Cannot use SDP in Debian Squeezy

I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this: mlx4_ib # Mellanox ConnectX cards #ib_mthca # some mellanox ...
5
votes
4answers
3k views

switchless Infiniband between two servers on RHEL 6.3

I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards >lspci | grep -i infini 07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02) I'm interested in connecting ...
-8
votes
3answers
926 views

infiniband network between 3 servers [closed]

Let's say I have 3 different servers, each one with an infiniband card. Each card has 2 different ports. (I don't know about the model yet) Is it possible to create 3 different networks and to allow ...
2
votes
0answers
227 views

How do I make dnsmasq serve IP addresses via IPoIB?

I have a cluster farm that I'm setting up. The nodes (computers in the farm) are connected via ethernet & IP over Infiniband. I'm needing to netboot the nodes and thought dnsmasq would fit well ...
-4
votes
2answers
143 views

What is the fastest way to connect two machines for a ping-pong test? [closed]

I need to do a latency test between two machines, basically a ping-pong packet test. I wonder what is the fastest connection I can have between those two machines. I imagine my options are: Ethernet ...
1
vote
3answers
819 views

allow infiniband for non root users

I got Infiniband running on RHEL 6.3 [root@master ~]# ibv_devinfo hca_id: mthca0 transport: InfiniBand (0) fw_ver: 4.7.927 node_guid: 0017:08ff:ffd0:6f1c ...