Skip to main content

Questions tagged [mellanox]

Mellanox Technologies Ltd. is an Israeli and American multinational supplier of computer networking products using InfiniBand and Ethernet technology

Filter by
Sorted by
Tagged with
0 votes
1 answer
78 views

What is this internal plug on my HP Mellanox ConnectX-3 card near the activity LEDs?

I recently bought a second-hand Mellanox ConnectX-3 card for a lab setup. There seems to be an internal connector on it near the activity LEDs, which I think might be for hooking them up to a front ...
Nick T's user avatar
  • 266
1 vote
2 answers
277 views

Mellanox ConnectX-7 - BW average drops from 395Gb/s to 248Gb/s in ib_write_bw test

I have two machines connected by two ConnectX-7 to each other. When running ib_write_bw test, the BW average starts from 395Gb/s, which is very good. But the speed drops very fast to less than 250Gb/s....
Xiaoquan Li's user avatar
-1 votes
1 answer
91 views

Could I enable ROCE v2 on both MCX623106AN-CDAT 100GbE NIC cards and connect each other without any switch?

Since I do not have ROCE v2 supported switch, and I am planning to connect two MCX623106AN-CDAT 100GbE NIC cards each other from two different server systems, the OS I plan to use is Windows Server ...
Jacky Lee's user avatar
1 vote
1 answer
383 views

mellanox connectx-3 (cx312a) link down though light is blinking

I have a system running TrueNas Scale. For some odd reasons, the system does not see the link is up even though dmesg reports it up and the light is blinking on the port. I'm not sure if this is optic'...
Teacup's user avatar
  • 11
0 votes
0 answers
127 views

Nothing but DHCP works after testing SR-IOV on Mellanox ConnectX-4 Lx

I was following the Proxmox guide for enabling PCIe passthrough and SR-IOV for my NIC, since I am running Vyos in a VM as a router. However, after undoing all the changes the NIC is not working ...
VMelnalksnis's user avatar
0 votes
0 answers
148 views

Number of receiving queues is double the number of cores of my server?

I'm using Mellanox ConnectX-5 100 GB NIC, Linux kernel 5.15 and my server has 32 cores (SMT disabled). But in the /sys/net//queues, the number of rx queues is 64, which is double the number of cores ...
Anh Phan's user avatar
-1 votes
1 answer
1k views

Very poor performance on Kubernetes with 100GbE network

we are using ConnectX-5 100GbE ethernet cards on our servers which is connected one to each other trough the mellanox switch. And we are using weavenet cni plugin on our Kubernetes cluster. When we ...
Zekeriya Akgül's user avatar
3 votes
0 answers
928 views

LinuxPTP slave not using master and sync timeout tx announce timeout

And: I'm using G8316 as swtich with ptp transparent enable and Mellanox 354 adapter. Ports using LACP bonding and trunk. First: Server A ptp4l -m -l 7 -i xxxx -i xxxx -S I can capture ServerA ...
Alceatraz's user avatar
  • 133
0 votes
1 answer
1k views

Only getting 25Gb/s instead of 100Gb/s from a Mellanox switch running Cumulus Linux

I have a Mellanox 100gb/s switch (running Cumulus Linux 4.1) that I use for connecting multiple servers, each with a Mellanox ConnectX 5 100gb/s card. These servers connect to the switch via a DAC ...
Jarmund's user avatar
  • 535
0 votes
1 answer
1k views

K8s nodeLocalDns pod times out connecting to coreDns after upgrading base os to ubuntu20.04 ConnectX-4 card

Team, I have Mellanox Nic ConnectX-4 on a k8s worker node and it hosts a nodeLocal dns pod on it. The nodeLocalDns pod is timing out when trying to connect to coreDns service on k8s cluster. Same ...
AhmFM's user avatar
  • 119
1 vote
1 answer
1k views

X710 and mlx5_core together on same server

I have a amd server with centos 7.9 I am trying to using the the kernel driver for 2 cards: Ethernet Controller X710 for 10GbE SFP+ 1572 --> i40e MT27800 Family [ConnectX-5] 1017' --> mlx5_core ...
yaron's user avatar
  • 113
0 votes
0 answers
2k views

Infiniband adapter down

edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2: I am not able to get my Infiniband adapter working. The output of ibstat states that it is down: CA 'mlx5_0' ...
Holger's user avatar
  • 51
2 votes
0 answers
409 views

Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using: Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s Description of ...
theenemy's user avatar
  • 121
1 vote
1 answer
259 views

Infiniband OpenSM N-to-N port routing configuration

I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch. My RDMA application runs as one process per NUMA ...
Hugo Maxwell's user avatar
2 votes
2 answers
1k views

Infiniband drivers : OFED or distro included?

I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome ! We are currently using Mellanox OFED drivers, but our infiniband ...
nirnaeth's user avatar
3 votes
0 answers
2k views

Why isn't SMB Direct working over RDMA?

In a Windows lab environment, I'm trying to test performance of SMB Direct using Mellanox RDMA adapters with global pause. I wanted to test first with Global Pause for simplicity before switching to ...
Elliott B's user avatar
  • 200
4 votes
0 answers
2k views

How to make systemd monitor processes started using a shell script in the service file?

I have written a service file in Linux that calls a script which starts multiple processes with one as primary and others as secondary. SERVICE FILE: [Unit] Description=TEST service Requires=abc....
Vedang's user avatar
  • 41
1 vote
1 answer
168 views

A way to configure DMZ zone on Ethernet Mellanox Switch

Can't find any information is it possible at all and how to configure DMZ zone on Mellanox MSN2100 switch. If I setup ACL disabling traffic from DMZ to LAN, I can't receive response to requests made ...
Mike's user avatar
  • 113
0 votes
1 answer
1k views

Infiniband P_Keys and the Linux kernel

I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
Server Fault's user avatar
  • 3,794
1 vote
0 answers
428 views

Missing Azure accelerated networking interfaces while VMs initializes

I'm running my application on top of Azure Ubuntu 16.04 VMs (Wrapped as a VHD image). In several cases I have observed missing SRIOV associated VFs interfaces, while VMs were starting. Need to mention ...
ido's user avatar
  • 11
2 votes
1 answer
2k views

Beegfs-client does not work with mellanox OFED driver

I am trying to configure and test beegfs with RDMA as explained in: https://community.mellanox.com/s/article/howto-configure-and-test-beegfs-with-rdma?t=1570613300675 My test configuration: OS: ...
Ramon's user avatar
  • 21
0 votes
1 answer
183 views

Mellanox adapter on GCP instance

I'm trying to install MOFED on an instance (preferably with a GPU) and was wondering if anyone knows if a Mellanox adapter is available at any Google Cloud Platform VM. The VM instances that I've been ...
afernandezody's user avatar
0 votes
1 answer
567 views

Connecting Intel X722 SFP+ to Mellanox SX1012 QSFP via 4-split breakout cable using ubuntu

I have two servers running Ubuntu 18.04, each equipped with two SFP+ interfaces, each showing up in lshw like the following. *-network:1 description: Ethernet interface product: ...
fbindel's user avatar
0 votes
1 answer
740 views

Can not setup Mellanox connectx-2 in 40g speed

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
MDK's user avatar
  • 101
9 votes
3 answers
2k views

QSFP28 - is there something like a crossover cable?

Can not find anything to that ;) Building a Hyperconvergend Cluster as development system. This is for us state of the art. Got 2 machines for a start, with a switch ordered to come in about 4 weeks....
TomTom's user avatar
  • 52k
0 votes
1 answer
212 views

Managed switch for Mellanox MCX354A-FCBT cards?

I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
Server Fault's user avatar
  • 3,794
1 vote
1 answer
1k views

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] ...
rg6's user avatar
  • 195
2 votes
2 answers
906 views

Dual-port RDMA Throughput Issue

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
Mohammad Hedayati's user avatar
1 vote
2 answers
837 views

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Good day, and my disclaimer that I am no IB expert and have not worked on IB directly yet. My question seems a trivial one, yet I could not find anything in google that answers it. Question: In ...
BeGa's user avatar
  • 21
-1 votes
1 answer
234 views

SFP+ Connector Won't Latch

We're trying to connect a Cisco Meraki MS225-48 switch to a Mellanox SN2100 switch using this breakout cable. However, the SFP+ end of the cable does not physically latch into the Meraki SFP+ ports (i....
Whimsical Seaplane's user avatar
0 votes
1 answer
1k views

Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA

SOLVED I got troubles setting up infiniband software from OpenSUSE repository. The same card works fine with SLES12 + Mellanox OFED stack. Installing everything containing 'infiniband' from yast I ...
catbus's user avatar
  • 1
1 vote
0 answers
2k views

Mellanox MT27500 Link to Port 1 is Down

I have Mellanox MT27500 dual port card which is installed on HP SL250s. Ubuntu 14.04.5 with 3.13.0-24-generic kernel is running on the server. Both port1 type and port 2 type are defined as eth. I ...
Gpost's user avatar
  • 11
2 votes
2 answers
5k views

Installing Mellanox ConnectX-2 10gbe PCIe Card on Centos 6

I just installed a Mellanox ConnectX-2 10gbe PCIe x8 Card into my server running CentOS 6.7. On boot, dmesg shows the mlx4_core driver being loaded automatically however I see no eth1 device ...
Crash Override's user avatar
2 votes
5 answers
2k views

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
John's user avatar
  • 23
0 votes
1 answer
2k views

VMware infiniband configuration

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup: The first card should work ...
Tigran Baluyan's user avatar
0 votes
2 answers
1k views

Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?

I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat. Now I'm running MPI tests to verify that InfiniBand is ...
Grigory's user avatar
  • 167
1 vote
1 answer
460 views

InfiniBand RDMA latency test on Xen's dom0 crashes

The short story: while setting up InfiniBand connection between two servers, I cannot complete the RDMA latency test. It crashes even with breaking the ssh connection. The long story. The first ...
Grigory's user avatar
  • 167
3 votes
1 answer
3k views

Is it possible to update Supermicro ConnectX-3 Firmware with newer firmware from Mellanox?

I have a Supermicro server with a X9DRT-HIBFF motherboard installed. It has on-board Infiniband based on the ConnectX-3 chipset from Mellanox. Supermicro supply firmware version 2.10.0700 and ...
hookenz's user avatar
  • 14.7k
4 votes
1 answer
4k views

How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?

I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server. What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines. ...
hookenz's user avatar
  • 14.7k
0 votes
1 answer
633 views

Cannot use SDP in Debian Squeezy

I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this: mlx4_ib # Mellanox ConnectX cards #ib_mthca # some mellanox ...
Ulf's user avatar
  • 387
1 vote
0 answers
1k views

openstack, bridging, netfilter and dnat

In a recent upgrade (from Openstack Diablo on Ubuntu Lucid to Openstack Essex on Ubuntu Precise), we found that DNS packets were frequently (almost always) dropped on the bridge interface (br100). ...
cas's user avatar
  • 6,793