Questions tagged [mellanox]
Mellanox Technologies Ltd. is an Israeli and American multinational supplier of computer networking products using InfiniBand and Ethernet technology
38
questions
0
votes
0
answers
16
views
Number of receiving queues is double the number of cores of my server?
I'm using Mellanox ConnectX-5 100 GB NIC, Linux kernel 5.15 and my server has 32 cores (SMT disabled). But in the /sys/net//queues, the number of rx queues is 64, which is double the number of cores ...
- 1
0
votes
0
answers
22
views
How does the mlx5 driver use the IOVA generated by IOMMU as DMA address?
I have enabled IOMMU on the physical machine. I expect the RDMA NIC to use the IOVA allocated by the IOMMU module for DMA after enabling IOMMU. However, in reality, the RDMA NIC does not use the IOVA ...
- 1
-1
votes
1
answer
305
views
Very poor performance on Kubernetes with 100GbE network
we are using ConnectX-5 100GbE ethernet cards on our servers which is connected one to each other trough the mellanox switch. And we are using weavenet cni plugin on our Kubernetes cluster. When we ...
- 141
0
votes
0
answers
29
views
Unable to clear virtual mac address on Mellanox ConnectX3 HP 546FLR
I recently had some problems with setting up bond interfaces during deployment with canonical maas. Two server stuck with the same mac address as virtual mac address and I have no idea where I can set ...
1
vote
0
answers
380
views
LinuxPTP slave not using master and sync timeout tx announce timeout
And:
I'm using G8316 as swtich with ptp transparent enable and Mellanox 354 adapter.
Ports using LACP bonding and trunk.
First: Server A ptp4l -m -l 7 -i xxxx -i xxxx -S
I can capture ServerA ...
- 111
0
votes
1
answer
721
views
Only getting 25Gb/s instead of 100Gb/s from a Mellanox switch running Cumulus Linux
I have a Mellanox 100gb/s switch (running Cumulus Linux 4.1) that I use for connecting multiple servers, each with a Mellanox ConnectX 5 100gb/s card. These servers connect to the switch via a DAC ...
- 535
0
votes
1
answer
720
views
K8s nodeLocalDns pod times out connecting to coreDns after upgrading base os to ubuntu20.04 ConnectX-4 card
Team,
I have Mellanox Nic ConnectX-4 on a k8s worker node and it hosts a nodeLocal dns pod on it. The nodeLocalDns pod is timing out when trying to connect to coreDns service on k8s cluster.
Same ...
- 119
1
vote
1
answer
747
views
X710 and mlx5_core together on same server
I have a amd server with centos 7.9
I am trying to using the the kernel driver for 2 cards:
Ethernet Controller X710 for 10GbE SFP+ 1572 --> i40e
MT27800 Family [ConnectX-5] 1017' --> mlx5_core
...
- 113
0
votes
0
answers
919
views
Infiniband adapter down
edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2:
I am not able to get my Infiniband adapter working.
The output of ibstat states that it is down:
CA 'mlx5_0'
...
- 51
2
votes
0
answers
267
views
Infiniband fabric with 3 nodes - newbie
I am trying to connect 3 HP z840 workstations using:
Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s
Description of ...
- 121
1
vote
1
answer
158
views
Infiniband OpenSM N-to-N port routing configuration
I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch.
My RDMA application runs as one process per NUMA ...
- 121
2
votes
2
answers
766
views
Infiniband drivers : OFED or distro included?
I'm setting up a Linux cluster with infiniband network, and I'm quite a newby in infiniband wolrd, any advice is more than welcome !
We are currently using Mellanox OFED drivers, but our infiniband ...
- 33
3
votes
0
answers
1k
views
Why isn't SMB Direct working over RDMA?
In a Windows lab environment, I'm trying to test performance of SMB Direct using Mellanox RDMA adapters with global pause. I wanted to test first with Global Pause for simplicity before switching to ...
- 200
4
votes
0
answers
1k
views
How to make systemd monitor processes started using a shell script in the service file?
I have written a service file in Linux that calls a script which starts multiple processes with one as primary and others as secondary.
SERVICE FILE:
[Unit]
Description=TEST service
Requires=abc....
- 41
1
vote
1
answer
136
views
A way to configure DMZ zone on Ethernet Mellanox Switch
Can't find any information is it possible at all and how to configure DMZ zone on Mellanox MSN2100 switch.
If I setup ACL disabling traffic from DMZ to LAN, I can't receive response to requests made ...
- 113
0
votes
1
answer
840
views
Infiniband P_Keys and the Linux kernel
I'm trying to configure Infiniband partitions between two Debian based Linux hosts running the 4.15 kernel and A Mellanox SX6036 switch. I've setup a "DMZ" partition on the swich using a PKey of ...
- 3,624
1
vote
0
answers
389
views
Missing Azure accelerated networking interfaces while VMs initializes
I'm running my application on top of Azure Ubuntu 16.04 VMs (Wrapped as a VHD image).
In several cases I have observed missing SRIOV associated VFs interfaces, while VMs were starting.
Need to mention ...
- 11
2
votes
1
answer
1k
views
Beegfs-client does not work with mellanox OFED driver
I am trying to configure and test beegfs with RDMA as explained in:
https://community.mellanox.com/s/article/howto-configure-and-test-beegfs-with-rdma?t=1570613300675
My test configuration:
OS: ...
- 21
0
votes
1
answer
141
views
Mellanox adapter on GCP instance
I'm trying to install MOFED on an instance (preferably with a GPU) and was wondering if anyone knows if a Mellanox adapter is available at any Google Cloud Platform VM. The VM instances that I've been ...
google-cloud-platform
0
votes
1
answer
445
views
Connecting Intel X722 SFP+ to Mellanox SX1012 QSFP via 4-split breakout cable using ubuntu
I have two servers running Ubuntu 18.04, each equipped with two SFP+ interfaces, each showing up in lshw like the following.
*-network:1
description: Ethernet interface
product: ...
- 1
0
votes
1
answer
611
views
Can not setup Mellanox connectx-2 in 40g speed
I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum ...
- 101
9
votes
3
answers
2k
views
QSFP28 - is there something like a crossover cable?
Can not find anything to that ;)
Building a Hyperconvergend Cluster as development system. This is for us state of the art.
Got 2 machines for a start, with a switch ordered to come in about 4 weeks....
- 51.5k
0
votes
1
answer
160
views
Managed switch for Mellanox MCX354A-FCBT cards?
I've inherited two Mellanox Infiniband environments. One on a LAN, and the other in a DMZ. Both IB networks have their own SX6005 unmanaged switches and a couple small desktop computers running the ...
- 3,624
1
vote
1
answer
806
views
Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS
I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9,
server1>lspci
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
...
- 185
2
votes
2
answers
671
views
Dual-port RDMA Throughput Issue
I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). ...
1
vote
2
answers
703
views
In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?
Good day,
and my disclaimer that I am no IB expert and have not worked on IB directly yet.
My question seems a trivial one, yet I could not find anything in google that answers it.
Question: In ...
- 21
-1
votes
1
answer
186
views
SFP+ Connector Won't Latch
We're trying to connect a Cisco Meraki MS225-48 switch to a Mellanox SN2100 switch using this breakout cable. However, the SFP+ end of the cable does not physically latch into the Meraki SFP+ ports (i....
0
votes
1
answer
978
views
Troubles with ibverbs in OpenSUSE Leap 42.2 for Mellanox ConnectX-3 HCA
SOLVED
I got troubles setting up infiniband software from OpenSUSE repository.
The same card works fine with SLES12 + Mellanox OFED stack.
Installing everything containing 'infiniband' from yast I ...
- 1
1
vote
0
answers
1k
views
Mellanox MT27500 Link to Port 1 is Down
I have Mellanox MT27500 dual port card which is installed on HP SL250s. Ubuntu 14.04.5 with 3.13.0-24-generic kernel is running on the server. Both port1 type and port 2 type are defined as eth. I ...
- 11
2
votes
2
answers
5k
views
Installing Mellanox ConnectX-2 10gbe PCIe Card on Centos 6
I just installed a Mellanox ConnectX-2 10gbe PCIe x8 Card into my server running CentOS 6.7. On boot, dmesg shows the mlx4_core driver being loaded automatically however I see no eth1 device ...
- 591
2
votes
5
answers
2k
views
set the Ca name on unmanaged mellanox switch
How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch?
I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or '...
- 23
0
votes
1
answer
2k
views
VMware infiniband configuration
I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup:
The first card should work ...
0
votes
2
answers
1k
views
Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?
I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat.
Now I'm running MPI tests to verify that InfiniBand is ...
- 167
1
vote
1
answer
427
views
InfiniBand RDMA latency test on Xen's dom0 crashes
The short story: while setting up InfiniBand connection between two servers, I cannot complete the RDMA latency test. It crashes even with breaking the ssh connection.
The long story. The first ...
- 167
3
votes
1
answer
3k
views
Is it possible to update Supermicro ConnectX-3 Firmware with newer firmware from Mellanox?
I have a Supermicro server with a X9DRT-HIBFF motherboard installed.
It has on-board Infiniband based on the ConnectX-3 chipset from Mellanox.
Supermicro supply firmware version 2.10.0700 and ...
- 14.4k
4
votes
1
answer
4k
views
How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?
I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server.
What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines.
...
- 14.4k
0
votes
1
answer
607
views
Cannot use SDP in Debian Squeezy
I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this:
mlx4_ib
# Mellanox ConnectX cards
#ib_mthca # some mellanox ...
- 387
1
vote
0
answers
1k
views
openstack, bridging, netfilter and dnat
In a recent upgrade (from Openstack Diablo on Ubuntu Lucid to Openstack Essex on Ubuntu Precise), we found that DNS packets were frequently (almost always) dropped on the bridge interface (br100). ...
- 6,743