Questions tagged [numa]
Non-Uniform Memory Access is what it stands for. For x86 architectures this is the method used to handle memory architectures where each processor has local memory and accessing another processor's memory is appreciably more expensive.
71
questions
3
votes
0
answers
55
views
NUMA CPU affinity issue
I have a Ubuntu server with 22.04 OS. I encountered an issue with NUMA affinity bound.
1.) Server has two NUMA nodes accordingly.
server-1:# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 ...
- 87
36
votes
1
answer
4k
views
Unexpected and unexplained slow (and unusual) memory performance with Xeon Skylake SMP
We've been testing a server using 2x Xeon Gold 6154 CPUs with a Supermicro X11DPH-I motherboard, and 96GB RAM, and found some very strange performance issues surrounding memory when compared to ...
CommunityBot
- 1
0
votes
0
answers
104
views
Optimizing CPU and Memory use on Multi-socket NUMA server
I am building a server that will be used by interns in my company to connect to and perform some tasks. The server is running AlmaLinux 9 and using KVM to run Windows 11 guests.
The server has 2 CPU ...
- 1
1
vote
1
answer
743
views
Should I worry about "Unknown NUMA node; performance will be reduced"?
I have a NUMA system with two Xeon E5-2680 v3's. System is running Debian 11.5 (Proxmox VE 7.3-3). When making sure that NUMA was enabled I noticed this in my dmesg:
root@(host):~# dmesg | grep -i ...
1
vote
2
answers
273
views
NPS4 on a Threadripper 3960x gives two nodes with no memory at all
I set my 3960x to NPS4 (Nodes Per Socket: 4) mode to experiment with NUMA on Linux. My system has 4 32 GiB DIMMs across 4 channels, so I expected each of the 4 nodes to get one. Instead, nodes 1 &...
- 119
0
votes
1
answer
221
views
Use of dual-port NIC
I was wondering, what is the purpose of a dual-port NIC on a blade compute unit, in the context of a datacenter rack, like which of these configurations is more in line with modern datacenter ...
- 11.7k
0
votes
1
answer
228
views
Uneven cpu utilization linux dual socket server
Recently I noticed on two of our servers strange distribution of tasks.
Both servers are dual cpu EPYC 7402 physically the same platforms, running the same tasks, differ in numa configuration, kernel ...
- 1
2
votes
1
answer
872
views
DELL PowerEdge R740xd - NUMA - memory performance
I have two physical database servers (both Windows Server 2016):
test server (5 years old):
DELL PowerEdge R730xd, 1x Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz (4C/8T), 192 GB RAM (12x 16GB PC4-...
- 1,108
2
votes
1
answer
1k
views
Recommended NUMA nodes per socket on dual AMD Epyc 7643 server with 1TB of RAM
What do you suggest to set the NUMA nodes per socket to? It is factory set to NPS1, but I'm not sure if this is the most optimal value. (We use this device for massively multithreaded bioinformatic ...
0
votes
1
answer
824
views
How to calculate the QuickPath Interconnect (QPI) bandwidth?
For Xeon E5-2697 v2, Intel lists:
Bus Speed = 8 GT/s
# of QPI Links = 2
According to Wikipedia, one must know the QPI frequency and link width to calculate the QPI bandwidth, but these don't seem to ...
- 187
1
vote
1
answer
253
views
Incorrect # of Hugepages in `numstat`
I asked a similar question years ago.
Now, my machine has four 1G hugepages and 256 2MB hugepages:
# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
4
# cat /sys/devices/...
- 33.3k
1
vote
1
answer
841
views
Latency of memory accesses via interconnectors
I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency.
I have obtained some benchmarks regarding one of the hardware ...
- 33.3k
2
votes
0
answers
408
views
Current single system image solutions
I'm designing a cluster for a small research institute. Since our computations require a large amount of memory, I'm looking for a solution that will allow our applications access to the whole memory ...
- 33
0
votes
4
answers
9k
views
Hyper-V: Not enough memory to start VM although there are plenty left
I'm having this error on my server: "Not enough memory in the system to start the virtual machine. Ran out of memory (0x8007000E)" when starting an 8 GB VMs on a 12 GB RAM FREE server.
Here is my set ...
2
votes
0
answers
684
views
VMware Cross-NUMA Performance penalty
I've been experiencing random unexpected slow-downs of a virtual SQL Server that I can't attribute to workload, storage or CPU (in fact it continued after the host was evacuated of other VMs).
I ...
- 21
4
votes
2
answers
4k
views
What does `numactl --localalloc` do?
What does numactl --localalloc do? Does it bind program allocation to only the single node that numactl --localalloc was run on? Or does it ensure that whenever the program allocates its given memory ...
- 143
2
votes
2
answers
2k
views
NUMA placement failed, performance might be affected
I'm running SuperMicro 6048R-E1CR36H Storageserver on Ubuntu Xenial 16.04.03 LTS and Xen
Kernel: 4.4.0-97-generic
Xen: xen-hypervisor-4.6-amd64:amd64/xenial-security 4.6.5-0ubuntu1.2
Problem: when ...
CommunityBot
- 1
3
votes
0
answers
727
views
Consecutive CPU numbering on multi-socket NUMA Linux system
I've noticed that CPUs are numbered by Linux according to quite different schemes on different multi-socket NUMA systems.
I mean the CPU numbers you use in kernel parameters such as isolcpus= or when ...
- 995
0
votes
1
answer
2k
views
how libvirt cputune config goes to qemu-kvm
I am porting xml libvirt config to qemu executable arguments. Simply I have .xml vm config from virsh edit vm-name and I want to run the same vm machine by executing qemu-system-x86_64 with special ...
- 16
0
votes
1
answer
694
views
NUMA 1 not used on SQL Server 2008 R2 on Windows Server 2008 R2 on ESXi
We've got Microsoft SQL Server 2008 R2 running on Windows Server 2008 R2, virtualized on VMWare ESXi 6.7.0.
The server has 2 CPU sockets with 22 cores each.
The VM is allocated 8 sockets with 3 cores ...
- 253
0
votes
1
answer
316
views
Incorrect # of Free Hugepages in `numastat`
$ numastat -vm
Per-node system memory usage (in MBs):
Node 0 Total
--------------- ---------------
MemTotal 32464.24 32464....
- 33.3k
1
vote
1
answer
978
views
Virtualbox performance on NUMA host (AMD Epyc)
At a small software development house, we virtualize our build servers (using VirtualBox) so that they can be easily backed up, snapshotted and the like. We've recently bought a new server using an ...
- 33.3k
2
votes
0
answers
40
views
Slurm - Does it maintain ccNUMA?
Does a SLURM cluster control, maintain or enforce Cache Coherence across the Nodes? Is it a configuration property, or does something like this not exist? I can't find anything inside the docs.
- 271
1
vote
0
answers
376
views
Linux NFS server NUMA affinity - pool_mode
From the Linux kernel-parameters.txt I have seen that it is possible to change the NFS pool behaviour to have the nfsd threads bound to NUMA zones.
The parameter in question is
sunrpc.pool_mode
and ...
- 4,305
2
votes
1
answer
1k
views
NUMA node interleaving doesn't work for MariaDB
Please help activating NUMA node interleaving for MariaDB.
Using MariaDB 10.1.21 on CentOS Linux release 7.3.1611.
Per instructions on https://mariadb.com/kb/en/mariadb/systemd/ added /etc/systemd/...
- 9,558
1
vote
0
answers
602
views
openstack shared PCI between numa with SR-IOV
I'm building SR-IOV supported compute node on HP 360g8 hardware and i have Qlogic interface card, my compute node has 32 core & 32GB memory.
Problem:
when i launch vm-1 (with flavor 16 vCPU core)...
- 672
9
votes
1
answer
722
views
How would I know if I should be disabling virtual NUMA on VMs running older applications?
From VMware's docs:
Virtual NUMA topology is available to hardware version 8 virtual
machines and is enabled by default when the number of virtual CPUs is
greater than eight. You can also ...
- 330
1
vote
1
answer
71
views
Opteron 6274 Cache Differs from Manufacturer's Specs in Windows Server 2016
I just upgraded a DL585 g7 server by replacing its Opteron 6172 CPUs with 4 Opteron 6274 CPUs. Every source I read says that the Opteron 6274s are supposed to have 8x2MB of L2 Cache and 16MB of L3 ...
2
votes
0
answers
230
views
Bad performance on better hardware
I have postgresql streaming replication on 2 hosts and I've faced with the problem of different performance compared between two servers. It looks like all sql queries on one host are slower on 70-90% ...
- 243
1
vote
0
answers
2k
views
Memcpy bandwidth ~1.6x faster on 1 vs 2 socket Intel Scalable (Skylake)?
I'm in the process of porting a complex performance oriented application to run on a new dual socket machine. I encountered some performance anomalies while doing so and, after much experimentation, ...
- 121
1
vote
1
answer
1k
views
ZFS on Linux and KVM: NUMA nodes for host
I am interested in using KVM images on zvols under ZFS on Linux, on a multi-socket system. I am wondering how I should pin NUMA nodes so as to maximize the benefits of ZFS ARC cache for all KVM images ...
- 50.7k
4
votes
0
answers
365
views
Hyper-V 2016 DDA issue: a VM boot fails when a drive is connected to an opposite NUMA node
Brief description: the host is Windows Server 2016 Datacenter Build 14393 (UEFI) running Hyper-V role. SR-IOV and Intel VT are enabled. 4x Intel® Optane ™ SSD 900P 280GB PCIe NVMe 3.0 drives are ...
- 3,918
0
votes
1
answer
5k
views
Does disabling "numa interleave" from bios cause memory page-out(when cpu-1 has no free memory left) to hdd on all dual-cpu systems?
For an example system of a dell dual 4114 silver with 24GB per CPU; how would it work if my application allocates 24 GB at once? Should I be concerned about write-life of my SSD because of pagefile ...
- 248k
7
votes
2
answers
9k
views
NUMA CPU numbering in Linux
I have access to two NUMA servers. One of them is Dell R720 and has these CPUs:
$ cat /proc/cpuinfo |grep Xeon|sort|uniq -c
24 model name : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
The other ...
CommunityBot
- 1
5
votes
1
answer
456
views
Server computational slowdown when RAM is used extensively
I have problem with server slowdowns in very specific scenario. The facts are:
1) I use computational application WRF (Weather Research and Forecast)
2) I use Dual Xeon E5-2620 v3 with 128GB RAM (...
- 71.3k
0
votes
1
answer
3k
views
Ryzen Threadripper CPU does not report multiple NUMA nodes [closed]
Just booted Arch Linux on a Ryzen Threadripper 1950X server that I built and use in my company. Please don't close this question. It is relevant for anyone using Linux on Threadripper and running NUMA-...
0
votes
1
answer
3k
views
Change the NUMA node where a PCIe device is attached
Modern servers using multiple physical CPU sockets have NUMA.
PCIe devices are attached to one specific NUMA node as the PCIe controller is embedded in the physical CPU chip.
Is it possible to change ...
- 198k
4
votes
3
answers
1k
views
HP DL360p with Intel E5-2630 NUMA Capable?
I couldn't find any information via search to determine this via the Intel or HP website. Can you enable NUMA nodes on this? How can you tell the side of each node if so? Is it always best practice to ...
- 18.4k
3
votes
1
answer
2k
views
Is a machine with a single NUMA node, actually a regular (non-NUMA) system?
First, let's check I got the fundamentals right:
As I understand it, NUMA systems are a (asymmetric) network of NUMA nodes, where a NUMA node is usually (but not always) a physical CPU package. In a ...
- 50.7k
2
votes
0
answers
103
views
find out NUMA locality of process RAM
I am doing an application benchmark with multiple instances of the same application. I found out that pinning their processes (with sched_setaffinity under Linux, with TaskManager under Windows) to ...
- 387
0
votes
0
answers
350
views
CPU & Memory Reservation in vSphere & Numa concept
by googling/studying vSphere documentation I have found the possibility to use "Reservation" concept in vSphere.
What it is not clear to me is : CPU and Memory reservations are configured separately ...
- 101
44
votes
1
answer
62k
views
Is there still a use for irqbalance on modern hardware?
This question has been asked before, but I believe that the world has changed enough for it to be asked again.
Does irqbalance have any use on today’s systems where we have NUMA-capable CPUs with ...
CommunityBot
- 1
1
vote
0
answers
137
views
Writing a Numa load balancer
I originally asked this question on StackOverflow, but as there came no answers, and this question is more about how to configure a server, this question might be more suited on ServerFault.
I have ...
- 227
2
votes
2
answers
235
views
Is NUMA always completely NUMA or are there also hybrid systems?
I am working on a high-end server application where performance is critical. Given that servers are often employ NUMA-architectures, the server application also uses NUMA-aware memory allocation ...
- 21
0
votes
1
answer
355
views
What is NUMA node limit in modern Windows OS
What is the highest number of NUMA nodes in Windows 10 / Server 2012? In Windows 7, it was documented that OS supports only up to 4 NUMA nodes1 but with modern systems supporting 320 logical ...
- 102k
-1
votes
2
answers
2k
views
Hyper-V NUMA setting
My machine is running on Window 2012 R2, with 48 Logical processors and 240GB RAM.
I have create a VM with 48 processors and 230GB ram.
I am double about the NUMA nodes I should create and memory ...
- 211
6
votes
1
answer
2k
views
DL380 Gen9 which processor and what impact on NUMA
We are about to build a new server and I was wondering what would the impact on NUMA with the options available.
The system will be built using a Proliant DL380 Gen9 8SFF which has 2 sockets, and 12 ...
- 198k
0
votes
1
answer
341
views
HyperV memory per NUMA node
I have some issue with memory allocation on my Hyper-V 2012 R2 server. Server has 16GB of RAM, with 2 x 12 core CPUs. When I run Get-VMHostNumaNode, I am getting following results:
NodeId ...
2
votes
1
answer
1k
views
Linux Opteron system appears to be UMA but should be NUMA
According to numactl, this dual CPU Opteron box is UMA rather than the expected NUMA:
$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 65534 ...
- 675
0
votes
1
answer
99
views
NUMA processor definition
NUMA, non-uniform memory access designates a symetric multi-processing system where processors are grouped into nodes, with each group sharing some level of memory, so that memory access on same node ...
- 102k