NUMA CPU numbering in Linux

Question

I have access to two NUMA servers. One of them is Dell R720 and has these CPUs:

$ cat /proc/cpuinfo |grep Xeon|sort|uniq -c
     24 model name  : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz

The other is a HPE DL360 Gen8 and has these CPUs:

$ cat /proc/cpuinfo |grep Xeon|sort|uniq -c
     24 model name  : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz

At work where we have many HPE Gen9 servers, I have been used to CPU numbering (socket0, socket1, socket0 HyperThreads, socket1 HyperThreads). It seems the HPE DL360 Gen8 uses this numbering:

$ cat /proc/cpuinfo |grep physical.id|uniq -c
      6 physical id : 0
      6 physical id : 1
      6 physical id : 0
      6 physical id : 1

But the Dell R720 server uses different numbering:

$ cat /proc/cpuinfo |grep physical.id|uniq -c
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1
      1 physical id : 0
      1 physical id : 1

My question is, what causes this difference? The servers have two slightly different kernel versions:

Dell R720:

$ uname -a
Linux dell 4.10.0-33-generic #37~16.04.1-Ubuntu SMP Fri Aug 11 14:07:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

HPE DL360 Gen8:

$ uname -a
Linux hpe 4.11.0-14-generic #20~16.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Is this caused by different kernel versions? Or by different CPUs? Or by different motherboards / BIOSes?

Edit: I updated the kernels on both machines and rebooted, so now both machines use exactly the same kernel version. Nevertheless, the difference is still there.

Answer 1

Stop the grepping and uniq and run lscpu and lstopo --of png > server.png and visualize the results...

[root@LA_Specialty ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz
Stepping:              4
CPU MHz:               3501.000
BogoMIPS:              7013.88
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-5,12-17
NUMA node1 CPU(s):     6-11,18-23

Answer 2

lscpu [1] will more concisely express the numa layout of each system. lstopo [2] gives a hierarchic view of the processor relations.

Enumeration is determined by CPU + BIOS + kernel. At a high level, the motherboard has different sockets for 0 and 1, it always launches 0. The CPU in socket 0 then starts core 0 at a particular address and launches the BIOS, which then enumerates the logical CPUs on this chip and additional CPUs (possibly not in that order)[3]. The BIOS passes the enumeration data to the OS as needed, but the OS is free to number the CPUs however it wants (imagine what hotplugging CPUs does to numbering).

If you're worried about affinities and caching, the apicid is a useful number to use. The bitfield should be defined such that memory/caches that are closest together have apicids that are numerically close by ordering the bits Socket | Core | SMT. The widths of these fields are not fixed though, so you can't count on the LSB to always mean SMT, it could have no SMT and this bit is part of the core id.