0

Just booted Arch Linux on a Ryzen Threadripper 1950X server that I built and use in my company. Please don't close this question. It is relevant for anyone using Linux on Threadripper and running NUMA-aware software. In fact I've found the answer and would like to post it.

numactl --hardware reports:

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 128876 MB
node 0 free: 2114 MB
node distances:
node   0 
  0:  10 

Threadripper is a NUMA platform and the memory latency increases from ~90ns to ~140ns when crossing dies (source: https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/). I would like NUMA-aware software to apply its optimizations.

Is this a known problem? How can I fix it?

My hardware:

  • Ryzen Threadripper 1950X
  • ASRock Fatal1ty X399 Professional Gaming
  • Samsung DDR4-2400 ECC UDIMM (8 DIMMs)

My software:

  • Linux 4.12.13 (Arch Linux) (various CONFIG_NUMA kernel options enabled)

closed as off-topic by fuero, kasperd, Greg Askew, Ward - Reinstate Monica, yagmoth555 - GoFundMe Monica Oct 8 '17 at 10:04

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions on Server Fault must be about managing information technology systems in a business environment. Home and end-user computing questions may be asked on Super User, and questions about development, testing and development tools may be asked on Stack Overflow." – fuero, Greg Askew, Ward - Reinstate Monica, yagmoth555 - GoFundMe Monica
If this question can be reworded to fit the rules in the help center, please edit the question.

  • Check. Your. Bios. Basically: it works as set up. – TomTom Oct 7 '17 at 13:21
  • Dudes, this is a business environment. This is the most cost efficient workstation environment, even though the boards are named gaming. I am buying some of those for Quantitative Analysis work - 16 sores, 64gb ECC memory, for a price intel dreams of. Gamers (who comes up with those names, damn) actually will use Ryzen - easier fast memory and a lot cheaper. ThreadRipper is really workstation level stuff. – TomTom Oct 7 '17 at 13:23
  • Fucking idiots. Really. – Aleksandr Dubinsky Oct 13 '17 at 10:38
  • 1
    The correct answer is to set "Memory Interleave" to "Channel" in the BIOS. Linux then sees multiple NUMA nodes. See reddit.com/r/Amd/comments/6stemx/… – Aleksandr Dubinsky Oct 21 '17 at 9:38
2

Check your Bios.

Threadripper has two work modes. Your bios sets it in Unified Memory Access Mode.

Not using that particualar board - so no ideea how they name the setting.

Note that the platform is quite new - there are a lot of issues with it through the bios. Can be this gets reset or does not work properly on that parciular bios.

  • Thank you for the tip. It looks like the ASRock does not have this option (yet). It's a shame, because I found this to be the best board given its 10GBase-T Ethernet and U.2 port. – Aleksandr Dubinsky Oct 8 '17 at 9:32
  • I wnent with the MSI X399 for our business. I like 2x M.2 and the 10G I can always add - Intel has great cards. – TomTom Oct 8 '17 at 12:57
  • What is the exact name of the setting in MSI's BIOS? Which board do you have? I'm trying to talk to ASRock support, and point out that a competitor has the feature. – Aleksandr Dubinsky Oct 12 '17 at 16:21
  • No idea. I uhave one MSI X399 Carbon board - prototype for the workstations we buy here now. Never even looked for this setting - the bios i still VERY flaky. Does not recognize my 2666 ECC Ram, does not properly wake up from hibernation. Waiting another month to see how it goes. – TomTom Oct 12 '17 at 16:42
  • Oh. Was there another board where you've seen this setting? Another platform? Only references I've found for configuring NUMA on Threadripper is Ryzen Master. pcper.com/reviews/Processors/… – Aleksandr Dubinsky Oct 12 '17 at 18:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.