4

I know it is possible to set the numa mode to "interleave" (see NB below) for a specific process using numactrl --interleave, but I'd like to know if it is possible to make this the system wide default (aka change the "system policy"). For example, if there a kernel boot flag to achieve this?

NB: here I'm talking about the kernel behavior which interleaves allocated pages across NUMA nodes - not the memory controller behavior setting at the BIOS level which interleaves cache lines across

  • Which specific OS and version are you using? – ewwhite Sep 10 '14 at 12:36
  • 1
    I've heard of premature optimization, but this sounds like premature un-optimization! I'm very curious as to what the use case is for this. – Michael Hampton Sep 11 '14 at 10:33
  • @MichaelHampton Some databases and large-memory applications recommend this (here, here and here). – ewwhite Sep 11 '14 at 10:40
  • 2
    At this point curiosity and the need to test different configurations. One of the favorite responses to any question on stackexchange sites seems to be "why would you want to even do that?!". Well one other common response is "You need to test that (configuration, idea, optmization, etc)". So to test things you need to configure them in different ways... – BeeOnRope Sep 11 '14 at 10:41
  • 1
    @ewwhite We are largely using RHEL, although I'm especially interested on options available on all modern Linux distros. – BeeOnRope Sep 11 '14 at 10:42
1

If using RHEL/CentOS/Fedora, I'd suggest using the numad daemon. (Red Hat paywall link).

While I don't have much use for the numactl --interleave directive, it seems you've determined that your workload requires it. Can you explain why this is the case in order to provide some better context?

Edit:

It seems that most applications that recommend explicit numactl definition either make a libnuma library call or incorporate numactl in a wrapper script.

For the numad side, there's a configuration option that can be specified on the command line or in /etc/numad.conf...

-K <0|1>
   This option controls whether numad keeps interleaved  memory  spread  across  NUMA  nodes,  or
   attempts to merge interleaved memory to local NUMA nodes.  The default is to merge interleaved
   memory.  This is the appropriate setting to localize processes in a  subset  of  the  system’s
   NUMA  nodes.   If  you  are running a large, single-instance application that allocates inter-
   leaved memory because the workload will have continuous unpredictable memory  access  patterns
   (e.g. a large in-memory database), you might get better results by specifying -K 1 to instruct
   numad to keep interleaved memory distributed.

Some say that trying this with something like numad -K 1 -u X, where X is 100 x core count, may help for this. Try it.

Also see HP's ProLiant Whitepaper on Linux and NUMA.

  • Basically I have a situation where it may be difficult to use numactrl explicitly to launch the process, so I was curious if there was a way to set the fault. The various policies, such as interleave do seem to exist (the kernel uses interleave at startup, for example) so it seemed that there would be some way to set the default. – BeeOnRope Sep 11 '14 at 9:43
  • I understand. Use numad instead. – ewwhite Sep 11 '14 at 9:59
  • Based on my limited understanding of numad, it doesn't seem like it can do what I want. It mostly moves memory around after the fact, trying to consolidate working sets that have become spread across nodes - but it doesn't seem like affects the initial allocation node. So it can only help me "de-interleave" but never increase the interleaving. – BeeOnRope Sep 11 '14 at 10:11
  • 1
    Let's say RHEL and x86 commodity servers (e.g., Dell poweredge stuff). – BeeOnRope Sep 11 '14 at 10:44
  • 1
    Let's say mostly 2-socket, but does it matter? – BeeOnRope Sep 11 '14 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.