0

I am collecting numbers for monitoring HPC servers and am debating the policy for handing out memory (overcommit or not). I wanted to show users a number on how much virtual memory their processes (the whole machine) requested vs. how much was actually used.

I thought I'd get the interesting values from /proc/meminfo using the fields MemTotal, MemAvailable, and Committed_AS. The latter is supposed to show how much memory has been committed to by the kernel, a worst-case number of how much memory would really be needed to fulfill the running tasks.

But Committed_AS is obviously too small. It is smaller than the currently used memory! Observe two example systems. One admin server:

# cat /proc/meminfo 
MemTotal:       16322624 kB
MemFree:          536520 kB
MemAvailable:   13853216 kB
Buffers:             156 kB
Cached:          9824132 kB
SwapCached:            0 kB
Active:          4854772 kB
Inactive:        5386896 kB
Active(anon):      33468 kB
Inactive(anon):   412616 kB
Active(file):    4821304 kB
Inactive(file):  4974280 kB
Unevictable:       10948 kB
Mlocked:           10948 kB
SwapTotal:      16777212 kB
SwapFree:       16777212 kB
Dirty:               884 kB
Writeback:             0 kB
AnonPages:        428460 kB
Mapped:            53236 kB
Shmem:             26336 kB
Slab:            4144888 kB
SReclaimable:    3863416 kB
SUnreclaim:       281472 kB
KernelStack:       12208 kB
PageTables:        38068 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    24938524 kB
Committed_AS:    1488188 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      317176 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:     90112 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      144924 kB
DirectMap2M:     4988928 kB
DirectMap1G:    13631488 kB

This is roughly 1.5G committed vs. 2.5G being in use without caches. A compute node:

ssh node390 cat /proc/meminfo
MemTotal:       264044768 kB
MemFree:        208603740 kB
MemAvailable:   215043512 kB
Buffers:           15500 kB
Cached:           756664 kB
SwapCached:            0 kB
Active:         44890644 kB
Inactive:         734820 kB
Active(anon):   44853608 kB
Inactive(anon):   645100 kB
Active(file):      37036 kB
Inactive(file):    89720 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      134216700 kB
SwapFree:       134216700 kB
Dirty:                 0 kB
Writeback:           140 kB
AnonPages:      44918876 kB
Mapped:            52664 kB
Shmem:            645408 kB
Slab:            7837028 kB
SReclaimable:    7147872 kB
SUnreclaim:       689156 kB
KernelStack:        8192 kB
PageTables:        91528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    345452512 kB
Committed_AS:   46393904 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      797140 kB
VmallocChunk:   34224733184 kB
HardwareCorrupted:     0 kB
AnonHugePages:  41498624 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      312640 kB
DirectMap2M:     7966720 kB
DirectMap1G:    262144000 kB

This is around 47G used vs. 44G committed. The system at question is a CentOS 7 cluster:

uname-a
Linux adm1 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

On my Linux desktop using a vanilla kernel, I see more 'reasonable' numbers with 32G being committed compared to 15.5G being in use. On a Debian server I see 0.4G in use vs. 1.5G committed.

Can someone explain this to me? How do I get a correct number for the committed memory? Is this a bug in the CentOS/RHEL kernel that should be reported?

Update with more data and a comparison between systems

A listing of used/committed memory for various systems I could access, with a note about the kind of load:

  • SLES 11.4 (kernel 3.0.101-108.71-default)
    • 17.6G/17.4G, interactive multiuser HPC (e.g. MATLAB, GIS)
  • CentOS 7.4/7.5 (kernel 3.10.0-862.11.6.el7 or 3.10.0-862.14.4.el7)
    • 1.7G/1.3G, admin server, cluster mgmt, DHCP, TFTP, rsyslog, …
    • 8.6G/1.7G, SLURM batch system, 7.2G RSS for slurmdbd alone
    • 5.1G/0.6G, NFS server (400 clients)
    • 26.8G/32.6G, 16-core HPC node loaded with 328 (need to talk to the user) GNU R processes
    • 6.5G/8.1G, 16-core HPC node with 16 MPI processes
  • Ubuntu 16.04 (kernel 4.15.0-33-generic)
    • 1.3G/2.2G, 6-core HPC node, 6-threaded scientific application (1.1G RSS)
    • 19.9G/20.3G, 6-core HPC node, 6-threaded scientific application (19G RSS)
    • 1.0G/4.4G, 6-core login node with BeeGFS metadata/mgmt server
  • Ubuntu 14.04 (kernel 3.13.0-161-generic)
    • 0.7G/0.3G, HTTP server VM
  • Custom build (vanilla kernel 4.4.163)
    • 0.7G/0.04G, mostly idle Subversion server
  • Custom build (vanilla kernel 4.14.30)
    • 14.2G/31.4G, long-running desktop
  • Alpine (kernel 4.4.68-0-grsec)
    • 36.8M/16.4M, some (web) server
  • Ubuntu 12.04 (kernel 3.2.0-89-generic)
    • 1.0G/7.1G, some server
  • Ubuntu 16.04 (kernel 4.4.0-112-generic)
    • 0.9G/1.9G, some server
  • Debian 4.0 (kernel 2.6.18-6-686, 32 bit x86, obviously)
    • 1.0G/0.8G, some reliable server
  • Debian 9.5 (kernel 4.9.0-6)
    • 0.4G/1.5G, various web services, light load, obviously
  • Debian 9.6 (kernel 4.9.0-8-amd64)
    • 10.9G/17.7G, a desktop
  • Ubuntu 13.10 (kernel 3.11.0-26-generic)
    • 3.2G/5.4G, an old desktop
  • Ubuntu 18.04 (kernel 4.15.0-38-generic)
    • 6.4G/18.3G, a desktop

SUnreclaim for SLES and CentOS rather large … 0.5G to 1G not uncommon, more if not flushing caches from time to time. But not enough to explain the missing memory in Committed_AS. The Ubuntu machines typically have below 100M SUnreclaim. Except the 14.04 one, that one has small Committed_AS and 0.4G SUnreclaim. Bringing kernels in order is tricky, as the 3.10 kernel from CentOS has many features of 4.x kernels backported. But there seems to be a line between 4.4 and 4.9 that affected the strangely low values of Committed_AS. The added servers from some of my peers suggest that Committed_AS also delivers strange numbers for older kernels. Was this broken and fixed multiple times?

Can people confirm this? Is this just buggy/very inaccurate kernel behaviour in determining the values in /proc/meminfo, or is there a bug(fix) history?

Some of the entries in the list are really strange. Having one slurmdbd process with a RSS of four times Committed_AS cannot be right. I am tempted to test a vanilla kernel on these systems with the same workload, but I cannot take the most interesting machines out of production for such games.

I guess the answer to my question is a pointer to the fix in the kernel commit history that enabled good estimates in Committed_AS again. Otherwise, please enlighten me;-)

  • I recently edited the Question with some more data (in two steps). The picture leaves me somewhat confirmed in my belief that there is something wrong with the Committed_AS estimate in, well, some ranges of kernel versions. – drhpc Nov 30 '18 at 13:41
0

Those boxes are not under significant memory pressure. Neither is paging (SwapFree). Second box is ~47 GB committed of 250 GB total. 200 GB is a lot to play with.

In practice, keep increasing the size of the workload until one of these happens:

  • User (application) response time degrades
  • Page out rate is higher than you are comfortable with
  • OOM killer murders some processes

Relationships between the memory counters is unintuitive, varies greatly between workloads, and probably is only really understood by kernel developers. Don't worry about it too much, focus on measuring obvious memory pressure.


Other descriptions of Comitted_AS, on the linux-mm list a while ago, emphasize it is an estimate:

Committed_AS: An estimate of how much RAM you would need to make a
              99.99% guarantee that there never is OOM (out of memory)
              for this workload. Normally the kernel will overcommit
              memory. That means, say you do a 1GB malloc, nothing
              happens, really. Only when you start USING that malloc
              memory you will get real memory on demand, and just as
              much as you use. So you sort of take a mortgage and hope
              the bank doesn't go bust. Other cases might include when
              you mmap a file that's shared only when you write to it
              and you get a private copy of that data. While it normally
              is shared between processes. The Committed_AS is a
              guesstimate of how much RAM/swap you would need
              worst-case.
  • This is not about the machines being in trouble. This is for monitoring resource usage of scientific computing jobs occupying the whole node (hence not a single process to work with). The usage pattern differs a bit from typical server. Since the memory is actually meant to be used by the user jobs, it can be normal that a job requests 300G or more and then starts filling that with data. I want to be able to tell the users that they allocated too much before OOM killing starts, or tell them that they might hit such a limit when scaling problem size. – drhpc Nov 29 '18 at 18:39
  • About Committed_AS being an estimate: I do not expect it to be exact. It can be off by quite some margin. But that it can be about half of the used non-reclaimable memory, so obviously wrong, led me to assume that there is something that I am missing. Any idea apart from reading the kernel source code and trying to get an answer on LKML? – drhpc Nov 29 '18 at 18:41
  • Workload dependent, please add specifics to your question about if the dataset is a file, shared memory (many DBMS systems use shared memory buffers), private pages, if huge pages are in use... You can read or memory map a TB sized file on such a box, and being file backed isn't likely to use much memory. But if you slurp 1 TB into private pages it is going to OOM. – John Mahowald Nov 29 '18 at 18:57
  • That's my point: I want to know if e.g. mappings of big files are counted in Committed_AS (I think they should). See my update of the question regarding more examples of servers and some workload indication. – drhpc Nov 30 '18 at 12:00
  • Why would file mappings count against committed? Flush to disk and free the memory, similar to file system cache. You could parse /proc/*/*maps for the details, but a fair accounting of shared memory is not trivial. In practice, migrate users to smaller nodes until the "free" metric is near zero. – John Mahowald Dec 2 '18 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.