10

There are numerous (hundreds?) of different measures for memory usage on a Linux machine, but what is a good heuristic/metric to use to help determine if a server needs more memory?

Some ideas:

  • Looking at MemTotal - Active - Inactive from /proc/meminfo as a measure of "wired" memory
  • Looking at the sum of RSS values from all processes in ps
  • Looking at Committed_AS in /proc/meminfo
1

6 Answers 6

14

Linux Kernel 4.20 added PSI, which stands for "pressure stall information". It gives you more insights why a machine is overloaded. And which resource is the bottleneck.

There are three new files under /proc/pressure:

  • /proc/pressure/cpu
  • /proc/pressure/memory
  • /proc/pressure/io

To quote from Tracking pressure-stall information concerning /proc/pressure/memory:

Its output looks like:

some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828
full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258

The some line is similar to the CPU information: it tracks the percentage of the time that at least one process could be running if it weren't waiting for memory resources. In particular, the time spent for swapping in, refaulting pages from the page cache, and performing direct reclaim is tracked in this way. It is, thus, a good indicator of when the system is thrashing due to a lack of memory.

The full line is a little different: it tracks the time that nobody is able to use the CPU for actual work due to memory pressure. If all processes are waiting for paging I/O, the CPU may look idle, but that's not because of a lack of work to do. If those processes are performing memory reclaim, the end result is nearly the same; the CPU is busy, but it's not doing the work that the computer is there to do. If the full numbers are much above zero, it's clear that the system lacks the memory it needs to support the current workload.


I have not access to a production server with Linux 4.20 yet, but here is a small experiment on my Desktop (which has no swap configured). Initially, I have no memory pressure at all (all counters are 0):

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Then I increased the memory usage until I eventually ran out of memory, which froze the machine until the OOM killed some processes. Before it froze, the pressure on the memory increased:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

some avg10=0.00 avg60=0.00 avg300=0.00 total=47047
full avg10=0.00 avg60=0.00 avg300=0.00 total=32839

some avg10=0.00 avg60=0.00 avg300=0.00 total=116425
full avg10=0.00 avg60=0.00 avg300=0.00 total=81497

some avg10=1.26 avg60=0.22 avg300=0.04 total=183863
full avg10=0.72 avg60=0.13 avg300=0.02 total=127684

Now, after the system has recovered, the memory pressure is again 0, and the total counters no longer increase:

$ cat /proc/pressure/memory 
some avg10=0.00 avg60=0.00 avg300=0.07 total=53910568
full avg10=0.00 avg60=0.00 avg300=0.02 total=27766222

...

$ cat /proc/pressure/memory 
some avg10=0.00 avg60=0.00 avg300=0.05 total=53910568
full avg10=0.00 avg60=0.00 avg300=0.00 total=27766222
2
  • Do you know how the total should be interpreted? Nov 29, 2023 at 17:30
  • 1
    @MikkoRantalainen I found this in the linked article (lwn.net/Articles/759781): The final number (total) is the total amount of time (in microseconds) during which processes were stalled. It is there to help with the detection of short-term latency spikes that wouldn't show up in the aggregated numbers. A system where a CPU is nearly always available but where occasional 10ms latency spikes are experienced may be entirely acceptable for some workloads, but not for others. For the latter group, the total count can be monitored to detect when those spikes are happening. Nov 29, 2023 at 19:09
2

There's no right answer to this.

Peter is correct in saying that the values you need to be looking at are reported in top and free (you can get the source code for the procps package which shows how to get these values from 'C' - but for scripts it's simpler to just run 'free')

If the system has unused memory (the first line of output from free) then it's unlikely to get much faster by adding more memory - but it might get faster by reducing the VFS cache pressure (keep stuff in cache longer).

Although there's no right answer, there's lots of wrong ones - you can't tell from userspace which pages are shared but accesed via different locations - looking at memory usage to determine how much memory is free just doesn't work.

As a starting point then you should be looking at the two values for free memory reported by 'free'

2

I have said this before, the best measure to get real time memory requirement is to observe the COmmitted_AS field in /proc/meminfo and comparing it over time to see how much memory you need.

Theoretically, if your Committed_AS is always over than (Memfree+swapfree) then you are fine. But if it is less than that and you accumulate your workload on the system over time, you are approaching towards a OOM situation. Committed_AS value determines how much memory is required to the system if all the memory requests were being granted to the system at this very instant.

Monitoring it is a good measure over time to see whether you need to increase RAM or you need to decrease workload.

6
  • Committed_AS is a theoretical estimate, it doesn't relate to memory pressure but to the possibility of an OOM condition that doesn't really happen in practice.
    – Tobu
    Oct 11, 2012 at 15:07
  • That's what I said when I said theoretically. It says that if all the memory allocations are granted at any point of time then how much memory is required so that the system doesn't go OOM under 99.99% situations. Oct 11, 2012 at 17:37
  • while Committed_AS is not a precise estimate, the rule of thumb "keep actual RAM > Committed_AS" has never failed for me. Sometimes I randomly go way over that amount (it's easy with VMs) and I can't see any measurable difference in how systems handle current workload with extra ram. Maximum possible workload will of course vary, Committed_AS is only a guesstimate on the current one.
    – Luke404
    Nov 13, 2013 at 17:53
  • 1
    if your Committed_AS is always over than (Memfree+swapfree) then you are fine. Shouldn't it be less than available memory? A larger committed value indicates a commitment that cannot be fulfilled in the worst-case scenario, right?
    – Isaac
    Aug 12, 2017 at 8:27
  • @Luke404 if you have more actual RAM than Committed_AS you have way more RAM than actually required. Obviously you won't get OOM situations in that case. My system has Committed_AS bigger than RAM+swap combined. Jan 6, 2018 at 11:55
2

Really it all depends on the application(s), however you can use the method employed by the kernel to determine memory pressure which should give you a general overview on the hosts capability to manage the memory.

Memory pressure is ideal since it is devoid of worrying about page cache, swappiness or even how much memory you actually have.

Memory pressure is effectively a count of how many pages want to be marked active as per /proc/meminfo. The kernel measures memory pressure by keeping track of how many pages go from 'inactive' to 'active' in the page table. A lot of shifting between these two statuses indicates you probably do not have a lot of spare memory available to make more pages active.

Low memory pressure is indicated by having very few promotions from inactive to active (because the kernel clearly has enough space to make active pages stay active).

This script will measure pressure every PERIODIC seconds. The more data you can collect the better. The idea here is you graph the data and stick your Y axis with 0 at the centre. In ideal circumstances the graph should a horizontal line following 0. If the lines regularly spike outside of 0 (particularly 'Active' being positive, or spiking quite high regularly), the memory pressure on the host is high and more memory would be beneficial.

#!/usr/bin/python
import os
import sys
import re
import time

PERIODIC = 1
pgs = re.compile('Active:\s+([0-9]+) kB\nInactive:\s+([0-9]+) kB')
meminfo = open('/proc/meminfo')

def read_meminfo():
    content = meminfo.read(4096)
    m = pgs.search(content, re.M)
    active, inactive = int(m.group(1)), int(m.group(2))
    active = active / 4
    inactive = inactive / 4
    meminfo.seek(0, 0)
    return active,inactive  

if __name__ == "__main__":
    oldac, oldin = read_meminfo()
    while True:
        time.sleep(PERIODIC)
        active, inactive = read_meminfo()
        print "Inactive Pressure:\t%d" % (inactive - oldin)
        print "Active Pressure:\t%d" % (active - oldac)
        oldac = active
        oldin = inactive
1

You can run the command top to see an overview of all the major component in linux including memory usage. When viewing top for the first time, do note that memory used includes buffers and cache if any.

There is also the free command that's for memory. You can execute like free -m to view memory free in megabytes.

There are many more tools, but I think that has sufficiently answered the tool part of the question.

As to when you need more memory depends on the application you are running. Does it need burst capacity? Does it benefit heavily from a large cache size? But generally, if you're hitting swap and often, you really need more ram.

2
  • Thanks Peter. Rather than using basic linux utils, I'm looking to algorithmically generate a single "RAM health" metric that I can monitor over time and anticipate when a server will need more memory, if I'm hitting swap then it is probably too late.
    – loopj
    Oct 11, 2012 at 6:25
  • @loopj Hmm... now I'm more confused as to what's being asked. Are you trying to graph the expected time when memory will become a certain point? But even so, you can still take the outputs of basic linux tools or meminfo and plot them over time either automatically or manually. Perhaps you should clarify your point further. Or maybe you are looking for monitoring solutions like nagios? What application are you running? Or trying to be generic? I don't understand the scenario.
    – Grumpy
    Oct 11, 2012 at 7:01
1

If I were you, I would collect data on load, free memory, free -m and the main performance characteristic of your server (e.g. latency per request), and graphed in Calc/Excel, trying to discern the "swapout cliff" for several datapoints (memory configurations - 8 G, 16G, 32G etc.). Then, I would try various regressions to find the link between the "cliff" and memory available.

A search of existing literature at CiteSeerX would also help.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .