Linux Kernel 4.20 added PSI, which stands for "pressure stall information". It gives you more insights why a machine is overloaded. And which resource is the bottleneck.
There are three new files under /proc/pressure
:
/proc/pressure/cpu
/proc/pressure/memory
/proc/pressure/io
To quote from Tracking pressure-stall information concerning /proc/pressure/memory
:
Its output looks like:
some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828
full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258
The some
line is similar to the CPU information: it tracks the percentage of the time that at least one process could be running if it weren't waiting for memory resources. In particular, the time spent for swapping in, refaulting pages from the page cache, and performing direct reclaim is tracked in this way. It is, thus, a good indicator of when the system is thrashing due to a lack of memory.
The full
line is a little different: it tracks the time that nobody is able to use the CPU for actual work due to memory pressure. If all processes are waiting for paging I/O, the CPU may look idle, but that's not because of a lack of work to do. If those processes are performing memory reclaim, the end result is nearly the same; the CPU is busy, but it's not doing the work that the computer is there to do. If the full
numbers are much above zero, it's clear that the system lacks the memory it needs to support the current workload.
I have not access to a production server with Linux 4.20 yet, but here is a small experiment on my Desktop (which has no swap configured). Initially, I have no memory pressure at all (all counters are 0):
$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
Then I increased the memory usage until I eventually ran out of memory, which froze the machine until the OOM killed some processes. Before it froze, the pressure on the memory increased:
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.00 avg60=0.00 avg300=0.00 total=47047
full avg10=0.00 avg60=0.00 avg300=0.00 total=32839
some avg10=0.00 avg60=0.00 avg300=0.00 total=116425
full avg10=0.00 avg60=0.00 avg300=0.00 total=81497
some avg10=1.26 avg60=0.22 avg300=0.04 total=183863
full avg10=0.72 avg60=0.13 avg300=0.02 total=127684
Now, after the system has recovered, the memory pressure is again 0, and the total
counters no longer increase:
$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.07 total=53910568
full avg10=0.00 avg60=0.00 avg300=0.02 total=27766222
...
$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.05 total=53910568
full avg10=0.00 avg60=0.00 avg300=0.00 total=27766222
/proc/meminfo
and/proc/vmstat
): stackoverflow.com/a/3031924