I have very strange case of memory usage on our ubuntu server. One process (which is searchd from sphinxsearch) allocated almost all available memory, and its VSize, RSS and SHR are almost equal (about 18GB). But what makes me really astonished, is that command
free treats most of this memory as "cached" - which I always thought is "kernel owned", that is - not bound to particular process. Also, in the same time it's marked as "shared", although there're no other processes with such high memory usage.
So, free -h shows:
root@st3:/proc/31633# free -h total used free shared buffers cached Mem: 23G 22G 649M 18G 62M 18G -/+ buffers/cache: 4.4G 19G Swap: 0B 0B 0B
But for the searchd process we have:
VmPeak: 20451512 kB VmSize: 20413352 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 20325488 kB VmRSS: 20287332 kB VmData: 1344768 kB VmStk: 136 kB VmExe: 4268 kB VmLib: 16204 kB VmPTE: 39924 kB VmSwap: 0 kB
So I can't really understand what is real usage here - it seems most of memory is used only for cache, so it shouldn't be a concern, OTOH we already encountered several failures with "Cannot allocate memory" for simple fork, so that's why I'm trying to understand it.
If you want more, here's the full meminfo from that machine, and here is the
searchd process' full list of memory mappings.
And looking at the last one, I see a lot of:
7f7c905b7000-7f7c90713000 rw-s 00000000 00:05 226801755 /dev/zero (deleted) 7f7c90713000-7f7c91fff000 rw-s 00000000 00:05 225400928 /dev/zero (deleted) 7f7c92055000-7f7c921c7000 rw-s 00000000 00:05 226767567 /dev/zero (deleted) 7f7c921da000-7f7c92338000 rw-s 00000000 00:05 226774289 /dev/zero (deleted)
...so I can only guess this could be a point, that searchd is doing some clever tricks to keep memory but in the same point have it available for the system when needed. And maybe it doesn't fully work as expected. But that's only my wild guess, and I can be completely wrong here.