KVM host won't use memory for caching

Question

I have a KVM host (Linux 3.14.32) running 8 virtual machines. They are overprovisioned (memory allocated for them is greater than available physical memory on the host). However they consume only about 85% of physical memory which should leave some room for cache. However, everything on host is unbearablly slow and free reports that almost no memory is used for caching:

             total       used       free     shared    buffers     cached
Mem:         32224      26767       5457          0          0          3
-/+ buffers/cache:      26762       5461
Swap:         1021        414        607

As you can see 5.5 GB is wasted. After closing the VMs, everything returns to normal - kernel uses 260M of cache.

/proc/meminfo:

MemTotal:       32997976 kB
MemFree:         5543224 kB
MemAvailable:    5584332 kB
Buffers:            2612 kB
Cached:            83876 kB
SwapCached:        72524 kB
Active:         24721804 kB
Inactive:        2214632 kB
Active(anon):   24675412 kB
Inactive(anon):  2175048 kB
Active(file):      46392 kB
Inactive(file):    39584 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1046520 kB
SwapFree:         622068 kB
Dirty:                84 kB
Writeback:             4 kB
AnonPages:      26806104 kB
Mapped:            28436 kB
Shmem:               488 kB
Slab:              60024 kB
SReclaimable:      25076 kB
SUnreclaim:        34948 kB
KernelStack:        3232 kB
PageTables:        62212 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    17545508 kB
Committed_AS:   53713960 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      431560 kB
VmallocChunk:   34359138492 kB
HardwareCorrupted:     0 kB
DirectMap4k:        9728 kB
DirectMap2M:    33536000 kB

I have swappiness set to 0. Disk images of the VM are on LVM with cache=writeback.

Answer 1

It's recommended to bypass cache (cache=none) when using RAW devices. This uses O_DIRECT mode, so the host page cache is bypassed and I/O happens directly between the qemu-kvm userspace buffers and the storage device.

QEMU also supports a wide variety of caching modes. If you're using raw volumes or partitions, it is best to avoid the cache completely, which reduces data copies and bus traffic.

Ref: Tuning KVM

Answer 2

I would suggest that you might've some I/O issues rather than memory issues. Have a look at top (check out IO_WAIT) and see if it's anything above ~5%.

Also as you're swapping it might be that a process is trying to allocate more than it's able to. Look into KSM https://www.kernel.org/doc/Documentation/vm/ksm.txt and it might be able to dedup you to a decent level or try to lower the RAM allocations by a few fractions and see if it makes a difference.