4

I have a KVM host (Linux 3.14.32) running 8 virtual machines. They are overprovisioned (memory allocated for them is greater than available physical memory on the host). However they consume only about 85% of physical memory which should leave some room for cache. However, everything on host is unbearablly slow and free reports that almost no memory is used for caching:

             total       used       free     shared    buffers     cached
Mem:         32224      26767       5457          0          0          3
-/+ buffers/cache:      26762       5461
Swap:         1021        414        607

As you can see 5.5 GB is wasted. After closing the VMs, everything returns to normal - kernel uses 260M of cache.

/proc/meminfo:

MemTotal:       32997976 kB
MemFree:         5543224 kB
MemAvailable:    5584332 kB
Buffers:            2612 kB
Cached:            83876 kB
SwapCached:        72524 kB
Active:         24721804 kB
Inactive:        2214632 kB
Active(anon):   24675412 kB
Inactive(anon):  2175048 kB
Active(file):      46392 kB
Inactive(file):    39584 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1046520 kB
SwapFree:         622068 kB
Dirty:                84 kB
Writeback:             4 kB
AnonPages:      26806104 kB
Mapped:            28436 kB
Shmem:               488 kB
Slab:              60024 kB
SReclaimable:      25076 kB
SUnreclaim:        34948 kB
KernelStack:        3232 kB
PageTables:        62212 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    17545508 kB
Committed_AS:   53713960 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      431560 kB
VmallocChunk:   34359138492 kB
HardwareCorrupted:     0 kB
DirectMap4k:        9728 kB
DirectMap2M:    33536000 kB

I have swappiness set to 0. Disk images of the VM are on LVM with cache=writeback.

2 Answers 2

3

It's recommended to bypass cache (cache=none) when using RAW devices. This uses O_DIRECT mode, so the host page cache is bypassed and I/O happens directly between the qemu-kvm userspace buffers and the storage device.

QEMU also supports a wide variety of caching modes. If you're using raw volumes or partitions, it is best to avoid the cache completely, which reduces data copies and bus traffic.

Ref: Tuning KVM

2

I would suggest that you might've some I/O issues rather than memory issues. Have a look at top (check out IO_WAIT) and see if it's anything above ~5%.

Also as you're swapping it might be that a process is trying to allocate more than it's able to. Look into KSM https://www.kernel.org/doc/Documentation/vm/ksm.txt and it might be able to dedup you to a decent level or try to lower the RAM allocations by a few fractions and see if it makes a difference.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.