4

VM Host -> Xenon E5-2440 w/ 48G mem... Everything runs CentOS6.5 (2.6.32-431)

I have 4 guest VMs running each w/ 2G of memory. Their disk images are local files. The host machine is also running a lightly used NFS server but not much else. (swappiness=0) After a day or so as the buffer cache grows to nearly 40G some of these VMs wind up nearly fully swapped to disk. (viewed via: grep VmSwap /proc/PID/status)

The problem I'm having is while these VMs may not be regularly used they have to be at the ready. However in practice they are being swapped out causing serious issues with their response time.

I'm certainly all for having my guest VMs use a reasonable sized swap file and let the OS decide the balance between buffer cache & swap but this doesn't seem to be working for the host machine in my use case.

Any option to keep the VMs responsiveness from being degraded besides disabling swap on the host machine? Attempt with cgroups or just pull the plug on the swap file for this use case?

2

You can lock the pages into memory in later versions of libvirt:-

http://libvirt.org/formatdomain.html#elementsMemoryBacking

Careful: This doesn't appear when using Fedora 19 as a hypervisor, nevertheless according to the changelog for the latest RPM (I can find) for EL6.5 libvirt this exists;

  • Thu Jul 18 2013 Jiri Denemark - 0.10.2-21
    • conf: Avoid NULL deref for pmsuspended domain state (rhbz#822306)
    • libvirt: Define domain crash event types (rhbz#822306)
    • qemu: Refactor processWatchdogEvent (rhbz#822306)
    • qemu: Expose qemuProcessShutdownOrReboot() (rhbz#822306)
    • qemu: Implement 'oncrash' events when guest panicked (rhbz#822306)
    • qemu: Implement 'oncrash' coredump events when guest panicked (rhbz#822306)
    • conf: Fix a memory leak when parsing nat port XML nodes (rhbz#851455)
    • security_manager: Fix comparison (rhbz#984793)
    • qemu: Prevent crash of libvirtd without guest agent configuration (rhbz#984821)
    • qemu: Fix double free of returned JSON array in qemuAgentGetVCPUs() (rhbz#984821)
    • qemu_agent: Add support for appending arrays to commands (rhbz#924400)
    • Add support for locking domain's memory pages (rhbz#947118)
    • qemu: Implement support for locking domain's memory pages (rhbz#947118)
    • qemu: Check for -realtime mlock=on|off support (rhbz#947118)
    • qemu: Move memory limit computation to a reusable function (rhbz#947118)
    • util: New virCommandSetMax(MemLock|Processes|Files) (rhbz#947118)
    • qemu: Set RLIMIT_MEMLOCK when memoryBacking/locked is used (rhbz#947118)
    • Add Gluster protocol as supported network disk backend (rhbz#849796)
    • qemu: Add support for gluster protocol based network storage backend. (rhbz#849796)
    • tests: Add tests for gluster protocol based network disks support (rhbz#849796)
| improve this answer | |
  • libvirt.org/formatdomain.html#elementsMemoryTuning (which is needed for this to function) reads like "You can fly without help when you jump from a cliff. The only tricky part is the impact some seconds later, which yet is an unsolved problem." – Tino Apr 9 '16 at 9:42
  • I added step-by-step instructions as another answer. – Joachim Wagner Sep 19 '18 at 9:10
1

You can use cgroups, and set up swappiness per cgroup

http://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt

http://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

A KVM virtual CPU is just a thread on the host, so it can be controlled as any other process.

| improve this answer | |
  • Is this typically how things are handled in KVM deployments? It's foreign to me, as it wouldn't be an issue on a VMware install. – ewwhite Dec 15 '13 at 21:15
  • Link only answers are bad. Your link broke, so I edited them to the new location. However it is likely they will break again in future. – Tino Apr 9 '16 at 9:35
  • The answer is right there, following the links is optional. – dyasny Apr 10 '16 at 7:18
1

Here step-by-step instructions for Matthew's solution:

  1. Shut down the VM
  2. virt-xml $VMNAME --edit --memorybacking locked=on
  3. systemctl restart libvirtd (not sure whether that's needed)
  4. Start the VM

where $VMNAME is the name of the VM.

I successfully passed a situation where previously my VM would be swapped out completely. Now the VM's qemu process has 0 swap usage and is responsive.

Caveat: According to libvirt documentation, all memory of qemu will be locked, it may grow unpredictably and one should set hard_limit to protect the host system (the VM will be killed if necessary to stay within the limit).

Edit: Simplified step 2 (was virsh edit $VMNAME and add <memoryBacking><locked/></memoryBacking> after <domain type='kvm'>)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.