What are the performance and stability implications of running resource intensive services in a hypervisor vs guest OS?

I want to set up a workstation with both Linux and Windows, hosted by (probably) KVM. Both will use some of the same services: For example, the VMs will be stored on zvols and share a zpool devoted to file storage.

I am hung up on deciding which services to run in the hypervisor OS, and which to run in a VM. If I put services like ZFS in a VM, I am worried that they might be slower (as they will have to work via the hypervisor) or more error prone, as they will depend on controller pass-through (one more thing to have problems with).

On the other hand, I am worried that burdening the hypervisor with too many extra tasks will cause problems, and I am also worried about how much memory ZFS will consume if I don’t put it in a virtual machine with defined limits (I’m thinking to devote 32 Gb to whichever OS is running ZFS. And I’m worried that having too much stuff installed in the hypervisor will make conflicts which affect the whole system more likely (as those conflicts would occur inside the hypervisor).


Hypervisors, regardless of the flavor, should be 100% dedicated to serving its VMs. This is for a few reasons.

First, one of the key benefits of virtualization is hardware independence - if a host dies, it's very simple to restore those VMs to a different host and be up and running in a very short period. As soon as you start deploying services directly on the host OS, that process becomes much more complicated.

Second, hypervisor kernels are tuned to deal with virtualized load, not userland processes running directly on the host OS.

Third, process configuration and deployment becomes overly complicated if you have to consider processes running on VMs and processes running on bare metal. This isn't insurmountable, but it isn't ideal and should be avoided if possible.

So to answer your question directly: don't run any unnecessary processes directly on the host OS. Let the hypervisor do its thing, and use VMs for your services.

  • ZFS in particular does interesting things to Linux's memory management. And by interesting I mean WTF. I've certainly run self-contained hypervisors this way, and it works, but you need to be much more aware of what ZFS is doing, and of course prefer to have a separate storage server if you can. – Michael Hampton Mar 8 '17 at 20:44
  • I'm aware that ZFS loves lots of RAM for its cache, but if I go with ZFS, I plan to go all in and give it a good chunk of memory to work with. By WTF memory use, do you mean anything other than that by default it uses as much as it can? @MichaelHampton – Stonecraft Mar 9 '17 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.