I am interested in using KVM images on zvols under ZFS on Linux, on a multi-socket system. I am wondering how I should pin NUMA nodes so as to maximize the benefits of ZFS ARC cache for all KVM images on the system. Obviously, I should pin each VM to one socket/node so that all of its memory is on one controller. But I am puzzled about how to set up the host so that the ARC maximally benefits each VM.

It seems logical that if I keep the host on one NUMA node, that would mean that the VMs on the other node will have slower access to the ARC. However, if the host is spread across NUMA nodes, its performance will suffer, which could harm performance in the guest.

From what I've read, I'm not even clear on if the host can be tied to specific nodes, or if by definition it uses all of them. And if that is the case, how aware is ZFS on Linux ARC of NUMA nodes?

  • 1
    Why are you even thinking about pinning the VMs? In most circumstances you should just let Linux do its job. – Michael Hampton Feb 26 '17 at 10:49

Manually pinning resources/processes to specific sockets its something that should be only done in very specific circumstances. Moreover, ARC itself is a global resource with (mostly) unpredictable access patterns/hits.

I strongly advise you against this kind of fine tuning unless proven to be needed for good performance. Try running your linux installation with default settings, without messing with NUMA at all, and observe your system behavior using numastat

If more tuning is needed, try using automatic optimization via numad. Only and only if this is not enough, try optimizing using numactl and/or KVM guest CPU pinning.

  • Ok, so Linux itself is good at balancing between NUMA nodes. But when you say that ARC is a global resource, you mean that it does nothing to take into account NUMA nodes? If that's the case, it would seem that using a NUMA system might cause problems for ZFS + VMs if a VM needs to access stuff cached on another node's RAM. And the same goes for L2ARC.. is it possible to, for example, have an M.2 drive devoted to L2ARC on each node, and for zfs to use them intelligently? – Stonecraft Feb 26 '17 at 20:35
  • only if you have a zpool per VM, as the L2ARC/SLOG is attached to the zpool, not the zvol/zfs – Hvisage Oct 6 '18 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.