1

I am interested in using KVM images on zvols under ZFS on Linux, on a multi-socket system. I am wondering how I should pin NUMA nodes so as to maximize the benefits of ZFS ARC cache for all KVM images on the system. Obviously, I should pin each VM to one socket/node so that all of its memory is on one controller. But I am puzzled about how to set up the host so that the ARC maximally benefits each VM.

It seems logical that if I keep the host on one NUMA node, that would mean that the VMs on the other node will have slower access to the ARC. However, if the host is spread across NUMA nodes, its performance will suffer, which could harm performance in the guest.

From what I've read, I'm not even clear on if the host can be tied to specific nodes, or if by definition it uses all of them. And if that is the case, how aware is ZFS on Linux ARC of NUMA nodes?

Improve this question
1
  • 1
    Why are you even thinking about pinning the VMs? In most circumstances you should just let Linux do its job.
    – Michael Hampton
    Feb 26, 2017 at 10:49

1 Answer 1

Reset to default
4

Manually pinning resources/processes to specific sockets its something that should be only done in very specific circumstances. Moreover, ARC itself is a global resource with (mostly) unpredictable access patterns/hits.

I strongly advise you against this kind of fine tuning unless proven to be needed for good performance. Try running your linux installation with default settings, without messing with NUMA at all, and observe your system behavior using numastat

If more tuning is needed, try using automatic optimization via numad. Only and only if this is not enough, try optimizing using numactl and/or KVM guest CPU pinning.

Improve this answer
2
  • Ok, so Linux itself is good at balancing between NUMA nodes. But when you say that ARC is a global resource, you mean that it does nothing to take into account NUMA nodes? If that's the case, it would seem that using a NUMA system might cause problems for ZFS + VMs if a VM needs to access stuff cached on another node's RAM. And the same goes for L2ARC.. is it possible to, for example, have an M.2 drive devoted to L2ARC on each node, and for zfs to use them intelligently?
    – Stonecraft
    Feb 26, 2017 at 20:35
  • only if you have a zpool per VM, as the L2ARC/SLOG is attached to the zpool, not the zvol/zfs
    – Hvisage
    Oct 6, 2018 at 12:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .