Question: Would it ever be a good idea to disable page cache inside guest VM's and instead rely on the ZFS ARC (and SSD based L2ARC) of the host?

Context: I'm asking since I'm running a Proxmox cluster which is always showing around 90% use of RAM for all VM's, regardless of how much it actually needs. This is to be expected due to the guest kernel's use of page cache. Since I've been hearing a lot of good things about ZFS's ARC it got me thinking that perhaps I could increase the reliance on those and reduce the reliance on the page cache of the guests. In essence the ARC would kind of be a shared page cache for all VM's.

By doing this I would get the additional benefit of more accurate proxmox statistics and graphs, thus giving me a better picture on how much memory each VM actually needs. This in turn would give me the information I need in order to better tune the sizes of each VM's RAM and allow me to increase the size of the host's ARC by the same amount.

I haven't actually tried any of this, I thought I would run it by you guys first. So, am I stupid for thinking this way?

Follow up question: HOW would I go about disabling (or limiting) the page cache in a Linux VM? ONE method would be to use a cronjob and regularly write "3" to /proc/sys/vm/drop_caches, like once every minute or something. But it feels kinda hacky, isn't there a better way?

P.S. Yes I realize that I'm only talking about read cache, not write cache which is affected by the amount of dirty pages. So I would probably still need some amount free RAM space to allow for that (but that should be visible in the VM's RAM usage statistics in Proxmox, so everything above should still apply).

  • Are your VMs short on RAM? Is everything right-sized?
    – ewwhite
    May 24 '19 at 15:07
  • No, they have more than they need, which is actually (kind of) the point of the question.
    – A. Nilsson
    May 24 '19 at 15:26
  • Why would you want to hurt your VMs' performance in this way? May 24 '19 at 17:19
  • But that's the main point of the question, why would it hurt my VM's performance? Unless I have missunderstood how this works (which I might have, which is why I asked the question in the first place) I'm simply moving the page cache outside of the VM but not actually moving it outside of RAM, right? So it should be the same performance?
    – A. Nilsson
    May 24 '19 at 17:51
  • 1
    If you memory limit your VMs or use a balloon driver this basically happens automatically, page cache and buffer cache will shrink to a minimum. Not having any cache won’t be very efficient since you still want to reduce roundtrips by having multi block reads or read ahead and also async flushes. Also the host cache will mostly benefit shared blocks (shared snapshot bases) only.
    – eckes
    May 25 '19 at 10:59

I often (but not always, see below) optimize my hypervisors similar to what you suggest: let VMs to heavily relay on shared host disk cache.

However, using the drop_caches approach seems too heavy-handed to me, as it can evict too much cache memory from guest. At the same time, I don't know any method to limit pagecache (short of configuring your application for using direct I/O). So, the key is to correctly size your VM RAM resources: try to assign only the memory a guest really needs, plus 1 or 2 GB for having some "breathing room".

Managing memory in this manner has some important advantages:

  • being managed by the host, cache memory can be dynamically allocated to guests based on their I/O requirements;
  • by the virtue of being dynamically managed, treating your memory as a true resource pool, you reduce wasted resources and increase efficiency;
  • if using ZFS on your host, you tap into the very advanced ARC/L2ARC and its trash-resistant behavior

But there are some disvantages as well:

  • being a shared resource, your host cache memory can be trashed by a rogue VM (impacting other, more important, VMs);
  • being located some context switches and vmexit/vmenter away, the peak and sustained speed of any host-based cache will be lower than the corrispettive guest-side cache (and this is the reason I suggest you to avoid repeated drop_caches in the guest);
  • albeit more advanced and with much higher hit-rate, ARC is slower than linux pagecache when the workload fits entirely in cache. So, for maximum guest speed on performance-critical guests, you probably want to give to the VM sufficient memory for pagecache.
  • Thank you, this was the kind of nuanced information I was looking for! I knew I was oversimplifying the issue somehow, but I wasn't clear on the details.
    – A. Nilsson
    May 25 '19 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.