8

My test system setup is a simple raidz 5 disk (8 TBs each) zpool with 256 GB RAM. Performance starts out good, achieving around 650 MB/sec throughput. Eventually, the Linux Buffer cache fills up and starts to cause the ZFS ARC cache to shrink until it reaches around 32 MB in size. Once this occurs, throughput drops to around 60 MB/sec.

I have tried some kernel tuning options such as setting vm.vfs_cache_pressure to 200 but did not change the problem from occurring.

My current workaround is to set /sys/module/zfs/parameters/zfs_arc_min to 128 GB and zfs_arc_max to 252 GB. Because of the Linux buffer cache, the ARC never grows above 128 GB but performance stays at a throughput level of 650 MB/sec for the entire data transfer.

Ideally, I would like to be able to disable Linux buffer cache all together, and let the ZFS ARC cache be between 0 and 256 GB of memory, leaving the rest of the memory available to application work load. If it were possible to give the ZFS ARC cache higher priority than the Linux buffer cache, that would also be an acceptable answer as it would effectively disable the Linux buffer cache.

Thanks, Zack

Edit 1: The issue I am trying to solve is the Linux buffer cache evicting the ZFS ARC from memory. I would like the application workload to be able to use all of the memory, and the ZFS ARC shrink accordingly. As workaround, I had to set the ZFS ARC min size to a large value as the Linux buffer cache would evict the ARC from memory, if the min size was set to 0.

Improve this question
3
  • While setting a minimum ARC size does work, it may be going the wrong way, depending on what your ZFS ARC hit rate is. If the cache hit rate is really low even with a huge 128 GB minimum ARC size, the ARC isn't doing you any good anyway. In that case, severely restricting the maximum ARC size to just a few GB would be better, since it's the thrashing back and forth between the ARC and the page cache that seems to be your problem here. See louwrentius.com/zfs-on-linux-monitor-cache-hit-ratio.html for a short script that computes your ARC hit rate.
    – Andrew Henle
    Jun 27, 2017 at 11:39
  • Thanks Andrew, I will trying lowering the minimum ARC size to 32 GB. The problem I was having is that when I didn't set a minimum size, the page cache would cause the ARC to shrink down to 32 MB, then my write performance on my raidz volume would drop by 90%. It appears that ZFS requires a minimum ARC size, or write performance is horrible.
    – user1492285
    Jun 29, 2017 at 17:31
  • Check your ARC hit rate both before and after lowering the minimum ARC size. For what it's worth, a 32 GB cache is HUGE for most workloads. There's a good chance if you set both your ARC min and max to something like 4 GB you wouldn't see much if any performance change. Once a cache hit rate reaches about 90-95%, there's not much gained performance-wise from adding more to the cache and raising the hit rate even higher, and it normally takes a lot of additional cache to raise hit rates once they reach into the 90s. But it does all depend on your exact workload.
    – Andrew Henle
    Jun 29, 2017 at 19:03

2 Answers 2

Reset to default
5

Set a minimum ARC value. That's all that's necessary.

The ARC will grow/shrink as needed. Whatever is eating buffer cache must not be on a ZFS file system. What is this server doing?

Improve this answer
2
  • 1
    I am copying data to the ZFS pool from an NFS mount. So just the "cp" command. The Linux buffer cache appears to be evicting the ARC. My workaround solution was to set zfs_arc_min to a large value (128 GB).
    – user1492285
    Jun 22, 2017 at 21:23
  • You don't need the ARC minimum to be 128GB. Try 32GB or something like that.
    – ewwhite
    Jun 22, 2017 at 22:44
2

Try mounting your NFS directory with the sync mount option.

Moreover, to limit Linux pagecache usage, you can periodically run echo 3 > /proc/sys/vm/drop_caches

EDIT: as by eewhite comment, the last command seems to drop not only the pagecache, but the ARC also. So, issue it with care and only when necessary.

echo 1 >/proc/sys/vm/drop_caches will touch Linux buffer cache, but leave the ARC alone

Improve this answer
4
  • 1
    I wouldn't run that with ZFS. It'll drop the ARC contents.
    – ewwhite
    Jun 22, 2017 at 22:43
  • Ah, good to know! I was under the impression that ARC was not released the same as the pagecache. Thank you for reporting.
    – shodanshok
    Jun 23, 2017 at 10:58
  • I've found (especially in the ZoL 0.6.x series) that there are certain memory usage peaks (especially with KVM based VMs not starting) it's necessary to drop_caches to provide the kernel the needed RAM to start the VM(s)
    – Hvisage
    Oct 6, 2018 at 12:22
  • Funny enough, I have to call echo 3 > /proc/sys/vm/drop_caches multiple times to finally beat the ARC down.
    – Torsten Bronger
    Apr 21, 2022 at 10:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .