Update: because recordsize defaults to 128k, the amount of data read by the test program is much larger than the ARC on an 8GB system and still slightly larger than the ARC on a 16GB system. Reducing the recordsize allows less data to be read and therefore it fits in the ARC. I was underestimating the size of the data being read, the effect of recordsize on that, and therefore drawing some poor conclusions. So far, disabling prefetch doesn't seem to make much difference in this case although I am going to try all the recordsize options with and without prefetch enabled.

This load is similar to an IMAP/Maildir scenario with many directories, many files, and potentially only small amounts of data read from a each file.

I have tested using FreeBSD 10 and Fedora 19 with zfsonlinux. I have tested various linux native filesystems, like extX/xfs/jfs and even btrfs. On FreeBSD I tested using the native ufs filesystem as well. My workload is simply scanning a largish music collection using amarok/winamp/etc. My test program is amarok_collectionscanner because it can be run from the command line easily. The pattern is always the same. An initial run of the collection scanner takes around 10 minutes depending on the filesystem but ZFS performs similarly to non-ZFS filesystems

Subsequent runs of a scan are incredibly fast using a non-zfs filesystem, usually around 30 seconds. ZFS makes only marginal improvements with subsequent runs. It's also obvious from watching iostat that after an initial run on an non-ZFS filesystem the OS doesn't touch the disk. It's all in the filesystem cache.

Using an SSD cache for ZFS improves the time, but it never gets anywhere near 30 seconds.

Why doesn't ZFS cache this load? One possibility I explored was that the size of the ARC was limited to less than what a non-ZFS filesystem is allowed to use for caching. I tested again on a machine with more memory available to the ARC than the entire available memory on the first test system and the numbers stayed the same.

I hope to find/create a fio recipe that duplicates this kind of load. Basically it would need to create thousands of smallish files, scan all the directories looking for the files, open each file and read a small amount of data from each one. It is like the world's worst database! I will probably test OpenIndiana next but I expect the results to be the same.

The data set is 353GB, and 49,000 files. The test systems had 8GB-16GB of RAM. The zpool configuration made very little difference but the tests I care about were always just one whole disk. I used ST3500630AS and WDC WD20EZRX-00D8PB0 among other drives. The drives made almost no difference. The amount of RAM or the speed of the CPUs made little or no difference. Only the filesystem in use changed the results appreciably and those differences were quite substantial as I noted above. I actually have mountains of data points regarding the various filesystem parameters I tried and these are some of the variables I checked: mdadm raid configurations (0 and 1) zpool configurations, mirror and stripe zfs recordsize mdadm chunk size filesystem blocksize

On a single ST3500630AS drive I got these numbers for the default filesystem options for the following filesystems. This was on Fedora 19, 8GB of RAM, 3.11.10-200 kernel, ZFS 0.6.2-1. The values are in seconds. Subsequent scans were run without any attempt to clear the cache.

ZFS: 900, 804, 748, 745, 743, 752, 741
btrfs: 545, 33, 31, 30, 31, 31
ext2: 1091, 30, 30, 30, 30, 30...
ext3: 1014, 30, 30, 30, 30, 30...
ext4: 554, 31, 31, 32, 32, 31, 31...
jfs: 454, 31, 31,31,31...
xfs: 480, 32, 32, 32, 32 ,31 ,32, etc.

On FreeBSD 10, single drive WD20EZRX-00D8PB0, faster machine, 16GB of memory, ARC allowed to grow to 12GB:

ufs: 500, 18, 18...
zfs: 733, 659, 673, 786, 805, 657

Although the the above variables sometimes had an effect on the initial cold cache scan of the data, it's the subsequent runs that look all the same. Standard filesystems cache it all and therefore as long as nothing else thrashes the cache perform lightning fast. ZFS doesn't exhibit that behaviour.

  • ZFs is not well suited to desktop workloads. Mar 6, 2014 at 18:10
  • 2
    So scanning a bunch of files and reading data from them is a desktop workload? It certainly seems to me that this workload would be a lot like a Maildir/IMAP server. The files are slightly larger but the amount of data read from each one is small in both cases. This was tested on Xeon systems with ECC memory although the hard drives were SATA drives.
    – aranc23
    Mar 6, 2014 at 18:17
  • 2
    Can you provide some solid numbers? How much RAM is in the system? What is the size of the data set? How big are the disks? How big are the SSD devices? What is your zpool configuration? How about your zfs filesystem settings?
    – ewwhite
    Mar 6, 2014 at 18:31

2 Answers 2


Start with disabling atime if not already done.

You might also investigate setting primarycache=metadata impact.

  • I do disable atime on all the filesystems where that is an option, but it was a good suggestion. It would seem that setting primarycache to metadata would cause zfs to cache even less than it currently does but since it costs me nothing but time to try it, I probably will.
    – aranc23
    Mar 7, 2014 at 15:38
  • You do not tell your ARC statistics. I'm assuming your cache is too small for all metadata to fit so suggest to reserve all of its size for metadata instead of (possibly useless) data.
    – jlliagre
    Mar 7, 2014 at 16:35
  • On one system with 8GB of memory regular filesystems cache the whole working data set. On a system with 16GB of RAM and 12GB arc, zfs is still reading from the disk and has not cached all of the working data. The size of the ARC on all systems tested is greater than the size of the data read by the test program.
    – aranc23
    Mar 7, 2014 at 19:58
  • Prefetch might also play a role here. You might try disabling it.
    – jlliagre
    Mar 7, 2014 at 20:29
  • I have disabled prefetch and I am testing again.
    – aranc23
    Mar 7, 2014 at 21:38

On FreeBSD, install sysutils/zfs-stats

the 'zfs-mon' tool that is part of that package will give you details about the cache hit/miss ratios for each of the different types of cache in ZFS (ARC, ARC Metadata, ZFETCH, Prefetch, etc).

Also, 'zpool iostat 1' while scanning might help

By default, the 'metadata' cache is limited to 1/4 of the ARC, you can adjust this value with the vfs.zfs.arc_meta_limit loader.conf tunable

In FreeBSD 10, the ARC stats are included in 'top', watching how those values change while you are scanning may provide some insight

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .