zfs on FreeBSD 10

Question

I have encountered to be what I consider a very strange performance issue

I have an application that processes some data and creates 100GB or so of files. This runs fine, and I can scp the files off the machine afterwards. Local disks are ZFS, Maybe that's the problem.

If however I run another utility that mmaps the files and swaps the bytes of the files around, before copying the files off, the files: 1) transfer much slower, maybe 50mb/s (or worse) off a solid state disk, systat -iostat shows a much higher disk transactions level. 2) the Wired memory of the machine goes through the roof, to accommodate a 70+ GB ARC A 70GB arc seems ridiculous since that's around a quarter of the local disk storage. If after transfer the files are deleted, the ARC drops, but the wired memory doesn't seem to. Any ideas on what is going on?

Answer 1

ZFS on non-Solaris platforms seems to have very poor performance when mmap() is used to write data.

See this thread from the old OpenSolaris mailing list:

Remedies for suboptimal mmap performance on zfs

I'm getting sub-optimal performance with an mmap based database (mongodb) which is running on zfs of Solaris 10u9.

...

That was on an older version of Solaris. That's relevant because of this July 2016 thread from the FreeBSD mailing list (emphasis added):

Short story: ZFS was tacked on the kernel and was never properly integrated into the VM page management, which leads to DRAMATIC poor performance for anything which uses mmap() for write IO. This was solved in Oracle Solaris with the great VM allocator rewrite which landed after Opensolaris was made closed source again.

Without a complete rewrite of the VM system this problem is unsolvable.

with this follow-up:

... since the initial import very little has been done to improve integration, and I don't know of anyone who is up to the task taking an interest in it. Consequently, mmap() performance is likely "doomed" for the foreseeable future.

So the problem has been fixed on Solaris, but not on FreeBSD.

If you need to have faster mmap() performance, I'd recommend dramatically limiting the size of your ZFS ARC. That should reduce the impact of coherency problems between the page cache that mmap() uses and the ARC. That might improve performance. Then again, it might not. But it's worth trying. Just don't go too small, or your ARC limit will be ignored.