2

I have encountered to be what I consider a very strange performance issue

I have an application that processes some data and creates 100GB or so of files. This runs fine, and I can scp the files off the machine afterwards. Local disks are ZFS, Maybe that's the problem.

If however I run another utility that mmaps the files and swaps the bytes of the files around, before copying the files off, the files: 1) transfer much slower, maybe 50mb/s (or worse) off a solid state disk, systat -iostat shows a much higher disk transactions level. 2) the Wired memory of the machine goes through the roof, to accommodate a 70+ GB ARC A 70GB arc seems ridiculous since that's around a quarter of the local disk storage. If after transfer the files are deleted, the ARC drops, but the wired memory doesn't seem to. Any ideas on what is going on?

Improve this question

1 Answer 1

Reset to default
2

ZFS on non-Solaris platforms seems to have very poor performance when mmap() is used to write data.

See this thread from the old OpenSolaris mailing list:

Remedies for suboptimal mmap performance on zfs

I'm getting sub-optimal performance with an mmap based database (mongodb) which is running on zfs of Solaris 10u9.

...

That was on an older version of Solaris. That's relevant because of this July 2016 thread from the FreeBSD mailing list (emphasis added):

Short story: ZFS was tacked on the kernel and was never properly integrated into the VM page management, which leads to DRAMATIC poor performance for anything which uses mmap() for write IO. This was solved in Oracle Solaris with the great VM allocator rewrite which landed after Opensolaris was made closed source again.

Without a complete rewrite of the VM system this problem is unsolvable.

with this follow-up:

... since the initial import very little has been done to improve integration, and I don't know of anyone who is up to the task taking an interest in it. Consequently, mmap() performance is likely "doomed" for the foreseeable future.

So the problem has been fixed on Solaris, but not on FreeBSD.

If you need to have faster mmap() performance, I'd recommend dramatically limiting the size of your ZFS ARC. That should reduce the impact of coherency problems between the page cache that mmap() uses and the ARC. That might improve performance. Then again, it might not. But it's worth trying. Just don't go too small, or your ARC limit will be ignored.

Improve this answer
2
  • OK, so you're saying the only solution is to reformat to either UFS or tmpfs? A part of me thinks tmpfs might be better for what is essentially a scratch space, but then I've experienced the horror of tmpfs on solaris not supporting sparse files
    – camelccc
    Mar 16, 2017 at 12:19
  • @camelccc No, I said that ZFS on FreeBSD seems to have a performance issue with modifying data via mmap() and if addressing that is important to you, you can try limiting the ZFS ARC to see if that improves performance to an acceptable level.
    – Andrew Henle
    Mar 16, 2017 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.