I have been tasked to build a large zpool for a data recording system. Using ZFS on Linux (, on CentOS 6.5.

Hardware specs:

  • DL380 Gen8
  • 80 GB ram
  • 2x HP HBA H221
  • 2x HP D6000, each with 70x 3TB SAS disks

There are some special circumstances:

  • Mirroring is out of the question, we would lack storage. The total required amount of TB needed is pretty much set. Adding D6000 units is not an option.
  • There will be 2 of these zpool's in the datacenter, using application level redundancy. Thus the loss of the entire pool is problematic, but not critical. All data will be backed up to tape as well.
  • Using HP Accelerator cards is not an option right now (too expensive)
  • Using PCIe NVMe cards is not an option, due to being restricted to 2.6.32 kernel.

Performance requirements:

  • Writing performance is not critical, there's a basically a steady stream of writing going on, at perhaps 4 MB/sec.
  • Reading performance is useful, but there's not really any hot data set, very random which data could or would be read, depending on need.
  • Indexing is very important. There will be about 1.5 million files once the zpool is fully loaded, getting a list of these files should be as fast as possible. Does ARC help here, a lot of RAM?

My plan is to build the zpool as follows:

  • 14 VDEV's of 10 disks, RaidZ2.
  • Compression on
  • Autoreplace on
  • atime off

I don't really have any good options for SLOG and L2ARC. Using HP SSD's is not really an option, as they would need to run as single device on hardware Raid0. From what I read on this site, that's asking for trouble.

My questions:

  1. More RAM would be nice, sure, but do I really need it?
  2. Should I expect major issues once the zpool is loaded with 95%+ data. Lesser performance is OK, as stated, the performance requirements are quite low.
  3. Any other vdev layouts recommended, without lowering the available storage.
  4. Any special ZFS tuning I should consider? I am at least considering increasing the share of metadata stored in the ARC. To help with indexing speed, make sense?
  5. Should I expect any issues foregoing the use of L2ARC and SLOG device?
  6. Any other tips to make the above work, much appreciated!

closed as too broad by Iain, Ward, ewwhite, womble, mdpc Aug 5 '15 at 5:49

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I'll reply later. But please answer why you would only use EL6.5. You should be on 6.6 and planning 6.7. – ewwhite Jul 23 '15 at 11:36
  • I was hoping you would jump in, big fan! The DL380 mentioned above is part of a larger system rollout. OS version is strictly controlled and out of my hands, even getting ZFS packages added is/was challenging. – redbull666 Jul 23 '15 at 11:46
  • 3
    I'm available for $consulting$. – ewwhite Jul 23 '15 at 12:01
  • I could answer this, but your use case is very specific and unless you're familiar with the tuning needs of the ZFS setup, it makes sense for this to go down the path of paid advice. – ewwhite Jul 23 '15 at 23:11
  • Ok that's a shame, but understandable. Got the setup running now, 1.6 GB/sec writing speed is fun to play around with! – redbull666 Jul 26 '15 at 12:25

More RAM might be a good option. If read performance is not critical, but metadata operations are: I would recommend setting "primarycache" to "metadata" for your filesystems. If you have SSDs for L2ARC, you can also choose to use memory (ARC) for caching "all" and secondarycache (L2ARC) as "metadata". Please note that the other way around does not work (L2ARC gets filled from ARC).

Should I expect major issues once the zpool is loaded with 95%+ data

Yes and no. This would depend on the platform/version. Older versions have a huge speed decrease at 80% capacity. To fix this: Make 100% sure that your slabtables are stored in memory! (Solaris: metaslab_debug=1). By default, older versions only store 2 slabs per disk in memory. This will kill performance. I think FreeBSD and newer versions do this by default. You need some memory however (especially with that many disks).

I would recommend using a SLOG (ZIL) device anyways. This will make sure your writes are acknowledged immediately, but written to slow disks every x seconds. This is usefull, because you only have to do 1 run over the platter for multiple writes. This leaves more disk-time for reads, etc.

I do not recommend using ZFS on Linux for environments this large. The memory allocation system is still not 100% perfect, causing ZFS and Linux fight for memory.

  • Thank you for your suggestion on primarycache/metadata, that makes sense. I will also enable metaslab debugging. Regarding the SLOG. It's not yet clear if our writing will be sync or async. It may even be Linux "cp" only. As I understand it, when using plain file copy, the SLOG will not be used, as it's not synchronous, am I mistaken there? Regarding memory handling, I understand that ZFS has made great improvements in this matter, the issue seems to have been closed on the ZOL bug tracker. – redbull666 Jul 24 '15 at 1:54
  • How did this all turn out? – ewwhite Apr 1 at 23:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.