Advice for building production 140 disk (420 TB) ZFS zpool [closed]

Question

I have been tasked to build a large zpool for a data recording system. Using ZFS on Linux (0.6.4.2), on CentOS 6.5.

Hardware specs:

• DL380 Gen8
• 80 GB ram
• 2x HP HBA H221
• 2x HP D6000, each with 70x 3TB SAS disks

There are some special circumstances:

• Mirroring is out of the question, we would lack storage. The total required amount of TB needed is pretty much set. Adding D6000 units is not an option.
• There will be 2 of these zpool's in the datacenter, using application level redundancy. Thus the loss of the entire pool is problematic, but not critical. All data will be backed up to tape as well.
• Using HP Accelerator cards is not an option right now (too expensive)
• Using PCIe NVMe cards is not an option, due to being restricted to 2.6.32 kernel.

Performance requirements:

• Writing performance is not critical, there's a basically a steady stream of writing going on, at perhaps 4 MB/sec.
• Reading performance is useful, but there's not really any hot data set, very random which data could or would be read, depending on need.
• Indexing is very important. There will be about 1.5 million files once the zpool is fully loaded, getting a list of these files should be as fast as possible. Does ARC help here, a lot of RAM?

My plan is to build the zpool as follows:

• 14 VDEV's of 10 disks, RaidZ2.
• Compression on
• Autoreplace on
• atime off

I don't really have any good options for SLOG and L2ARC. Using HP SSD's is not really an option, as they would need to run as single device on hardware Raid0. From what I read on this site, that's asking for trouble.

My questions:

1. More RAM would be nice, sure, but do I really need it?
2. Should I expect major issues once the zpool is loaded with 95%+ data. Lesser performance is OK, as stated, the performance requirements are quite low.
3. Any other vdev layouts recommended, without lowering the available storage.
4. Any special ZFS tuning I should consider? I am at least considering increasing the share of metadata stored in the ARC. To help with indexing speed, make sense?
5. Should I expect any issues foregoing the use of L2ARC and SLOG device?
6. Any other tips to make the above work, much appreciated!

Answer 1

More RAM might be a good option. If read performance is not critical, but metadata operations are: I would recommend setting "primarycache" to "metadata" for your filesystems. If you have SSDs for L2ARC, you can also choose to use memory (ARC) for caching "all" and secondarycache (L2ARC) as "metadata". Please note that the other way around does not work (L2ARC gets filled from ARC).

Should I expect major issues once the zpool is loaded with 95%+ data

Yes and no. This would depend on the platform/version. Older versions have a huge speed decrease at 80% capacity. To fix this: Make 100% sure that your slabtables are stored in memory! (Solaris: metaslab_debug=1). By default, older versions only store 2 slabs per disk in memory. This will kill performance. I think FreeBSD and newer versions do this by default. You need some memory however (especially with that many disks).

I would recommend using a SLOG (ZIL) device anyways. This will make sure your writes are acknowledged immediately, but written to slow disks every x seconds. This is usefull, because you only have to do 1 run over the platter for multiple writes. This leaves more disk-time for reads, etc.

I do not recommend using ZFS on Linux for environments this large. The memory allocation system is still not 100% perfect, causing ZFS and Linux fight for memory.