11

I am a bit new to ZFS but I am bringing up a new storage pool with 12 physical drives.

My current plan is to have RAIDZ2 across 10 drives and two hot spares.

But I wonder if I would not be better off with RAIDZ3 across all 12 drives and no hot spares.

The reasoning is that hot spares are basically powered-up, idle drives. It may be months or years before they are called into service and at that time I might find out that they were not viable. If they were part of a RAID stripe I could at least get feedback as to whether they were good or not.

I haven't seen much discussion of this on the web. Does anyone have advice?

Improve this question
2
  • What are your availability requirements? How will the data be backed up?
    – Andrew Henle
    Nov 13, 2017 at 11:01
  • 1
    At this size, it's the point where the dRAID (OpenZFS 2.1) becomes very interesting to setup the "hot spare" (using holes) striped over the diskgroup, so the capacity doesn't increase, but the "Recovery" from failed to restored is accelerated significantly
    – Hvisage
    Nov 7, 2021 at 15:50

3 Answers 3

Reset to default
16

About hot spares

Hot spares are set to a specific pool, but can be attached to any vdev of the pool automatically in case of failures. If you only have a single vdev consisting of all your disks, you are better off directly incorporating the disks (except if you already have RAIDZ3 and still disks to spare).

Additionally, resilvering takes time and occurs in a vulnerable (RAIDZ1, 2-way mirrors) or performance-reduced state (RAIDZ2, RAIDZ3, 3-way mirrors), which would not have occurred if you had already attached the device to the vdev.

Basically hot spares are a thing for large arrays. If you have 27 disks split into 3 vdevs of 9 disks in RAIDZ3, you can add 3 hot spares to reduce the "It's 2 AM and 3 disks have crashed, now I have to get up and fix this mess" moments (assuming a 32 drive bay system). Smaller systems usually don't have enough disks to even get to the "2+ vdevs and Z2/Z3" situation. An exception would be mirrors (e. g. 6 x 2), where crashes are much closer to being fatal for the pool (and you don't have enough disks to make them 6 x 3).

Optimal pool layout

Some advice from Nex7's blog regarding pool layout:

  • Do not use raidz1 for disks 1TB or greater in size.
  • For raidz1, do not use less than 3 disks, nor more than 7 disks in each vdev (and again, they should be under 1 TB in size, preferably under 750 GB in size) (5 is a typical average).
  • For raidz2, do not use less than 6 disks, nor more than 10 disks in each vdev (8 is a typical average).
  • For raidz3, do not use less than 7 disks, nor more than 15 disks in each vdev (13 & 15 are typical average).
  • Mirrors trump raidz almost every time. Far higher IOPS potential from a mirror pool than any raidz pool, given equal number of drives. Only downside is redundancy - raidz2/3 are safer, but much slower. Only way that doesn't trade off performance for safety is 3-way mirrors, but it sacrifices a ton of space (but I have seen customers do this - if your environment demands it, the cost may be worth it).
  • For >= 3TB size disks, 3-way mirrors begin to become more and more compelling.

This means in your case you would have the following options:

  1. 9 disks usable: (Z3 with 9+3)
  2. 8 disks usable: (Z2 with 4+2) ++ (Z2 with 4+2)
  3. 5 disks usable: (2-mirrors) * 5 ++ (hot spare) * 2
  4. 4 disks usable: (3-mirrors) * 4

I would rank them (descending) as:

  • In terms of usable space: 1, 2, 3, 4
  • In terms of safety: 1, 2/4, 3
  • In terms of speed: 4, 3, 2, 1
  • In terms of ability to extend/add drives: 3, 4, 2, 1

I would not use RAIDZ1 regardless of size, because you might want to later replace them with larger disks and then the problems will show (meaning you would not wnat to upgrade this way and might not be able to grow the storage space without adding additional disks).

Improve this answer
1
  • Wow this is a great posting; I hope many people have the chance to refer to it. You covered a lot of tradeoffs in a very short space. In my case I am loading about 5-10TB of stuff from systems being retired, then after that I expect a very high read/write ratio. Since performance isn't my priority it looks like #1 is my best choice. Thanks again.
    – AlanObject
    Nov 17, 2017 at 3:20
5

I've just been benchmarking a test ZFS setup to answer that very question in regards with performances (on a pair of old dusty servers revived from their ashes).

My setup is:

  • 2x Intel Xeon L5640 CPU @ 2.27GHz (total: 12 cores; HT disabled)

  • 96GiB DDR3 RAM @ 1333MHz

  • Adaptec 5805Z controller, exporting all disks as JBODs (with write-cache enabled, thanks to the controller's battery-backed NVRAM)

  • 12x 15kRPM 146GB SAS disks (Seagate ST3146356SS)

  • per-disk DRBD replication (protocol C) via IP-over-Infiniband (20Gb/s Mellanox MT25204)

  • ZFS 0.7.6 on Debian/Stretch

  • zpool create -o ashift=12 ... /dev/drbd{...} (Note: DRBD works with a replication "unit" size of 4KiB)

  • zfs create -o recordsize=128k -o atime=off -o compression=off -o primarycache=metadata ... (the last two for benchmarking purposes only)

Below the bonnie++ results for all possible interesting combinations of RAIDz2 and RAIDz3 (averaged across 5 runs of 12 synchronized bonnie++ processes):

TEST: # data bandwidth
      bonnie++ -p <threads>
      for n in $(seq 1 <threads>); do
        bonnie++ -r 256 -f -s 1024:1024k -n 0 -q -x 1 -y s &
      done
      # create/stat/delete operations
      bonnie++ -p <threads>
      for n in $(seq 1 <threads>); do
        bonnie++ -r 256 -f -s 0 -n 128:0:0:16 -q -x 1 -y s &
      done

CASE: 1*RAIDz2(10d+2p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=278273, RW=150845, RD=487315
 ops/s: SCr=132681, SDl=71022, RCr=133677, RDl=71723

CASE: 1*RAIDz3(9d+3p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=276121, RW=158854, RD=480744
 ops/s: SCr=132864, SDl=71848, RCr=127962, RDl=71616

CASE: 1*RAIDz2(9d+2p+1s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=260164, RW=151531, RD=541470
 ops/s: SCr=137148, SDl=71804, RCr=137616, RDl=71360

CASE: 1*RAIDz3(8d+3p+1s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=269130, RW=184821, RD=672185
 ops/s: SCr=134619, SDl=75716, RCr=127364, RDl=74545

CASE: 1*RAIDz2(8d+2p+2s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=255257, RW=135808, RD=509976
 ops/s: SCr=136218, SDl=74684, RCr=130325, RDl=73915

CASE: 2*RAIDz2(4d+2p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5(data)/3(ops)
 MiB/s: WR=379814, RW=225399, RD=586771
 ops/s: SCr=120843, SDl=69416, RCr=122889, RDl=65736

DATA: WR  = Sequential Write
      RW  = Sequential Rewrite
      RD  = Sequential Read
      SCr = Sequential Create
      SDl = Sequential Delete
      RCr = Random Create
      RDl = Random Delete

As far as performances are concerned:

  • 2*RAIDz2(4d+2p+0s) is the winner for balanced read/write performances

  • 1*RAIDz3(8d+3p+1s) for maximum read performances (quite strangely)

As to how to interpret/explain those results; my 1-pennies:

  • 8 data disks divides the 128k recordsize exactly, it might explain (?) why they always outperform 9 or 10 data disks (given the test is run with 1024k chunk size, which aligns exactly on all disks)

  • I would expect RAIDz3 to perform worse than RAIDz2, yet the 1*RAIDz3(8d+3p+1s) case very strangely contradicts this

  • the significantly smaller VDEVs size of the 2*RAIDz2(4d+2p+0s) case might explain (?) why it performs significantly better for writes

EDIT 1

In response to @AndrewHenle comment, below are additional benchmarks with varying "chunk" sizes. Unfortunately, bonnie++ does not allow chunk sizes other than power of 2; so I reverted to (5 averaged runs) of dd: PS: remember, ZFS read cache (ARC) is disabled

TEST: # WR: Sequential Write
      rm /zfs/.../dd.*
      for n in $(seq 1 <threads>); do
        dd if=/dev/zero of=/zfs/.../dd.${n} bs=<chunk> count=<count> &
      done
      # RD: Sequential Read
      for n in $(seq 1 <threads>); do
        dd of=/dev/null if=/zfs/.../dd.${n} bs=<chunk> count=<count> &
      done

CASE: 1*RAIDz2(10d+2p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024   (n/a)    1024   10240  327680(32768 for RD)
 MiB/s: WR  418.64   (n/a)  434.56  404.44  361.76
        RD  413.24   (n/a)  469.70  266.58   15.44

CASE: 1*RAIDz3(9d+3p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024    1024    1024   10240  327680(32768 for RD)
 MiB/s: WR  428.44  421.78  440.76  421.60  362.48
        RD  425.76  394.48  486.64  264.74   16.50

CASE: 1*RAIDz3(9d+2p+1s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024    1024    1024   10240  327680(32768 for RD)
 MiB/s: WR  422.56  431.82  462.14  437.90  399.94
        RD  420.66  406.38  476.34  259.04   16.48

CASE: 1*RAIDz3(8d+3p+1s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024   (n/a)    1024   10240  327680(32768 for RD)
 MiB/s: WR  470.42   (n/a)  508.96  476.34  426.08
        RD  523.88   (n/a)  586.10  370.58   17.02

CASE: 1*RAIDz2(8d+2p+2s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024   (n/a)    1024   10240  327680(32768 for RD)
 MiB/s: WR  411.42   (n/a)  450.54  425.38  378.38
        RD  399.42   (n/a)  444.24  267.26   16.92

CASE: 2*RAIDz2(4d+2p+0s), ashift:12(4k), recordsize:128k, threads:12, runs:5
 chunk:      1280k   1152k   1024k    128k      4k
 count:       1024   (n/a)    1024   10240  327680(32768 for RD)
 MiB/s: WR  603.64   (n/a)  643.96  604.02  564.64
        RD  481.40   (n/a)  534.80  349.50   18.52

As for my 1-pennies:

  • ZFS obviously optimizes writes intelligently enough (even for chunk sizes below the record size) and/or (?) the Adaptec controller (non-volatile, 512MiB) cache significantly helps in this regards

  • Obviously again, disabling the ZFS read cache (ARC) is very detrimental for chunk sizes close or below the record size; and it seems the Adaptec controller cache is (surprisingly ?) not used for read purposes. Bottom-line: disabling ARC for benchmark purposes allows to have insights into "raw, low-level" performances but is ill-advised for production use (apart from specific cases, like a seldom-used library of large files)

  • Adjusting the chunk size to match the VDEVs size appears not to play a positive role [WRONG ASSUMPTION; see EDIT 2]

EDIT 2

About RAIDz and block size (ashift) and record size (ZFS filesystem):

And a WARNING:

BOTTOM LINE (confirming what has already been said in other responses)

  • (Striping) smaller VDEVs - with less data disks - perform better than large ones; computing/verifying the parity is obviously a costly operation, which grows worse than linearly over the quantity of data disks (cf. 8d <-> 2*4d cases)

  • Same-size VDEVs with more parity disk(s) perform better than fewer parity disk(s) and hot spare(s), and provide better data protection

  • Use hot spare(s) to address "don't wake me up in the middle of the night" concerns, if you still have disk(s) to spare after favoring parity disk(s) [WARNING! see EDIT 2]

POST SCRIPTUM

My eventual use case being hosting a (long term) time-series database (steady medium-sized writes and potentially very large sporadic reads), for which I have very little detailed documentation on the I/O patterns (save for an "optimized for SSD" recommendation), I will personally go for 1*RAIDz2(8d+3p+1s): maximum security, little less capacity, (2nd) best performances

Improve this answer
3
  • I can't interpret the bonnie++ arguments offhand, but how big were your reads/writes? They look like 1 MB, which likely spans across entire vdevs. Try something like unaligned writes of 4 kb to a RAIDZ array, which is a more typical IO size for things like email clients and other user applications.
    – Andrew Henle
    May 2, 2018 at 9:38
  • 1
    @AndrewHenle I confirm the 1MB nicely VDEV-s aligned writes/reads "chunk" size (for 8 data disks cases). Also, I did run some tests with that "chunk" size as low as 4KiB (way below the 128KiB recordsize), which resulted in catastrophic reads amplication (and drop of performances) in the absence of primarycache=all (which absence you really would not want for non-benchmark, real life use patterns). What might be interesting is running additional tests with the chunk size neatly matching the VDEVs for 9 and 10 data disks (I might look into it and report back; no promise made though).
    – Cédric Dufour
    May 3, 2018 at 11:26
  • @CédricDufour I think you mean "1*RAIDz3" not "1*RAIDz2" in your final sentence?
    – Davor Cubranic
    Jul 2, 2021 at 21:47
3

My recommendation is:

2 x 5-disk RAIDZ1 + two spares

or

3 x 3-disk RAIDZ1 + spares

or

10-disk RAID mirrors

or 2 x RAIDZ2 of 5 or 6 disks with or without spares

This depends on the disk type in use. If 7200 RPM drives over 2TB, go towards RAIDZ2. If 2TB and user, RAIDZ1 is fine.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .