On my newly purchased HP MDS600 with 70x 4TB Nearline SAS disks, I now need to configure the disks, and I want raidz2, so the raid sizes follows 2^n+2. The options I therefore have are

  • 7 raid sets of 10 disks each
  • 11 raid sets of 6 disks each + 4 hot spare


Are there any general rule of when to use hot spare and how many on a given raid level?


From a design perspective, I don't advocate large RAIDZ2 setups... especially with the hardware you've spec'd.

Back in the day, I would use the 48-bay Sun x4540 Thumper/Thor systems. They were designed to hold a few spares in the chassis, with the idea that you wouldn't actually replace the disks. With the 48-bay units, I'd RAID and keep 2 active hot-spare disks. memories...

enter image description here

However, for the configuration you have, I'd recommend 3-4 global spares. Remember, you can associate spare disks with multiple ZFS pools... Be sure the autoreplace property is set on the zpool.

Of course, it will also depend on type of disk in use. Will these be SATA disks or are they Nearline SAS or Enterprise SAS drives?

  • It is Nearline SAS from HP. May 28 '14 at 16:34
  • @JasmineLognnes And which ZFS distribution? Official Solaris? OpenIndiana? NexentaStor? BSD? Linux?
    – ewwhite
    May 28 '14 at 16:35
  • OmniOS. How many pools would you recommend when each disk is 4TB? May 28 '14 at 16:36
  • @JasmineLognnes Forgot about OmniOS. I don't know what you're trying to do with this configuration, so I can't make a better recommendation.
    – ewwhite
    May 28 '14 at 16:40

http://nex7.blogspot.co.uk/2013/03/readme1st.html Scroll down to #12.

The problem with hot spares is that you could be removing single points of failure. For instance, if you have mirrors, each with a drive in two different JBOD's, and you lose a drive in JBOD A and Solaris/Omni/Etc chooses to use a hotspare from JBOD A, you now have a single point of failure.

Some general guidelines from the blog:

The idea, of course, is to determine if hot spares are seemingly required, or if warm spares would do, or if cold spares are acceptable. Here's the ruleset in my head that I use after they tell me the answers to that question (and obviously, this is just my opinion on the numbers to use):

Under 24 hours for remote access, but physical access or lack of disks could mean physical replacement takes longer Warm spares

Under 24 hours for remote access, and physical access with replacement disks is available by that point as well Pool is 2-way mirror or raidz1 vdevs Warm spares

Pool is >2-way mirror or raidz2-3 vdevs Cold spares

Over 24 hours for remote or physical access Hot spares start to become a potential risk worth taking, but serious discussion about best practices and risks has to be had - often is it's 48-72 hours as the timeline, warm or cold spares may still make sense depending on pool layout; > 72 hours to replace is generally where hot spares become something of a requirement to cover those situations where they help, but at that point a discussion needs to be had on customer environment that there's a > 72 hour window where a replacement disk isn't available

  • But it's one JBOD. An HP MDS600.
    – ewwhite
    Jun 13 '14 at 15:49
  • 1
    I'm not familiar with the internals of that JBOD but it still remains true if you exchange the word "jbod" in my post for "expander" or "backplane" or "hba". I was just trying to say that hot spares are "acceptable" only if they don't add a single point of failure :) . Jun 13 '14 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.