49

There are plenty of resources available online that discuss using SSD drives in RAID configurations - however these mostly date back a few years, and the SSD ecosystem is very fast-moving - right as we're expecting Intel's "Optane" product release later this year which will change everything... again.

I'll preface my question by affirming there is a qualitative difference between consumer-grade SSDs (e.g. Intel 535) and datacenter-grade SSDs (e.g. Intel DC S3700).

My primary concern relates to TRIM support in RAID scenarios. To my understanding, despite it being over 6 years since SSDs were introduced in consumer-grade computers and 4 years since NVMe was commercially available - modern-day RAID controllers still do not support issuing TRIM commands to attached SSDs - with the exception of Intel's RAID controllers in RAID-0 mode.

I'm surprised that TRIM support is not present in RAID-1 mode, given the way drives mirror each other, it seems straightforward. But I digress.

I note that if you want fault-tolerance with disks (both HDD and SSD), you would use them in a RAID configuration - but as the SSDs would be without TRIM it means they would suffer Write-Amplification which results in extra wear, which in turn would cause SSDs to fail prematurely - this is an unfortunate irony: a system designed to protect against drive failure might end-up directly resulting in it.

So:

  1. Is TRIM support necessary for modern (2015-2016 era) SSDs?

    1.1. Is there any difference in the need for TRIM support between SATA, SATA-Express, and NVMe-based SSDs?

  2. Often drives are advertised as having improved built-in garbage-collection; does that obviate the need for TRIM? How does their GC process work in RAID environments?

    1.1. For example, see this QA from 2010 which describes pretty-bad performance degradation due to not-TRIMming ( https://superuser.com/questions/188985/how-badly-do-ssds-degrade-without-trim ) - and this article from 2015 makes the case that using TRIM is strongly recommended ( http://arstechnica.com/gadgets/2015/04/ask-ars-my-ssd-does-garbage-collection-so-i-dont-need-trim-right/ ). What is your response to these strong arguments for the necessity of TRIM?

  3. A lot of articles and discussion from earlier years concerns SLC vs MLC flash and that SLC is preferable, due to its much longer lifespan - however it seems all SSDs today (regardless of where they sit on the Consumer-to-Enterprise spectrum) are MLC thesedays - is this distinction of relevance anymore?

    1.1 And what about TLC flash?

  4. Enterprise SSDs tend to have have much higher endurance / write-limits (often measured in how many times you can completely overwrite the drive in a day, throughout a drive's expected 5 year lifespan) - if their write-cycle limit is very high (e.g. 100 complete writes per day) does this mean that they don't need TRIM at all because those limits are so high, or - the opposite - are those limits only attainable by using TRIM?

  • 6
    While I can't answer your question I think it should be considered in light of the fact that our industry is trying it's best to kill off proprietary RAID, the public cloud providers all use SSD's now on compute and storage services , surely they solved this with software, erasure coding etc. Cloud computing innovations have exposed things like hardware RAID, the Cisco IOS and proprietary storage area networks as pointless commodities fermenting at the top of the food chain and actually hampering innovation. hardware RAID can't be sold at scale (to AWS, Azure, CERN) so.... – Sum1sAdmin May 13 '16 at 10:14
  • @Sum1sAdmin the public cloud providers all use SSD's now on compute and storage services Are you saying that AWS or BlackBlaze store data only on SSDs? – A.L May 13 '16 at 13:39
  • @A.L well no, I'm only pointing out that the offer SSD for block, file, object and ephemeral storage – Sum1sAdmin May 13 '16 at 14:44
  • I completely disagree with the "qualitative difference" between commercial and consumer. I can assure you there is no difference in manufacturing of NAND. There are certainly different manufacturers (Samsung vs Intel) but neither has a special consumer manufacturing process. There are certainly feature differences, but not quality differences. – Jim B May 13 '16 at 16:47
  • Speaking of Optane, we are almost half way through 2016 and it's nowhere in sight... Anyone know what the deal is? – Jeff Meden May 13 '16 at 16:55
24

Let's try to reply one question at a time:

  • Is TRIM support necessary for modern (2015-2016 era) SSDs?

Short answer: in most cases, no. Long answer: if you reserve sufficient spare space (~20%), even consumer-grade drive usually have quite good performance consistency values (but you need to avoid the drives which, instead, choke on sustained writes). Enterprise-grade drives are even better, both because they have higher spare space by default and because their controller/firmware combo is optimized toward continuous use of the drive. For example, take a look at the S3700 drive you referenced: even without trimming, it has very good write consistency.

  • Often drives are advertised as having improved built-in garbage-collection, does that obviate the need for TRIM? How does their GC process work in RAID environments

The drive garbage collector does its magic inside the drive sandbox - it does not know anything about the outside environment. This means that it is (mostly) unaffected by the RAID level of the array. That said, some RAID levels (the parity-based one, basically) can sometimes (and in some specific implementation) increase the write amplification factor, so this in turn means higher work for the GC routines.

  • A lot of articles and discussion from earlier years concerns SLC vs MLC flash and that SLC is preferable, due to its much longer lifespan, however it seems all SSDs (regardless of where they sit on the Consumer-to-Enterprise spectrum) are MLC thesedays - is this distinction of relevance anymore

SLC drives have basically disappeared from the enterprise, being relegated mainly to military and some industrial tasks. The enterprise marked is now divided in three grades:

  • HMLC/MLCe flash is the one with the better binned MLC chips, and certified to sustain at least 25000/30000 rewrite cycles;
  • 3D MLC chips are rated at about 5000-10000 rewrite cycles;
  • normal planar MLC and 3D TLC chips are rated at about 3000 rewrite cycles.

In reality, any of the above flash types should provide you with plenty of total write capacity and, in fact, you can find enterprise drives with all of the above flash types.

The real differentiation between enterprise and consumer drives are:

  • the controller/firmware combo, with enterprise drives much harder to die due to unexpected controller bug;
  • the power-protected write cache, extremely important to prevent corruptions to the Flash Translation Layer (FTL), which is stored on the flash itself.

Enterprise grade drivers are better mostly due to their controllers and power capacitors, rather than due to better flash.

  • Enterprise SSDs tend to have have much higher endurance / write-limits (often measured in how many times you can completely overwrite the drive in a day, throughout a drive's expected 5 year lifespan), does this obviate any concerns over Write-Amplification caused by not running TRIM?

As stated above, enterprise grade drives have much higher default spare space (~20%) which, in turn, drastically lowers the need for regular TRIMs

Anyway, as a side note, please consider some software RAIDs that support TRIMs (someone said Linux MDRAID?)

  • Only 35000 write cycles?! That doen't sound like very many. I guess it makes sense if the increased drive replacements are cheaper than buying the same capacity in SLC. – immibis May 13 '16 at 21:52
  • ~30000 minimum guaranteed rewrite cycles are nothing bad: with the increased capacity brought by the switch to 2-bit-per-cell NAND, MLC drives are much cheaper than SLC ones while maintaining similar endurance rating. Moreover, the days when 50-nm class SLC cells were rated at >100000 rewrite cycles are probably gone: even enterprise drives have 34/25nm (or smaller) class flash, with intrinsically lower endurance (which affect SLC drives also). – shodanshok May 13 '16 at 22:21
  • @shodanshok Your conclusion is "Use enterprise-grade (SAS) SSDs attached to a normal RAID controller and don't worry about it" - and that I won't see a performance hit in the 5 years a drive is warranted for? And that the performance problems documented by other users only affect consumer-grade drives? – Dai May 14 '16 at 0:55
  • @Dai for "enterprise-grade" drives I do not mean only SAS SSDs, rather also selected SATA SSDs can be considered "enterprise-grade". And yes, enterprise drives have very good performance consistency, even in steady state (ie: completely full). For an example of such drives, see here. Even some consumer drives, when coupled with generous overprovision, can be quite consistent. See here for an example. – shodanshok May 14 '16 at 10:03
  • 1
    Binning is the process through silicon chips are examined and grouped based on their quality. So, enterprise MLC NAND chips are basically the "better made", better tested chips. – shodanshok May 19 '16 at 6:38
9

TRIM isn't something I ever worry about when using SSDs on modern RAID controllers. The SSDs have improved, hardware RAID controller features have been optimized for these workloads, and endurance reporting is usually in place.

TRIM is for lower end SATA drives. For SAS SSDs, we have SCSI unmap, and perhaps that's the reason I don't encounter TRIM needs...

But the other commenter is correct. Software-Defined Storage (SDS) is changing how we use SSDs. In SDS solutions, RAID controllers are irrelevant. And things like TRIM tend to be less important because SSDs are filling specified roles. I think of Nimble storage read cache or the ZFS L2ARC and ZIL... They all meet specific needs and the software is leveraging the resources more intelligently.

  • 3
    UNMAP and TRIM do exactly the same thing. – Michael Hampton May 13 '16 at 16:38
  • 2
    Trim/unmap is always required- without it, you'd have to completely rely on internal garbage collection – Jim B May 13 '16 at 16:50
  • Internal garbage collection is no substitute for TRIM. There is no firmware function that can replace what TRIM does. It is a little alarming so many answers here don't understand what TRIM actually does and why it is needed. Refer to articles such as this arstechnica.com/gadgets/2015/04/… – Shiv Apr 19 '18 at 5:04
1

RAID levels with SSD An answer above suggests that RAID levels with parity, like RAID 5, increase write amplification. There is really more than one way to interpret that: the impact on one drive or the impact on the set of drives.

Compared to no redundancy, RAID 5 does add writes to the set as it adds checksum parity. Compared to a RAID 0 array of (n-1) drives, the per drive impact of RAID 5 array with n drives is nothing. Each of the n drives receives just as many writes. RAID 5 adds 1/(n-1) extra writes to the set. RAID 1 and RAID 10 however, add 100% extra writes to the set, because everything written to one SSD is written to its mirror.

So, in terms of write to a RAID 5 set vs a RAID 10 set with the same number of drives, the SSDs in the RAID 5 set will receive fewer writes. And that stays true even if you increase the number of SSDs in the RAID 10 set to equalize usable capacity.

0

shodanshok touched on the real answer here. If you reserve extra space, "over-provision", your SSD's endurance and write performance consistency will both be improved over time, and the lack of TRIM support becomes mostly irrelevant. Reserving that extra space can be done as simply as, starting with a new SSD, partitioning less than the full capacity. Most of the in-drive controllers treat never used space the same as reserved space and thereby significantly reduce write amplification. For boot and OS, 10% reserved space is probably enough. For drives that are re-written often, increase that space.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.