4

Dell Perc RAID cards (among others) allow you to set the disk cache policy to be either on (meaning, the individual hard disks use their built-in caches) or off (meaning, the individual disk caches are disabled). In reading discussions on the net, I find conflicting information about which setting is best. Some people say to disable the disk caches because a power failure can cause corruption of data; others say you can leave the disk caches enabled if your computer is connected to an uninterruptible power supply, and that enabling the caches improves disk performance even in RAID configurations.

Is there a definitive conclusion to which way the disk caches should be set?

Note that this is not about the RAID card's cache and caching policy – this is about the disks used in the array, not the card cache or the battery backup on the card itself.

2
  • What you're using the storage for seems to be important to this question. What are you using it for?
    – Edwin
    Aug 6, 2015 at 0:37
  • This will be a file-intensive computer, with millions of various-sized files stored on disk and various programs reading and analyzing the files.
    – mhucka
    Aug 8, 2015 at 23:50

5 Answers 5

2

You have answered the question yourself. If you have a UPS you can leave them on, if you don't then they should be off or else you risk data loss.

On most servers in the datacenter they will generally be using OEM firmware that uses the cache in a read only mode. (The equivalent of off) Writes will be cached by the RAID card with battery backed memory.

1
  • 3
    Agreed. This is a calculated risk, and there is no single "right answer". Generally leaving the disk cache disabled is considered best practice, as you still have the hardware RAID card's onboard cache to cover that functionality and most cards have battery-backed cache.
    – JimNim
    Aug 6, 2015 at 15:48
1

This depends not only on personal preferences, but also on the raid controller in question. 3ware cards generally enable the disk cache on their own under "normal" conditions and this is safe w.r.t. power outages. Perc controllers also generally enable the pdcache, but with basically all sata drives, this very likely causes data loss on a power outage.

Weighing the chance of loss of volume due to a power outage vs. recovery/restore costs is not something that can be definitely answered, as it depends on a lot of factors that differ between places.

Of course, it would be nice if more controllers simply enabled the disk cache on their own and make that safe, which would make this question moot.

1

You should consider a single point of failure, which may be not outside of the server, but inside too. For instance, if the server has only one power supply, a UPS will not help in case of its failure, and all drives' cache will be cleared in a moment.

I personally would follow vendor's recommendations, as their engineers who developed these controllers really know what they say.

Also, you may want to check whether you are able to meet your DRP RTO, and test performance to learn if the risk worth it.

1

Quite old question, but "disk cache params" are always interesting :

Some elements for Dell PERC Raid card. Source : https://www.dell.com/support/manuals/en/us/frbsdt1/dell-openmanage-server-administrator-v8.3/omss_ug/disk-cache-policy?guid=guid-d8bb9348-2881-4a1b-813d-416d2bc78727&lang=en-us.

The Disk Cache Policy feature is supported on : 
 - Non-RAID SAS/SATA physical disks (HDDs) connected to PERC hardware controllers and 
 - on RAID SAS/SATA physical disks connected to Software RAID S130
   controller

So if you have some virtual disk with RAID level 1,5,6,10,... and using a hardware PERC raid card, disk cache has no effect. (Benchmarks we made recently confirm this on a recent PowerEdge R740xd RAID6 with PERC H840 and H740).

--

jm

0

The entire reason why all of these cache toggles / batteries / PLP SSDs exist, is to ensure that the OS has full control and knowledge of when data is written permanently to the disk.

So if you have a battery backed RAID, for example. Then theoretically, as soon as data is written to the cache, it can be considered permanently written to non-volatile medium. That is because: If the power fails, the battery will kick in and ensure that the data will be written to the disk (eventually).

The problem with consumer drive caches is that they are not battery backed (unlike a RAID). So let's say that you have no RAID and a plain old consumer SATA SSD. What happens if the OS writes some data to it (which would go into the cache first) and then the power fails before the cached data could be written to the disk? The data could very well be lost, depending on the timing of things.

In order to prevent this issue, software-based techniques exist to ensure data consistency for such drives. For example, write barriers and journaling file systems. In essence, when the OS wants to make absolutely sure that the data that it wrote to the disk is saved onto its non-volatile medium, it will issue a special flush command. The flush (aka fsync) command will write everything in the cache to the disk before returning a successful completion. Apps and the OS use these fsyncs as a "write barrier". And combining that with other techniques like journaling, they can safely recover your file system or database in the case of a power failure to a 100% known good state even if you have no UPS or battery backed drives. There is a performance overhead for doing all of this, but it is 100% necessary to ensure data integrity for non-RAID consumer drives.

So now that we understand how drives work without a RAID, let's consider what happens when we plug this consumer-level drive into a RAID which has its own cache and battery backup (in addition to the cache on the drive). The first thing to consider is, what do the OS and apps do when they see such a RAID? Properly designed systems will recognize that there is no volatile cache present on this virtual RAID disk. And thus, there is no need to perform flushes (i.e. they will disable write barriers), as every byte written to the drive is considered persistent as soon as it hits the controller. So there is now less performance overhead for running database systems (particularly for journaling operations), and your file systems will also be faster.

But what about the actual disks that comprise the battery backed RAID? Do we need to worry about their write caches? Absolutely! It is extremely important to make sure that when the controller writes data to the actual disk, that it is persisted to a non-volatile medium. If it's not, then the entire guarantee that the controller gave to the OS (that the bits written to its cache are safe and persistent) breaks down! This means that your storage subsystem is not functioning correctly and your data is at risk.

So in short:

If you have a battery-backed RAID:

  • If the drives part of the RAID are consumer drives, and have a cache (which most do). That on-disk cache absolutely MUST be disabled.
  • If the drives are enterprise drives with PLP (power loss protection), then the cache can be enabled, as the drives will ensure that any data written to the drive's cache will be persisted to the non-volatile medium.

I don't want to introduce the concept of a UPS here because: From my point of view, a UPS's job is not to ensure the data integrity of the bits on your disks. It is there to offer some extra time for your apps to perform some work before safely shutting down. And, of course, it can also protect your disks in the case of a misconfiguration. But I think it is incorrect to rely on your UPS to take the place of other technologies that are designed specifically for this purpose, such as write barriers, journaling, FBWC, PLP drives, etc...

This is as best as I understand how this entire system works.

New contributor
AlexPi is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
2
  • Most of what you are saying is correct, however drive cache in general is bad and controllers such as HP disable them by default in all drives. Btw, all of this can be traced back to Microsoft Exchange 4.0 databases being trashed, after that Microsoft added their own disablement using forced unit access. That's also why no databases on crap SATA drives. Nothing to do with the battery, it was the inability to control/disable the cache using FUA.
    – Greg Askew
    Mar 23 at 8:52
  • Yeah, the HPE RAID controllers that I've used do disable the on-disk cache by default on SATA drives with a volatile cache.
    – AlexPi
    Mar 24 at 7:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .