Data consistency with RAID 5 and BBWC

Question

I have a few questions about RAID 5, BBWC (Battery Backed Write Cache), filesystem (ext3) and optionally mysql.

I have a hypothesis and I ask Server Fault community to validate or not this one:

-The system is UP, applications works with database and use transactions

-The RAID system is configured with RAID5,write caching and battery (BBWC)

With this solution, when OS receive write ACK, data are possibly in the RAID controller cache, and not in a persistent memory (my RAID drives disk).

If a power failure appears and data are in the RAID controller cache, the BBWC maintains data with the battery pack. Until then, everything is good.

But battery has a certain capacity, and it’s not infinite. If the power failure is longer than two or three days, the RAID cache is lost.

For me it’s a dramatic and likely scenario.

My questions are:

• Ext3 can solve this problem? Probably not

• Is it possible to corrupt mysql database?

• Mysql transactions are lost? I think so

Thank you for your help

Hardware: HP DL380 G7

OS: Centos 5 with ext3

Answer 1

Yes, that is the limitation of a battery-backed write cache -- that if the battery runs out, your unwritten data is toast. It's why I prefer the newer NVRAM-based RAID controllers. Filesystems can't help you, because they think the data's been written -- at best they can provide a consistent view of the data, not ensure it's up-to-date.

Answer 2

You should get a UPS so the server can gracefully shutdown and the BBWC doesn't need to be used at all. Even a relatively small UPS can provide a few minutes of shutdown time, which is generally plenty enough to get those caches flushed to disk.

Flash Backed Write Caches can help, though I think most would agree that a UPS is a better idea.

A more costly option: You may want to consider getting a generator if you commonly experience 3+ day blackouts. I have found such occurrences to be very uncommon, but I see you're across the pond.

Answer 3

How much of a power outage/interruption are you expecting? The batteries have a lifetime of a few days. However, since the G7 ProLiant servers, the use of a flash-backed cache (FBWC) unit has become more popular. It's a straight retrofit if your Smart Array P410 controller doesn't already have one. The technology is described in more detail here. The point of the super-capacitor in the FBWC unit is to provide enough power to write the write cache contents to flash memory, where it will persist until the system power is restored. The other benefit is no batteries to deal with and dispose of. So from a manufacturer's standpoint, it's advantageous to promote the use of flash versus the old-school batteries.

I would think that having a power outage long enough for the battery life to be a concern also means that there are other infrastructure issues to worry about. Same thing if someone is not available onsite for that period of time to provide power relief. The sane recommendation is to protect your systems with a solid UPS solution.

As to the amount of data contained in the cache at any point, it's typically small. The cache is probably 512MB or 1024MB in your case. Random and streaming writes need to live there long enough until they are flushed to disk. At any point, you're probably looking at a few megabytes of data. I've haven't had major corruption following a sudden crash where the cache data was lost (failed RAID battery). This probably won't be an issue for you unless your system's write pattern is abnormally heavy and you lose power at the same time in an unexpected manner...(like a lightning strike). ext3 is pretty resilient, though.

Answer 4

Thanks you for your response. I know alternatives, like FBWC, but my questions are based on my scenario :

On my work, we have already deployed this solutions and I have to anticipate problems.

Did you think mysql can be corrupted and refused to start with my scenario?