Windows Server 2008 Software Raid 5 - Data integrity issues

Question

I've got a server running Windows Server 2008 R2, with a (windows native) software raid-5 array. The array consists of 7x 1TB Western Digital RE3 and RE4 drives. I have offline backups of this array.

The problem is this: I noticed a few days ago after copying a large file to the disk that there was an integrity issue with that file - it was a ~12GB file that I had downloaded via uTorrent. After moving it to the raid array, I used uTorrent to relocate the download location, and performed a re-check so I could seed it from that location. The recheck found that only 6308/6310 chunks of the copied file were intact.

My next step was to write a quick powershell script that would copy files to the array, while performing a SHA1 hash of the original and resultant files and comparing them. Smaller files (100-1000MB) copied over just fine. When I started copying larger data (~15GB), I found that the hash check failed about 2/3rds of the time. The corrupt files had very, very small inconsistencies - less than .01% (EDIT - later experimentation has shown that the corrupted chunks of data are always 60 bytes in length, with one to three generally manifesting per 15GB copied file. Corrupted data appears random, with no consistent pattern of flipped bits). I further eliminated the possibility of networking or client issues by placing this large file on the C:\ of the server, and copying it repeatedly from there to the array, seeing similar results.

Copying the data via explorer, powershell, or the standard windows command prompt yield the same results. None of the copies fail or report any problems. The raid array itself is listed as healthy in disk management.

After a few experiments, I shut down the server and ran memtest overnight. No errors were detected. A basic run of chkdsk found no problems, but I did not use the /R flag, as I was unsure how that might affect a software raid-5 volume.

I next ran Crystal Disk Info to check the smart data on the drives - but found that CDI only detected 5 out of 7 of the disks in the array. I have no idea why. Nevertheless, CDI shows the following "caution" flags on a single one of the drives:

05 199 199 140 000000000001 Reallocated Sectors Count
C5 200 200 __0 000000000001 Current Pending Sector Count

Which is a little bit alarming, but I don't really know what to do with the information. I hardly feel like one reallocated sector could be causing this.

At this point, I'm looking for some guidance on what to do next. I need to determine the cause of this issue, but I'm hesitant to run chkdsk /R or any bootable disk health checkers because I'm afraid they might break the array. I've considered triggering a re-sync of the array, but I'm not actually sure how to do that without doing something silly like manually dropping a disk and then restoring it.

Any advice that could help me ferret out the precise cause of this issue would be greatly appreciated.

Answer 1

As none of the operations you are performing is failing, running chkdsk /R probably would not yield any results - chkdsk would not be able to recover anything which is not detected as corrupt.

Data corruptions like the ones you are seeing would have a couple of possible sources:

1. bit flips in software RAID algorithm execution before writing data
2. bit flips in hardware implementation when writing data
3. bit flips on magnetic media
4. bit flips in hardware implementation when reading data

You should choose a methodic approach to exclude the ones you can exclude:

• Number 4 - bit flips when reading - should quite easily be recognizeable by the fact that the flips would occur for different areas of data, so md5 or sha1 hashes would differ from time to time you are trying to compute them over a large file

• Number 3 - flips on magnetic media - are rather unlikely to go undetected since every hard disk does include forward error correction algorithms as well as error detection checksums and you definitely would see unrecoverable sector read errors by a number of magnitudes more often than bit flips sliding through - taking a look at the SMART unrecoverable read errors should be sufficient to exclude this one

• Number 2 - this one can be quite hard to detect. Although the SATA protocol protects the transmitted data by error correction algorithms where the logic of the 3. case would apply and slow any transmission to a crawl before letting through a flipped bit sector, the corruption might happen somewhere else and go undetected - in buffers for example.

• Number 1 - I would regard this the most likely case. Either a bug in the implementation or (more likely as a bug of this significance probably would be noted and documented by somewhere else 4 years after the OS release) a hardware failure like defective RAM could cause this kind of bit flips. Do a couple of memtest passes to exclude the RAM, especially if you are not using ECC memory. Re-run your tests in a similar environment with the same software configuration (preferably an image of your system) to exclude a software-based cause.

You also might extend your tests to copying 15 GB worth of smaller files just to see if the corruption also would affect one of them after a certain amount of data written. If this were the case (which appears likely given your description), you should assume that similar corruption has happened to data already placed on your disks - try comparing to original data or known-good cryptographic hashes with larger files to estimate the degree of corruption.

Also, the ability to run a re-calculation of the XOR checksums and comparing them to the parity data stored on the disks would have been nice and most RAID 5 systems offer this functionality which is typically called "scrubbing". With Windows, there seems to be no way to do this out-of-the-box. I was only able to find data recovery services doing this for you.

Good hunting.

Answer 2

chkdsk (without the /R) is going to be the first thing that you run. If there's a problem run chkdsk /R. You might be having a problem with the disk controller and not the disks. If chkdsk reports different blocks are bad each time to you run it (without the /R) then it's a controller not a disk problem.