Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a server running Windows Server 2008 R2, with a (windows native) software raid-5 array. The array consists of 7x 1TB Western Digital RE3 and RE4 drives. I have offline backups of this array.

The problem is this: I noticed a few days ago after copying a large file to the disk that there was an integrity issue with that file - it was a ~12GB file that I had downloaded via uTorrent. After moving it to the raid array, I used uTorrent to relocate the download location, and performed a re-check so I could seed it from that location. The recheck found that only 6308/6310 chunks of the copied file were intact.

My next step was to write a quick powershell script that would copy files to the array, while performing a SHA1 hash of the original and resultant files and comparing them. Smaller files (100-1000MB) copied over just fine. When I started copying larger data (~15GB), I found that the hash check failed about 2/3rds of the time. The corrupt files had very, very small inconsistencies - less than .01% (EDIT - later experimentation has shown that the corrupted chunks of data are always 60 bytes in length, with one to three generally manifesting per 15GB copied file. Corrupted data appears random, with no consistent pattern of flipped bits). I further eliminated the possibility of networking or client issues by placing this large file on the C:\ of the server, and copying it repeatedly from there to the array, seeing similar results.

Copying the data via explorer, powershell, or the standard windows command prompt yield the same results. None of the copies fail or report any problems. The raid array itself is listed as healthy in disk management.

After a few experiments, I shut down the server and ran memtest overnight. No errors were detected. A basic run of chkdsk found no problems, but I did not use the /R flag, as I was unsure how that might affect a software raid-5 volume.

I next ran Crystal Disk Info to check the smart data on the drives - but found that CDI only detected 5 out of 7 of the disks in the array. I have no idea why. Nevertheless, CDI shows the following "caution" flags on a single one of the drives:

05 199 199 140 000000000001 Reallocated Sectors Count
C5 200 200 __0 000000000001 Current Pending Sector Count

Which is a little bit alarming, but I don't really know what to do with the information. I hardly feel like one reallocated sector could be causing this.

At this point, I'm looking for some guidance on what to do next. I need to determine the cause of this issue, but I'm hesitant to run chkdsk /R or any bootable disk health checkers because I'm afraid they might break the array. I've considered triggering a re-sync of the array, but I'm not actually sure how to do that without doing something silly like manually dropping a disk and then restoring it.

Any advice that could help me ferret out the precise cause of this issue would be greatly appreciated.

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

As none of the operations you are performing is failing, running chkdsk /R probably would not yield any results - chkdsk would not be able to recover anything which is not detected as corrupt.

Data corruptions like the ones you are seeing would have a couple of possible sources:

  1. bit flips in software RAID algorithm execution before writing data
  2. bit flips in hardware implementation when writing data
  3. bit flips on magnetic media
  4. bit flips in hardware implementation when reading data

You should choose a methodic approach to exclude the ones you can exclude:

  • Number 4 - bit flips when reading - should quite easily be recognizeable by the fact that the flips would occur for different areas of data, so md5 or sha1 hashes would differ from time to time you are trying to compute them over a large file

  • Number 3 - flips on magnetic media - are rather unlikely to go undetected since every hard disk does include forward error correction algorithms as well as error detection checksums and you definitely would see unrecoverable sector read errors by a number of magnitudes more often than bit flips sliding through - taking a look at the SMART unrecoverable read errors should be sufficient to exclude this one

  • Number 2 - this one can be quite hard to detect. Although the SATA protocol protects the transmitted data by error correction algorithms where the logic of the 3. case would apply and slow any transmission to a crawl before letting through a flipped bit sector, the corruption might happen somewhere else and go undetected - in buffers for example.

  • Number 1 - I would regard this the most likely case. Either a bug in the implementation or (more likely as a bug of this significance probably would be noted and documented by somewhere else 4 years after the OS release) a hardware failure like defective RAM could cause this kind of bit flips. Do a couple of memtest passes to exclude the RAM, especially if you are not using ECC memory. Re-run your tests in a similar environment with the same software configuration (preferably an image of your system) to exclude a software-based cause.

You also might extend your tests to copying 15 GB worth of smaller files just to see if the corruption also would affect one of them after a certain amount of data written. If this were the case (which appears likely given your description), you should assume that similar corruption has happened to data already placed on your disks - try comparing to original data or known-good cryptographic hashes with larger files to estimate the degree of corruption.

Also, the ability to run a re-calculation of the XOR checksums and comparing them to the parity data stored on the disks would have been nice and most RAID 5 systems offer this functionality which is typically called "scrubbing". With Windows, there seems to be no way to do this out-of-the-box. I was only able to find data recovery services doing this for you.

Good hunting.

share|improve this answer
    
Thanks for your thorough reply. The corrupted files always reproduce the same "wrong" hash once they have been written, so we can eliminate #4. I tried copying about 25 GB worth of smaller (~800MB) files, and found that all of those copied correctly - this makes me think system ram may be the culprit after all, despite a few successful passes in memtest. My plan tonight is to run the system with only one stick of ram at a time, and repeat my hash-copy-hash check of a large file a dozen times each. Since the problem is easily reproducible currently, this should help me narrow it down. –  Fopedush Nov 26 '12 at 16:32
    
@Fopedush If you don't have ECC-protected memory, consider replacing it with ECC memory (if your memory controller allows for it) as well - this at least would give you the option of detection of single-bit errors in RAM. –  the-wabbit Nov 27 '12 at 17:20
    
I haven't had time to do the one-stick of ram at a time test, but I did write myself a little tool to do a binary-diff of giant files. What I've found after testing multiple corrupt copies of a 20gb file is that the corrupted data always appears in chunks of 60 contiguous bytes. The damaged data appears to be total garbage, there isn't any discernible pattern to which bits are being flipped. Comparing two corrupt 60-byte chunks to one another, the pattern of flips did not match. Still can't rule out system memory, but now I know for sure these are not single bit flips. –  Fopedush Nov 27 '12 at 19:17
    
@Fopedush probably not the memory then. Would you mind taking a closer look at your storage controller and its drivers? You could either try updating the firmware / drivers or replace the controller with a different make / model. –  the-wabbit Nov 27 '12 at 22:26
    
After extensive memtesting and running full surface scans on every disk in the array (without incident), I was able to borrow a replacement SATA card from a coworker. Swapping the card out appears to have eliminated the problem - so I must conclude that my disk controller was to blame. I really can't imagine a more insidious problem - unreported write errors so sparse that it took me months to notice. Nevertheless, I'm glad to have the issue isolated. I'm now beginning the process of rebuilding the array from the last good backup. Thank you for your assistance. –  Fopedush Dec 3 '12 at 5:58
show 1 more comment

chkdsk (without the /R) is going to be the first thing that you run. If there's a problem run chkdsk /R. You might be having a problem with the disk controller and not the disks. If chkdsk reports different blocks are bad each time to you run it (without the /R) then it's a controller not a disk problem.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.