4

I have a Windows Server 2003 box on my small network. In it is a Promise Fasttrak RAID controller and two parallel ATA Western Digital drives in a RAID-1 (mirror) configuration. When I set this up I expected that it would be a reliable storage system, and the RAID controller would tell me when there's an issue so I could react.

However, it is failing on both counts now. When I copy files from that server, I find large files have been corrupted. For instance, I was recently copying the XP SP3 network install (~320MB) to another PC. The extraction failed. I thought that odd since I'd used that executable before. So I copied it from the network again twice, and using FileAlyzer, I discovered that the MD5 & SHA1 hashes of the 3 different copies varied. I performed similar tests from other PCs on my network, and I could replicate the fault. Worse, the RAID BIOS never complained about anything being wrong! Which leads me to believe the controller itself may be bad. (Note: I don't think it's the network, since other PCs can reliably copy files to each other.)

But my question is: What kind of tools exist for Windows to "certify" a file system is behaving reliably, RAID or otherwise?

For instance, I purchased a tool called GoldMemory to run an exhaustive memory test when I build a new PC. I won't trust a new PC until it survives 24 hours under GoldMemory with no memory errors. I also purchased Steve Gibson's SpinRite to test individual ATA disks.

Is there a tool I can run within Windows to test an NTFS file system, whether RAID-based or not, that will repeatedly read and write and check for corruption?

I can't trust my current server as is, and if I swap out components to try to repair, or else build a new system, I'd like to be reasonably sure my file systems are operating reliably before betting the farm. While I'd like to trust that a brand-name RAID controller and decent hard drives would be reliable, I now need to take a Horatio Caine approach: "Trust, but verify."

Thanks for your help! :-)


UPDATE:

So, I ran some local tests on the server (within cygwin) to rule out the network as the problem. This should give you an idea of what I am contending with. The problem happens most of the time with BIG files. (The one below is 462MB.)


$ md5sum VMware-workstation-6.5.2-156735.exe
7bf6145eb7d3e4fbcc945d87017fb6bd *VMware-workstation-6.5.2-156735.exe

$ for (( c=1; c<=50; c++ )); do md5sum VMware-workstation-6.5.2-156735.exe; done
545c2f8e9363823af3aa703a1cbd35e3 *VMware-workstation-6.5.2-156735.exe
b47d4aa75aae27264cfd6396fbfe646a *VMware-workstation-6.5.2-156735.exe
b47d4aa75aae27264cfd6396fbfe646a *VMware-workstation-6.5.2-156735.exe
... etc... (repeats)

$ for (( c=1; c<=50; c++ )); do md5sum VMware-workstation-6.5.2-156735.exe; done
9d2fbb3fa46194f6915d6328f0881a24 *VMware-workstation-6.5.2-156735.exe
9d2fbb3fa46194f6915d6328f0881a24 *VMware-workstation-6.5.2-156735.exe
... etc... (repeats)

$ for (( c=1; c<=50; c++ )); do md5sum VMware-workstation-6.5.2-156735.exe; done
512181c3838e91a02a92280462e2f4c3 *VMware-workstation-6.5.2-156735.exe
512181c3838e91a02a92280462e2f4c3 *VMware-workstation-6.5.2-156735.exe
...(repeats a dozen or so times, then changes!)
7a84da59a83f203506244e23507bb4df *VMware-workstation-6.5.2-156735.exe
7a84da59a83f203506244e23507bb4df *VMware-workstation-6.5.2-156735.exe
... aargh!

5
  • Note: I have actually used ad-hoc approaches (e.g. cygwin find + xargs + md5sum commands, rinse, repeat) to confirm the problem, too. But what I'm really wondering here is if there are specific tools or techniques commonly used by admins to validate a file system is reliable before it is put into production use? Or, do we mostly just implicitly trust our RAIDs, cross our fingers, and hope we don't see symptoms? :-) Jul 5, 2009 at 14:00
  • 1
    Step back a moment. Are both hard disks server qualified (having TLER set to server values), or are you using desktop drives? You mentioned Western Digital, some of their RE drives from a couple of years ago require firmware updates to be stable in RAID. Finally, does Promise provide any Windows based RAID management tool? If such a tool doesn't come with the controller, consider upgrading to a reputable manufacturer such as 3Ware.
    – kmarsh
    Jul 6, 2009 at 16:30
  • Specifically, WD4000YS and WD5000YS firmwares are suspect until upgraded. There may be others.
    – kmarsh
    Jul 6, 2009 at 16:32
  • The quote "Trust, but Verify" was actually promoted by Ronald Reagan, as a reference to nuclear disarmament talks he was holding with the Soviet Union, although he's not the first to say it. en.wikipedia.org/wiki/Trust,_but_Verify Jul 11, 2009 at 20:51
  • Can you specify the model of RAID card? Some chipsets are known to be "defective by design". Jul 11, 2009 at 20:56

5 Answers 5

3

It should be easy to setup a shell script that repeats copying a file on the server and recalculates the checksum of each copy. After it fills your sever, you check all the checksums by hand.

My experience is that raid controllers that have Promise written on the outside are broken on the inside. Get rid of it. Sometimes even the Promise controllers only make a driver driven software raid. Try Areca or so.

If you plan for raid, put a pricetag on your data. Then put a pricetag on not being able to work a few days. Then check for prices of good raid controllers.

1
  • raid is DEFINITELY an area where it is not a good idea to skimp. I'd go for "middle of the curve" pricing at minimum (seems like around $200-$300 is where decent raid controllers start). Jul 7, 2009 at 1:12
3

You don't need to cough money for ram testing tools, because memtest86+ rules and is free. To test the filesystem integrity you could use afick, it works fine for me (but I didn't used it much on windows, though).

What is the make of your drives? a priori, I'd suspect the Promise card. They have a very long and painful history of absolutely shitty products, with abysmal performance, data corruption, buggy drivers, and various combinations of all of these.

2
  • WD makes drive known to work most of the time :) The culprit must almost certainly lie in the Promise card... If it has some sort of RAM DIMM, you could try to swap it with another known good; if not, you probably should be replacing it.
    – wazoox
    Jul 5, 2009 at 14:40
  • Thanks for the suggestion - I'll have a look. Yes, I like WD drives. Even the odd time I've had one fail, WD was easy to deal with to get it replaced. Jul 6, 2009 at 13:18
2

Are you sure it's the RAID controller? I've experienced similar problems that had to do with the network drivers / card failing.

You say that other PCs can copy files to each other, but that doesn't mean the server network card (or driver) isn't fritzy.

7
  • In my experience, I've had 2 network cards that failed out of 3000, while I had 3 Promise RAID cards failing out of a dozen. I know the sample is less representative but... :) However you're right, it could be useful to check that the corruption appears even when working locally, or else it could be the server network adapter.
    – wazoox
    Jul 6, 2009 at 13:46
  • Jeff Atwood ... great suggestion, I will do some local tests for sure. (And please forgive me for a moment as I lapse into Bill & Ted's "I'm not worthy!" "I'm not worthy!" :-) Jul 6, 2009 at 15:29
  • 2
    at the very least it has to be ruled out, otherwise you have another variable. Jul 6, 2009 at 18:36
  • I'm running some tests. So far, it (surprisingly) looks relatively reliable when tested locally. If it is the network, it could also be the NAT router I have between this box and the rest of my network. (The server is in my DMZ.) Jul 7, 2009 at 0:03
  • 1
    Local tests turned out to have failed. The early results were misrepresentative. I've added more detail in the question above. Jul 11, 2009 at 0:22
1

Chkdsk has always been my first-line tool for fixing NTFS. Comes in the box, and works like a charm. Full disclosure: I very seldom have the need to verify filesystems, so I've never needed another tool.

Out of the 100 or so servers I manage I've needed to use it once and that time was because of problems caused by a bad data sync on the SAN, not the RAID card. I'm with everyone else saying ditch the Promise card and get something better.

0

Try the Robocopy to copy large files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .