Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In a previous message I have asked how to rebuild a faulty disk in a RAID 5 array with 4 disks. I have mounted a new drive (drive 4) in place of the faulty one and started a rebuild. During the rebuild, another disk (drive 2) started throwing ECC errors and timeouts. AT 95% of the rebuild process, the computer rebooted and hang at the start screen, with the controller (3ware 9500s) showing an error (drive 2 not found) and a typical noise coming from the faulty drive (drive 2), could be heard. I have turned off & on the PC few times, no changes. Then I have left the PC off for an hour. Turned on again, his time the missing drive (drive 2) was back in place. I could bot the operating system awaiting for the rebuild, started automatically from the controller. At a certain point, the controller started gave a rebuild error and halted the rebuild process. The server is now running with drive 2 with errors and drive 4 with a OK status, but degraded as the rebuild process could not complete. It looks like I'm at a dead end: at least 3 drives need to be ok to make things good, however one drive has errors and one drive is not rebuild.. What can I try?

share|improve this question
add comment

4 Answers 4

Your best bet is to restore from backups. But I'm guessing you don't have those, or you wouldn't be asking the question.

So, failing backups, your next best bet is to copy as much of the data off as possible (from the sounds of things you'll have at least a couple unreadable sectors that won't be copyable) with whatever method you favor - file copy, disk image, disk-level copy, etc. Then once you have your data, you can replace the faulty drives, create a new RAID array and copy your data back.

Failing that, you can go through the expensive process of professional data recovery or just accepting your data loss and moving on, depending on how much your data is worth to you.

share|improve this answer
    
I do have backups, I just wanted to avoid re-installing everything, as the PC is rather old; will need a new PC, a new OS, a newer SQL server version....3/6 days downtime....... –  Riccardo Mar 6 '13 at 20:52
add comment

The easiest thing would be to restore from backup. But you're probably asking this question because you don't have one. In that case you are going to call a disk drive recovery center and see what they can do for you.

When you finally get this rebuilt you'll learn the real value of a backup system that works.

share|improve this answer
    
As I wrote, I do have the backup. See my comment to HopelessN00b –  Riccardo Mar 6 '13 at 21:11
add comment

Can you show the output of twcli /c0 show all?

If drive 2 is in ECC-ERROR state, you can possibly continue the rebuild by telling the controller to ignore the ECC errors on drive 2.

@Sergey Vasilov's answer in this thread What does 3Ware's tw_cli mean by a "DEGRADED" disk vs "ECC-ERROR"? has the right information. (I used to know this offhand, but had to look up the commands, and Sergey's answer had the first hit in a google search so I'll give him the credit). Because it's always better to actually quote the answer:

/cx/ux start rebuild disk=p [ignoreECC]
/cx/ux set ignoreECC=on|off

Even if this lets you rebuild the array, you may still have filesystem corruption, or dataloss. Or you may not.

share|improve this answer
    
Assuming I could force the rebuild ignoring errors, would there be a way to know where the failure occurred? –  Riccardo Mar 6 '13 at 20:59
    
I don't have any ideas for you finding out which blocks were affected, although I suspect there is a way. I'd definitely start by running a fsck on the filesystems though, as that is very likely to catch most of the errors that result. –  Daniel Lawson Mar 6 '13 at 21:28
    
Thanks Daniel. How about the way you suspect? –  Riccardo Mar 6 '13 at 21:50
    
Follow me on this: I have several backups with user data. The backups should be good, as no error occurred while saving data. So, I could perform the rebuild ignoring ECC errors, and restore saved data. Doing so it will ensure that backup data will be back in place. What could left out is damaged system/program files. In this case, sooner or later problems will pop out, and I could simply reinstall screweed files –  Riccardo Mar 6 '13 at 22:07
    
Like I say, I can't really help you find it out. I suspect there is a way to find out from the RAID controller which LBAs were affected, which in turn lets you find out which blocks on the filesystem, which in turn lets you see what files, if any, were affected. I suspect this way is possible because I've seen it done elsewhere in slightly different circumstances, but I really cannot give you any more information than this. –  Daniel Lawson Mar 6 '13 at 22:42
show 4 more comments

@Daniel this is the output from tw_cli

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    DEGRADED       -       -       64K     698.461   ON     ON

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     233.76 GB   490234752     V503YE9G
p1     ECC-ERROR        u0     233.76 GB   490234752     V503Y7VG
p2     OK               u0     233.76 GB   490234752     V503Y4GG
p3     DEGRADED         u0     465.76 GB   976773168     WD-WCAYUJ776908

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       255    18-Nov-2006
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.