Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Quick info: Dell R410 with 2x500GB drives in RAID1 on H700 Adapter

Recently one of drives in RAID1 array on server failed, lets call it Drive 0. RAID controller marked it as fault and put it offline. I replaced faulty disk with new one (same series and manufacturer, just bigger) and configured new disk as hot spare.

Rebuild from Drive1 started immediately and after 1.5 hour I got message that Drive 1 failed. Server was unresponsive (kernel panic) and required reboot. Given that half hour before this error rebuild was at about 40%, I estimated that new drive is not in sync yet and tried to reboot just with Drive 1.

RAID controller complained a bit about missing RAID arrays, but it found foreign RAID array on Drive 1 and I imported it. Server booted and it runs (from degraded RAID).

Here is SMART data for disks. Drive 0 (the one that failed first)

ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    1
  3 Spin_Up_Time            POS--K   142   142   021    -    3866
  4 Start_Stop_Count        -O--CK   100   100   000    -    12
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   086   086   000    -    10432
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    11
192 Power-Off_Retract_Count -O--CK   200   200   000    -    10
193 Load_Cycle_Count        -O--CK   200   200   000    -    1
194 Temperature_Celsius     -O---K   112   106   000    -    31
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   198   000    -    3

And Drive 1 (the drive which was reported healthy from controller until rebuild was attempted)

ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    35
  3 Spin_Up_Time            POS--K   143   143   021    -    3841
  4 Start_Stop_Count        -O--CK   100   100   000    -    12
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   086   086   000    -    10455
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    11
192 Power-Off_Retract_Count -O--CK   200   200   000    -    10
193 Load_Cycle_Count        -O--CK   200   200   000    -    1
194 Temperature_Celsius     -O---K   114   105   000    -    29
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    3
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0

In extended error logs from SMART I found:

Drive 0 has only one error

Error 1 [0] occurred at disk power-on lifetime: 10282 hours (428 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 18 00 00 00 6a 24 20 40 00  Error: IDNF at LBA = 0x006a2420 = 6956064

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 60 00 f8 00 00 00 6a 24 20 40 00 17d+20:25:18.105  WRITE FPDMA QUEUED
  61 00 18 00 60 00 00 00 6a 24 00 40 00 17d+20:25:18.105  WRITE FPDMA QUEUED
  61 00 80 00 58 00 00 00 6a 23 80 40 00 17d+20:25:18.105  WRITE FPDMA QUEUED
  61 00 68 00 50 00 00 00 6a 23 18 40 00 17d+20:25:18.105  WRITE FPDMA QUEUED
  61 00 10 00 10 00 00 00 6a 23 00 40 00 17d+20:25:18.104  WRITE FPDMA QUEUED

But Drive 1 has 883 errors. I see only few last ones and all errors I can see look like this:

Error 883 [18] occurred at disk power-on lifetime: 10454 hours (435 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  01 -- 51 00 80 00 00 39 97 19 c2 40 00  Error: AMNF at LBA = 0x399719c2 = 966203842

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 80 00 00 00 00 39 97 19 80 40 00  1d+00:25:57.802  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 00  1d+00:25:57.779  READ LOG EXT
  60 00 80 00 00 00 00 39 97 19 80 40 00  1d+00:25:55.704  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 40 00  1d+00:25:55.681  READ LOG EXT
  60 00 80 00 00 00 00 39 97 19 80 40 00  1d+00:25:53.606  READ FPDMA QUEUED

Given those errors, is there any way I can rebuild RAID back, or should I make backup, shutdown server, replace disks with new ones and restore it? What about if I dd faulty disk to new one from linux running on USB/CD?

Also, if anyone have more experiences, what could be causes for those errors? Crappy controller or disks? Disks are about 1 year old, but it is pretty unbelievable to me that both would die within so short timespan.

share|improve this question
    
The time to make a backup is before your disks fail. From your description it seems quite likely you'll have lost data. –  Michael Hampton Oct 5 '12 at 3:07
    
Who said there is no backup? Disk still works and last month of backups is stored on NAS. I was just wondering if there is any way to rebuild RAID even with faulty disk. I attempted another rebuild after firmware upgrade, but it failed at 98%, so I guess i will have to reinstall system and restore from backup –  overlord_tm Oct 9 '12 at 15:52
    
You can't rebuild when both disks are faulty. –  Michael Hampton Oct 9 '12 at 15:53

1 Answer 1

up vote 1 down vote accepted

Actually, if the disks were both from the same batch from the manufacturer, it's not all that surprising that they'd fail around the same time.

They've had the same manufacturing process, environment, and usage patterns. That's why I usually try to order identical model drives from different vendors.

My preferred course of action here is to contact the manufacturer, replace with better disks, restore from backup.

Nothing wrong with DD'ing either, but I'm usually in need of getting the service up ASAP.

Back in the day of the IBM Deskstars fiasco, I had an entire set of 8 disks go bad all within 6 weeks after 4 years of usage. I barely got out of that with my data intact.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.