soft raid6 recover second disk fault

Question

I had a disk failure in a hosted system and had the failed drive replaced. During recovery, a disk error on a different drive occurred.

when the original error occurred:

md2 : active raid6 sdf3[5](F) sdd3[3] sdg3[6] sdc3[2] sdb3[7] sde3[4] sdd3[3] sda3[0]
      104849920 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/6] [UUUUU_U]

adding drive after repair:

root@rescue ~ # mdadm /dev/md2 -a /dev/sdf3
mdadm: added /dev/sdf3
root@rescue ~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] 
md2 : active raid6 sdf3[7] sda3[0] sdg3[6] sde3[4] sdd3[3] sdc3[2]
      104849920 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/5] [U_UUU_U]
      [>....................]  recovery =  0.9% (200576/20969984) finish=5.1min speed=66858K/sec

it seems that sda3 has disappeared from the array

rebuild completed:

md2 : active raid6 sdf3[7](S) sda3[0] sdg3[6] sde3[4](F) sdd3[3] sdc3[2]
      104849920 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/4] [U_UU__U]
According to the error log the rebuild probably stopped at the error:
Jul 18 13:17:02 rescue kernel: [ 3648.976435] sd 6:0:0:0: [sde] Unhandled sense code
Jul 18 13:17:02 rescue kernel: [ 3648.976441] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 18 13:17:02 rescue kernel: [ 3648.976445] Sense Key : Medium Error [current] [descriptor]
Jul 18 13:17:02 rescue kernel: [ 3648.976451]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jul 18 13:17:02 rescue kernel: [ 3648.976464] sd 6:0:0:0: [sde]  
Jul 18 13:17:02 rescue kernel: [ 3648.976470] sd 6:0:0:0: [sde] CDB: 
Jul 18 13:17:02 rescue kernel: [ 3649.063660] md/raid:md2: read error not correctable (sec
tor 13785320 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063664] md/raid:md2: read error not correctable (sector 13785328 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063667] md/raid:md2: read error not correctable (sector 13785336 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063670] md/raid:md2: read error not correctable (sector 13785344 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063672] md/raid:md2: read error not correctable (sector 13785352 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063675] md/raid:md2: read error not correctable (sector 13785360 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063678] md/raid:md2: read error not correctable (sector 13785368 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063681] md/raid:md2: read error not correctable (sector 13785376 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063684] md/raid:md2: read error not correctable (sector 13785384 on sde3).
Jul 18 13:17:02 rescue kernel: [ 3649.063748] ata7: EH complete
Jul 18 13:17:02 rescue kernel: [ 3649.121786] md: md2: recovery done.

At that point, is there any way to recover (such as adding the sda3 back in)?

Answer 1

You are now three HDDs (not two: note state is [U_UU__U]) down on a software RAID6, two of which seem to have failed in the middle of a RAID rebuild. It's time to skip most of that hardware and restore from backups.

Answer 2

Data recovery is usually possible in these cases. You should be able to remake the RAID in degraded mode and copy most of your data but at some point in the copy process the RAID will likely crash. In which case you can remake the RAID again and copy additional data by avoiding the directory that crashed the RAID.

Or for best results you can take images of the drive(s) that are failing. And then perform recovery work using the images.

Check the SMART status of your drives with smartctl. It should help you verify which drives are good and which are having problems.

The SMART info plus the output from mdadm --examine will help determine the best way to remake the RAID.