4

I've got a 4-drive RAID 10 array that has just had a drive failure. I ignorantly have never practiced how to recover from a failure (I'm a programmer, just keeping this server as a hobbyist) so I'm having to learn this all the hard way right now.

I managed, through Google and this site (thank you guys!) to figure out how to fail, remove, add and resync the new drive, but it keeps failing during the resync and simply marking the new disk as a spare.

With more Googling and some more command-line-fu, I've figured out that the remaining "good" drive actually has some bad sectors producing read errors during the sync, so mdadm is aborting and marking as spare.

I've used badblocks to confirm the presence of bad sectors (there appear to be quite a few), but I have no idea if those sectors are actually in use or not (so far, I haven't noticed any corrupt data). I've also read that fsck can potentially repair this data, but I've also read that it has the real possibility of completely fubaring the drive, too. As such, I haven't tried it, yet.

I've tried using mdadm's --force flag to ignore these errors during re-sync, but it doesn't appear to help at all.

I have all my critical data backed up already, but there is a large amount of non-critical data that I'd really rather not lose if it can be avoided (it's all replaceable, but would take a very long time). Additionally, my critical backups are all in the cloud, so even restoring those, while easy, would be very time-consuming.

Also, if needed, I have one more unused, new replacement drive on hand.

Below is all the info about the system that I know to give. Please let me know if you need more! How do I get this array fully rebuilt?


Drive Layout

sda + sdb = RAID1A (md126)

sdc + sdd = RAID1B (md127)

md126 + md127 = RAID10 (md125)

The problem array is md126, the new unsynced drive is sdb, and the problem drive is sda


root@vault:~# cat /proc/mdstat

Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]
md125 : active raid0 md126p1[1] md127p1[0]
      5860528128 blocks super 1.2 512k chunks

md126 : active raid1 sda1[1] sdb1[2](S)
      2930265390 blocks super 1.2 [2/1] [U_]

md127 : active raid1 sdc1[1] sdd1[0]
      2930265390 blocks super 1.2 [2/2] [UU]

unused devices: <none>

root@vault:~# parted -l

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid

root@vault:~# sudo mdadm --detail /dev/md126

/dev/md126:
        Version : 1.2
  Creation Time : Thu Nov 29 19:09:32 2012
     Raid Level : raid1
     Array Size : 2930265390 (2794.52 GiB 3000.59 GB)
  Used Dev Size : 2930265390 (2794.52 GiB 3000.59 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jun  2 11:53:44 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           Name : :RAID1A
           UUID : 49293460:3199d164:65a039d6:a212a25e
         Events : 5200173

    Number   Major   Minor   RaidDevice State
       1       8        1        0      active sync   /dev/sda1
       2       0        0        2      removed

       2       8       17        -      spare   /dev/sdb1

Edit: Here are the contents of the kernel log during the failed recovery process.

root@vault:~# mdadm --assemble --update=resync --force /dev/md126 /dev/sda1 /dev/sdb1

root@vault:~# tail -f /var/log/kern.log

Jun  5 12:37:57 vault kernel: [151562.172914] RAID1 conf printout:
Jun  5 12:37:57 vault kernel: [151562.172917]  --- wd:1 rd:2
Jun  5 12:37:57 vault kernel: [151562.172919]  disk 0, wo:0, o:1, dev:sda1
Jun  5 12:37:57 vault kernel: [151562.172921]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 12:37:57 vault kernel: [151562.173858] md: recovery of RAID array md126
Jun  5 12:37:57 vault kernel: [151562.173861] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  5 12:37:57 vault kernel: [151562.173863] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Jun  5 12:37:57 vault kernel: [151562.173865] md: using 128k window, over a total of 2930265390k.
Jun  5 12:37:57 vault kernel: [151562.248457]  md126: p1
Jun  5 12:37:58 vault kernel: [151562.376906] md: bind<md126p1>
Jun  5 13:21:52 vault kernel: [154196.675777] ata3.00: exception Emask 0x0 SAct 0xffe00 SErr 0x0 action 0x0
Jun  5 13:21:52 vault kernel: [154196.675782] ata3.00: irq_stat 0x40000008
Jun  5 13:21:52 vault kernel: [154196.675785] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:52 vault kernel: [154196.675791] ata3.00: cmd 60/00:48:a2:a4:e0/05:00:38:00:00/40 tag 9 ncq 655360 in
Jun  5 13:21:52 vault kernel: [154196.675791]          res 41/40:00:90:a7:e0/00:05:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:52 vault kernel: [154196.675794] ata3.00: status: { DRDY ERR }
Jun  5 13:21:52 vault kernel: [154196.675797] ata3.00: error: { UNC }
Jun  5 13:21:52 vault kernel: [154196.695048] ata3.00: configured for UDMA/133
Jun  5 13:21:52 vault kernel: [154196.695077] sd 2:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:52 vault kernel: [154196.695081] sd 2:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:52 vault kernel: [154196.695085] sd 2:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:52 vault kernel: [154196.695090] sd 2:0:0:0: [sda] tag#9 CDB: Read(16) 88 00 00 00 00 00 38 e0 a4 a2 00 00 05 00 00 00
Jun  5 13:21:52 vault kernel: [154196.695092] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:52 vault kernel: [154196.695111] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.675248] ata3.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0
Jun  5 13:21:55 vault kernel: [154199.675252] ata3.00: irq_stat 0x40000008
Jun  5 13:21:55 vault kernel: [154199.675255] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:55 vault kernel: [154199.675261] ata3.00: cmd 60/08:c0:8a:a7:e0/00:00:38:00:00/40 tag 24 ncq 4096 in
Jun  5 13:21:55 vault kernel: [154199.675261]          res 41/40:08:90:a7:e0/00:00:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:55 vault kernel: [154199.675264] ata3.00: status: { DRDY ERR }
Jun  5 13:21:55 vault kernel: [154199.675266] ata3.00: error: { UNC }
Jun  5 13:21:55 vault kernel: [154199.676454] ata3.00: configured for UDMA/133
Jun  5 13:21:55 vault kernel: [154199.676463] sd 2:0:0:0: [sda] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:55 vault kernel: [154199.676467] sd 2:0:0:0: [sda] tag#24 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:55 vault kernel: [154199.676471] sd 2:0:0:0: [sda] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:55 vault kernel: [154199.676474] sd 2:0:0:0: [sda] tag#24 CDB: Read(16) 88 00 00 00 00 00 38 e0 a7 8a 00 00 00 08 00 00
Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.
Jun  5 13:21:55 vault kernel: [154199.855992] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.855995]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.855998]  disk 0, wo:0, o:1, dev:sda1
Jun  5 13:21:55 vault kernel: [154199.856000]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 13:21:55 vault kernel: [154199.872013] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.872016]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.872018]  disk 0, wo:0, o:1, dev:sda1
8
  • Can we see what the kernel logs when the rebuild fails?
    – MadHatter
    Commented Jun 2, 2016 at 22:14
  • If bad block from a good drive prevent the rebuild, then I would not thrust your array, as even if it sync back, you might have corruption and the worst is that the controller didnt flagged the other drive fail.. bad sign.
    – yagmoth555
    Commented Jun 2, 2016 at 22:50
  • @MadHatter I'd love to, but not sure which log to tail?
    – KOGI
    Commented Jun 3, 2016 at 3:43
  • @yagmoth555 agreed. However, I'd rather have some partially corrupted data than none at all.
    – KOGI
    Commented Jun 3, 2016 at 3:44
  • Which distro are you using, and have you messed with the syslog/rsyslog setup?
    – MadHatter
    Commented Jun 3, 2016 at 10:29

2 Answers 2

4

The critical lines are

Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.

To recover md126 the kernel needs to copy sda1 to sdb1 - but it's having a read error on sda, which is the only surviving half of that mirror. This array is toast. It's time to shotgun /dev/sda and restore from backups (or, absent backups, save what you can from the existing array before shotgunning it).

Edit: If you're trying to get data off the failing drive, tools like safecopy may come in useful (disclaimer: I have no connection with the authors or the project).

6
  • Is there any way to just force no data into that sector and continue? It would corrupt whatever file was using that sector, but allow the sync to continue...
    – KOGI
    Commented Jun 6, 2016 at 16:25
  • 1
    You're assuming it was a file. It might have been a directory entry, in which case you could lose a lot more.
    – MadHatter
    Commented Jun 6, 2016 at 17:42
  • So, is there any way to copy as much info as possible to a new RAID1 array, and when that's complete swap that new array into the RAID0? Basically replace /dev/md126 with a new array, but keep /dev/md127 and /dev/md125 in place?
    – KOGI
    Commented Jun 8, 2016 at 18:13
  • I do not believe so, no. md125 has the FS, and is RAID0, so must be recreated if any of its components is reinitialised.
    – MadHatter
    Commented Jun 9, 2016 at 5:07
  • So, my ENTIRE array is toast, then?
    – KOGI
    Commented Jun 9, 2016 at 15:15
3

Maybe give this guys approach a try if you are brave (or desperate) enough: http://matrafox.info/how-to-get-md-raid-array-to-rebuild-even-if-read-errors.html

His hack forces the badblocks to be reallocated on the "good" disk by writing nonsense to them. Which shouldnt matter for your data, since the sector in question is dead anyways, and the disks controller remaps the bad sector to a spare one.

Use at your own risk!

2
  • How do I know if my bad sectors are in use or not?
    – KOGI
    Commented Jun 3, 2016 at 3:44
  • 1
    There are various ways but the short answer is: if you read all your data your system will produce read errors for every bad sector in use in the dmesg / messages log.
    – tim
    Commented Jun 3, 2016 at 5:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .