0

I am encountering some problems with my RAID-5 array, using mdadm and Debian.

First of all, I lost a drive (completely, it is not even recognized by the BIOS), then I replaced it with a new one; rebuilding has started but has been interrupted by a read error on the second disk (and this one has been removed):

raid5:md0: read error not correctable (sector 1398118536 on sdd)

I suppose that this one will die in the next few days, but I want to re-add it to work with a degraded array to perform some backups (only few sectors are broken and I hope to save a maximum of data before it fails).

Here are my disks, in the RAID order:

  1. sdc - OK
  2. sdd - (the one with read errors, removed from array while rebuilding)
  3. sde - (the one that died, replaced by a spare, but obviously interrupted while rebuilding => I'm not convinced about its data integrity)
  4. sdf - OK

The fact is that I can't re-add sdd to the array, using this command:

# mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sdf1 --force --run
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.
# mdadm -D /dev/md0 
/dev/md0:
        Version : 0.90
  Creation Time : Tue Aug 24 14:20:39 2010
     Raid Level : raid5
  Used Dev Size : 1465039488 (1397.17 GiB 1500.20 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun Oct 23 01:57:22 2011
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 128K

           UUID : 01017848:84926c43:1751c931:a76e1cde (local to host tryphon)
         Events : 0.131544

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       0        0        1      removed
       2       0        0        2      removed
       3       8       81        3      active sync   /dev/sdf1

       4       8       49        -      spare   /dev/sdd1

As you can see, sdd is recognized as a spare instead of being in sync as RAID device #1.

And I have no idea how to tell mdadm that sdd is RAID device #1.

If anyone has any idea, it would be so great!

Thanks.


PS: if this helps, here is the output of mdadm disk examination:

# mdadm -E /dev/sd[cdef]1
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 01017848:84926c43:1751c931:a76e1cde (local to host tryphon)
  Creation Time : Tue Aug 24 14:20:39 2010
     Raid Level : raid5
  Used Dev Size : 1465039488 (1397.17 GiB 1500.20 GB)
     Array Size : 4395118464 (4191.51 GiB 4500.60 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0

    Update Time : Sun Oct 23 01:57:22 2011
          State : clean
 Active Devices : 2
Working Devices : 3
     Failed Devices : 2
  Spare Devices : 1
       Checksum : dfeeeace - correct
         Events : 131544

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8       49        4      spare   /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 01017848:84926c43:1751c931:a76e1cde (local to host tryphon)
  Creation Time : Tue Aug 24 14:20:39 2010
     Raid Level : raid5
  Used Dev Size : 1465039488 (1397.17 GiB 1500.20 GB)
     Array Size : 4395118464 (4191.51 GiB 4500.60 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0

    Update Time : Sun Oct 23 01:57:22 2011
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 2
  Spare Devices : 1
       Checksum : dfeeeae0 - correct
         Events : 131544

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     4       8       49        4      spare   /dev/sdd1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8       49        4      spare   /dev/sdd1
/dev/sde1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 01017848:84926c43:1751c931:a76e1cde (local to host tryphon)
  Creation Time : Tue Aug 24 14:20:39 2010
     Raid Level : raid5
  Used Dev Size : 1465039488 (1397.17 GiB 1500.20 GB)
     Array Size : 4395118464 (4191.51 GiB 4500.60 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Sat Oct 22 22:11:52 2011
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 2
  Spare Devices : 1
       Checksum : dfeeb657 - correct
         Events : 131534

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     4       8       65        4      spare   /dev/sde1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8       65        4      spare   /dev/sde1
/dev/sdf1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 01017848:84926c43:1751c931:a76e1cde (local to host tryphon)
  Creation Time : Tue Aug 24 14:20:39 2010
     Raid Level : raid5
  Used Dev Size : 1465039488 (1397.17 GiB 1500.20 GB)
     Array Size : 4395118464 (4191.51 GiB 4500.60 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0

    Update Time : Sun Oct 23 01:57:22 2011
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 2
  Spare Devices : 1
       Checksum : dfeeeb04 - correct
         Events : 131544

         Layout : left-symmetric
     Chunk Size : 128K

      Number   Major   Minor   RaidDevice State
this     3       8       81        3      active sync   /dev/sdf1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       0        0        1      faulty removed
       2     2       0        0        2      faulty removed
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8       49        4      spare   /dev/sdd1

1 Answer 1

1

What you need first is off-RAID copy of sdD. Use dd_rescue, for e. g.. Don't use this disk in that RAID while recovering it.

When you have this copy, use it to start the array w/o sdE — put instead of it keyword missing. Two hints how this can be done, even if direct way with --force fails:

1) You can re-create your RAID with --assume-clean. (Don't forget this option since with it only superblocks will be updated, not parity).

2) You can -A assemble array.

In both cases you have to provide absolutley the same config options (layout, chunk size, disk order, …) as your broken RAID has. In fact I'd recommend starting with -A-ssembling, since it even doesn't update superblock, while letting you access your data. Only when you're sure it's properly assembling, you can make it persistent with assume-clean re-creation.

As soon as you would have running RAID with 3 disks, just put your sdE instead of missing one.

You must log in to answer this question.