How to fix my broken raid10 array

Question

I have a RAID 10 consisting of 4x3TB drives. It had become degraded (sdd had dropped off). I thought I had fixed it and it was rebuilding, but now (I think) sde appears to be causing problems.

root@tower:~# mdadm -A --force /dev/md127 /dev/sd[b-e]
mdadm: /dev/md127 assembled from 2 drives and  1 rebuilding - not enough to start the array.

This is weird - everything I've googled suggests I should see the device name /dev/md127 here, but no:

root@tower:~# mdadm --examine --scan
ARRAY /dev/md/4x3TB metadata=1.2 UUID=acaef996:1ea7102b:9cd52d00:af0ef09e name=omv:4x3TB

Can anyone recommend a suitable course of action? Here's the output from each device:

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 08ee73a5:86c62890:3dba03bb:522093e8

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : d282c882 - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 05c34f6d:8bead38f:99a01f60:e6518283

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : da150f20 - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x12
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
Recovery Offset : 670607488 sectors
          State : clean
    Device UUID : 6d632b5f:5a22a515:5c63d798:f1b590ef

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : 1e80b6bc - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : e6a95b56:6541b0e1:6f3b9ce7:a392ceb9

    Update Time : Sun Jun  7 14:06:33 2015
       Checksum : d2c1c56c - correct
         Events : 2589703

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A ('A' == active, '.' == missing)

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdb[4](S) sdd[5](S) sdc[1](S)
      8790796680 blocks super 1.2

unused devices: 

Answer 1

Big fat warning:

Anything you do with your array (including stuff I suggest) may lead to a complete data loss. If there is a really valuable (expensive to regain) not-backed-up data, let someone experienced handle the situation for you. Including making binary copies of all four drives.

From your output it seems you have

   Device Role : Active device 3

twice in your --examine output. That would point to an attempt at recovery, but done wrong.

From /proc/mdstat it looks like your array gets assembled, but not run. There are some very weird device numbers (4,5,1), while your drives should be 0,1,2,3. That as well suggests, there are discrepancies in the metadata.

Another point of interest is the Events counters in the drives metadata. Those aggree for sd[b-d], but seems to be behind on sde. Are you sure that sdd was the drive that dropped out? As this would rather point to sde being out of the array for some time.

You could try assembling the array without the dropped out drive (mdadm -A -R /dev/md127 /dev/sd[bcd] or mdadm -A --force -R /dev/md127 /dev/sd[bce]). Doing so could prevent the conflict. If that works, and even if it works, do not write anything to the array, backup your data, and then try adding sdd back as a hot spare.

If it does not work, you might try updating your question with output of mdadm -D /dev/md127 after assembling the array (both suggested ways actually).

Answer 2

It seems like you've lost a whole subarray of your Raid10 (Drives sdd, sde were a mirror before?!) - if this would be the case, your data is lost.