1

I have a RAID 10 consisting of 4x3TB drives. It had become degraded (sdd had dropped off). I thought I had fixed it and it was rebuilding, but now (I think) sde appears to be causing problems.

root@tower:~# mdadm -A --force /dev/md127 /dev/sd[b-e]
mdadm: /dev/md127 assembled from 2 drives and  1 rebuilding - not enough to start the array.

This is weird - everything I've googled suggests I should see the device name /dev/md127 here, but no:

root@tower:~# mdadm --examine --scan
ARRAY /dev/md/4x3TB metadata=1.2 UUID=acaef996:1ea7102b:9cd52d00:af0ef09e name=omv:4x3TB

Can anyone recommend a suitable course of action? Here's the output from each device:

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 08ee73a5:86c62890:3dba03bb:522093e8

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : d282c882 - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 05c34f6d:8bead38f:99a01f60:e6518283

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : da150f20 - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x12
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
Recovery Offset : 670607488 sectors
          State : clean
    Device UUID : 6d632b5f:5a22a515:5c63d798:f1b590ef

    Update Time : Sun Jun  7 17:44:09 2015
       Checksum : 1e80b6bc - correct
         Events : 2589917

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.? ('A' == active, '.' == missing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : acaef996:1ea7102b:9cd52d00:af0ef09e
           Name : omv:4x3TB
  Creation Time : Wed Dec 12 13:48:21 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : e6a95b56:6541b0e1:6f3b9ce7:a392ceb9

    Update Time : Sun Jun  7 14:06:33 2015
       Checksum : d2c1c56c - correct
         Events : 2589703

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A ('A' == active, '.' == missing)
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdb[4](S) sdd[5](S) sdc[1](S)
      8790796680 blocks super 1.2

unused devices: 
  • What's in cat /proc/mdstat after assembling it? – Fox Jun 7 '15 at 20:32
  • cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md127 : inactive sdb[4](S) sdd[5](S) sdc[1](S) 8790796680 blocks super 1.2 unused devices: <none> – Rog Jun 7 '15 at 20:36
0

Big fat warning:

Anything you do with your array (including stuff I suggest) may lead to a complete data loss. If there is a really valuable (expensive to regain) not-backed-up data, let someone experienced handle the situation for you. Including making binary copies of all four drives.

From your output it seems you have

   Device Role : Active device 3

twice in your --examine output. That would point to an attempt at recovery, but done wrong.

From /proc/mdstat it looks like your array gets assembled, but not run. There are some very weird device numbers (4,5,1), while your drives should be 0,1,2,3. That as well suggests, there are discrepancies in the metadata.

Another point of interest is the Events counters in the drives metadata. Those aggree for sd[b-d], but seems to be behind on sde. Are you sure that sdd was the drive that dropped out? As this would rather point to sde being out of the array for some time.

You could try assembling the array without the dropped out drive (mdadm -A -R /dev/md127 /dev/sd[bcd] or mdadm -A --force -R /dev/md127 /dev/sd[bce]). Doing so could prevent the conflict. If that works, and even if it works, do not write anything to the array, backup your data, and then try adding sdd back as a hot spare.

If it does not work, you might try updating your question with output of mdadm -D /dev/md127 after assembling the array (both suggested ways actually).

  • 1
    mdadm -A --force -R /dev/md127 /dev/sd[bce] did the trick! Managed to mount it and am now doing one big rsync onto a hastily purchased 4TB USB drive! Thank you! – Rog Jun 8 '15 at 17:43
  • Sorry to come back to this again, but it was working fine then started generating read errors. It now thinks that there is no superblock on sde! It was managing before, is there any way of forcing it past this?? – Rog Jun 10 '15 at 22:04
  • root@tower:/home/jeremy# mdadm --assemble --force /dev/md127 /dev/sd[bce] mdadm: no RAID superblock on /dev/sde mdadm: /dev/sde has no superblock - assembly aborted – Rog Jun 10 '15 at 22:06
  • @Rog I'm sorry to hear that. Try to see what's wrong with the drive. But missing superblock really sounds like it has been wiped by something, or that the drive is completely dead. With RAID10 and NEAR layout you may be lucky enough and get it to work with just two drives. But if you are not, I'd probably go for a professional recovery, if the data is of value. It will be costly, but you may get some data. If you start messing around, you may destroy whats left of your data. – Fox Jun 10 '15 at 22:32
  • @Rog by the way this is second time something weird happened to your array (first there were two devices #3, now you have superblock gone missing). This really smells fishy and I suspect there is something, or someone, else doing the damage, rather than failed drives. You really should doublecheck everything before breaking it even more. – Fox Jun 10 '15 at 22:34
0

It seems like you've lost a whole subarray of your Raid10 (Drives sdd, sde were a mirror before?!) - if this would be the case, your data is lost.

  • I'm not sure - I added the 4 drives into a raid10 array, originally with openmediavault (which used to be installed) – Rog Jun 7 '15 at 20:38

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.