2

I was reshaping my array from 10 disks to 11 to a degraded state (drive I want to add already has data on it, but nowhere to put it). 2 drives disconnected while it was running (power issue).

Is it still possible to recovery this array?

After power cycling them I was unable to add them to the array again:

mdadm: /dev/md0 has failed so using --add cannot work and might destroy
mdadm: data on /dev/sdX1.  You should stop the array and re-assemble it.

Since rebooting, I've tried:

--assemble, fails due to "faulty" disks

--assemble --force, fails:

md: sdl1 does not have a valid v1.2 superblock, not importing!
md: sdk1 does not have a valid v1.2 superblock, not importing!
md/raid:md0: not enough operational devices (3/11 failed)
md/raid:md0: failed to run raid set.`

I've been reading the RAID Recovery article, but so far not been successful.

mdadm --create --chunk=64 --size=1953512448 --assume-clean --level=6 --raid-devices=11 /dev/md0 /dev/sd{f,h,e,g,m,i,k,l,n,d}1 missing, fails:

mdadm: /dev/sdf1 is smaller than given size. 1953512256K < 1953512448K + metadata

for all drives. My argument to --size is "Used Dev Size / 2" from mdadm --examine /dev/sdf1. I've downgraded mdadm to each version down to v3.1.2 (when default metadata was changed to 1.2, I know I never specified it manually).

Removing --size, I can create the array, but not mount:

XFS (md0): Mounting V4 Filesystem
XFS (md0): Log inconsistent (didn't find previous header)
XFS (md0): failed to find log head
XFS (md0): log mount/recovery failed: error -5
XFS (md0): log mount failed

Info

My mdadm --detail before reshape:

/dev/md0:
        Version : 1.2
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
     Array Size : 15628099584 (14904.12 GiB 16003.17 GB)
  Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
   Raid Devices : 10
  Total Devices : 10
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jun 17 14:16:09 2015
          State : clean
 Active Devices : 10
Working Devices : 10
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : ubuntu:0
           UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
         Events : 6037532

    Number   Major   Minor   RaidDevice State
       0       8       81        0      active sync   /dev/sdf1
       9       8      177        1      active sync   /dev/sdl1
      12       8       65        2      active sync   /dev/sde1
       3       8       97        3      active sync   /dev/sdg1
       4       8      145        4      active sync   /dev/sdj1
       6       8      193        5      active sync   /dev/sdm1
       7       8      113        6      active sync   /dev/sdh1
       8       8      129        7      active sync   /dev/sdi1
      10       8      161        8      active sync   /dev/sdk1
      11       8       49        9      active sync   /dev/sdd1

And mdadm --examine after failure and reboot with all disks visible again:

Device paths have changed as there was a hotswap disk added before reshape started

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 329fc32d:e9cf2ff4:3aa6c9a0:500aa445

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : 904d0c9c - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 9
   Array State : A.AAA...AA. ('A' == active, '.' == missing)


/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e59303ea:e613013e:ef8af657:1fc6ccab

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : b3b3f659 - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : A.AAA...AA. ('A' == active, '.' == missing)


/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 6aa0f9d8:e7b0cc66:d2f2a600:ef305279

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3148373376 (3002.52 GiB 3223.93 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : 3beac20c - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAAAAAA. ('A' == active, '.' == missing)


/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 4b1d87a9:16027400:df71810f:3ce53c50

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : 91a563ea - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : A.AAA...AA. ('A' == active, '.' == missing)


/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 27c8fefa:8b2b74a2:9a456d34:d1a60c20

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:29:09 2015
       Checksum : ee4ae103 - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAAAA..AA. ('A' == active, '.' == missing)


/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : bebc3764:9e582fe8:01de9766:2d8c452b

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:29:09 2015
       Checksum : 6632686d - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAA..AA. ('A' == active, '.' == missing)


/dev/sdk1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 986d9f31:3a74b90d:7800779e:31607539

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3148373376 (3002.52 GiB 3223.93 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:24:09 2015
       Checksum : de0a23b - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 6
   Array State : AAAAAAAAAA. ('A' == active, '.' == missing)


/dev/sdl1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
  Used Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a5f4ac69:f6bbac94:60c1b790:db2c223e

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:28:58 2015
       Checksum : c9909fb9 - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 7
   Array State : AAAAAA.AAA. ('A' == active, '.' == missing)


/dev/sdm1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 938d9190:582eecf8:b9157fce:38705df2

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : d2462ecd - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : A.AAA...AA. ('A' == active, '.' == missing)


/dev/sdn1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x5
     Array UUID : 70485ad1:0f5f2362:e8f5489a:577ac908
           Name : ubuntu:0
  Creation Time : Fri Jan 27 19:20:36 2012
     Raid Level : raid6
   Raid Devices : 11

 Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
     Array Size : 17581612032 (16767.13 GiB 18003.57 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 289f68f6:f43d8a40:2203e21c:e6cff371

Internal Bitmap : 2 sectors from superblock
  Reshape pos'n : 3196923264 (3048.82 GiB 3273.65 GB)
  Delta Devices : 1 (10->11)

    Update Time : Wed Jun 17 19:46:34 2015
       Checksum : 4db49d1a - correct
         Events : 6039833

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 8
   Array State : A.AAA...AA. ('A' == active, '.' == missing)
0

I think you didn't tell here whole story, I suspect that power failure was during raid resync, if so then --auto-assemble won't work. It's time to recover from backup or use professional help.

  • This is the whole story. It was being reshaped into a degraded state (10/11 drives), then 2 drives disconnected leaving it with 8/11. Looking at md0 in a hex editor, it seems alright. I can see filenames and data that I recognise (also the XFS header starts in the correct place). Currently running xfs_repair -n to try and find a secondary superblock.. but it's going to take a few days since it's a 15TB filesystem. – user294789 Jun 18 '15 at 9:44
0

Well the good news is that all ten drives have matching Event counts at 6039833.

I think the key to success is resolving why you are seeing...

md: sdl1 does not have a valid v1.2 superblock, not importing!
md: sdk1 does not have a valid v1.2 superblock, not importing!

This is an interesting case and certainly it will take some study to figure out a solution. If you would like some help contact me via the web link in my profile.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.