Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Had a power failure and now my mdadm array is having problems.

sudo mdadm -D /dev/md0 [hodge@hodge-fs ~]$ sudo mdadm -D /dev/md0 /dev/md0: Version : 0.90 Creation Time : Sun Apr 25 01:39:25 2010 Raid Level : raid5 Array Size : 8790815232 (8383.57 GiB 9001.79 GB) Used Dev Size : 1465135872 (1397.26 GiB 1500.30 GB) Raid Devices : 7 Total Devices : 7 Preferred Minor : 0 Persistence : Superblock is persistent

    Update Time : Sat Aug  7 19:10:28 2010
          State : clean, degraded, recovering
 Active Devices : 6
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 128K

 Rebuild Status : 10% complete

           UUID : 44a8f730:b9bea6ea:3a28392c:12b22235 (local to host hodge-fs)
         Events : 0.1307608

    Number   Major   Minor   RaidDevice State
       0       8       81        0      active sync   /dev/sdf1
       1       8       97        1      active sync   /dev/sdg1
       2       8      113        2      active sync   /dev/sdh1
       3       8       65        3      active sync   /dev/sde1
       4       8       49        4      active sync   /dev/sdd1
       7       8       33        5      spare rebuilding   /dev/sdc1
       6       8       16        6      active sync   /dev/sdb

sudo mount -a

[hodge@hodge-fs ~]$ sudo mount -a
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

sudo fsck.ext4 /dev/md0

[hodge@hodge-fs ~]$ sudo fsck.ext4 /dev/md0
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: Group descriptors look bad... trying backup blocks...
/dev/md0: recovering journal
fsck.ext4: unable to set superblock flags on /dev/md0

sudo dumpe2fs /dev/md0 | grep -i superblock

[hodge@hodge-fs ~]$ sudo dumpe2fs /dev/md0 | grep -i superblock
dumpe2fs 1.41.12 (17-May-2010)
  Primary superblock at 0, Group descriptors at 1-524
  Backup superblock at 32768, Group descriptors at 32769-33292
  Backup superblock at 98304, Group descriptors at 98305-98828
  Backup superblock at 163840, Group descriptors at 163841-164364
  Backup superblock at 229376, Group descriptors at 229377-229900
  Backup superblock at 294912, Group descriptors at 294913-295436
  Backup superblock at 819200, Group descriptors at 819201-819724
  Backup superblock at 884736, Group descriptors at 884737-885260
  Backup superblock at 1605632, Group descriptors at 1605633-1606156
  Backup superblock at 2654208, Group descriptors at 2654209-2654732
  Backup superblock at 4096000, Group descriptors at 4096001-4096524
  Backup superblock at 7962624, Group descriptors at 7962625-7963148
  Backup superblock at 11239424, Group descriptors at 11239425-11239948
  Backup superblock at 20480000, Group descriptors at 20480001-20480524
  Backup superblock at 23887872, Group descriptors at 23887873-23888396
  Backup superblock at 71663616, Group descriptors at 71663617-71664140
  Backup superblock at 78675968, Group descriptors at 78675969-78676492
  Backup superblock at 102400000, Group descriptors at 102400001-102400524
  Backup superblock at 214990848, Group descriptors at 214990849-214991372
  Backup superblock at 512000000, Group descriptors at 512000001-512000524
  Backup superblock at 550731776, Group descriptors at 550731777-550732300
  Backup superblock at 644972544, Group descriptors at 644972545-644973068
  Backup superblock at 1934917632, Group descriptors at 1934917633-1934918156

sudo e2fsck -b 32768 /dev/md0

[hodge@hodge-fs ~]$ sudo e2fsck -b 32768 /dev/md0
e2fsck 1.41.12 (17-May-2010)
/dev/md0: recovering journal
e2fsck: unable to set superblock flags on /dev/md0

sudo dmesg | tail

[hodge@hodge-fs ~]$ sudo dmesg | tail
EXT4-fs (md0): ext4_check_descriptors: Checksum for group 0 failed (59837!=29115)
EXT4-fs (md0): group descriptors corrupted!
EXT4-fs (md0): ext4_check_descriptors: Checksum for group 0 failed (59837!=29115)
EXT4-fs (md0): group descriptors corrupted!

Please Help!!!

share|improve this question
    
show /etc/mdadm/mdadm.conf –  bindbn Sep 19 '10 at 12:10
    
Had the exact same problem on a Raid 1 power outage. –  Forkrul Assail Nov 24 '12 at 20:02

2 Answers 2

From your description and the errors, it looks to me as though there are some serious data corruption issues. Remember, RAID protects against a very specific issue; limited disk failure. A power outage isn't protected against; that's why you use UPSes and keep backups as well as using RAID.

The one thing that looks odd to me is the inclusion of /dev/sdb instead of /dev/sdb1 in the list of RAID devices. Is that correct, or did the last character get cut off perhaps?

I would try the remaining backup superblocks, just in case.

Other than that, you might look for disk recovery software. Ideally you'll be able to take a backup image of the current state of the disks; that will reduce the chance that further changes will damage the data irreparably.

share|improve this answer
    
Yes I agree :( - I tired all the superblocks but none worked - I tried using TestDisk - it can see the partition but most of the data on it is bad. I can see my folders but can't open them. - Cry:( –  Matthew Hodgkins Aug 8 '10 at 0:39
    
Yeah, sdb instead of sdb1 looks odd, you're right. @MatthewHodgkins, how is that? –  poige Apr 15 '12 at 4:42

You RAID setup had several flaws:

  1. RAID-5 with number of disks ≥ 3—4 is rather fragile. One disk's being kicked out and your data are in trouble.
  2. Not using write-intent bitmaps is dangerous and makes item #1 only worse.
  3. Spare could be more reasonable used as primary for RAID-6 or RAID-10 instead…

(I could add small chunk size and NOT using LVM-2 as disadvantages too, but they do not strongly affects overall status, of course.)

Now — never do anything to the array (fsck, and so on) until it's fully repaired. And I would strongly recommend you not trying to recover the data by yourself. You'd better find a specialist instead (in case you value them, of course).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.