Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Machine: linux centos 5.4 with 2 hdds and raid 5 (yes, 3rd disk missing).

Situation:

  1. all running fine (3rd disk missing)
  2. then a power off (system shut down itself when battery power ended).
  3. machine not coming back

message on screen:

Memory for crash kernel (0x0 to 0x0) notwithin permissible range
PCI: BIOS Bug: MCFC area at e0000000 is not E820-reserved
PCI: Not using MMCONFIG.
Red Hat nash version 5.1.19.6 starting
insmod: error inserting '/lib/raid456.ko': -1 File exists
md: md2: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md2
raid5: failed to run raid set md2
md: pers->run() failed ...
md: md2: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md2
raid5: failed to run raid set md2
md: pers->run() failed ...
EXT3-fs: unable to read superblock
mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
setuproot: moving /dev failed: No such file or directory00
setuproot: error mounting /proc: No such file or directory
setuproot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

so i installed sysresccd on memory stick and booted with it. then i run these tests:

smartctl -t short /dev/sda 
smartctl -X /dev/sda 
smartctl -l selftest /dev/sda 

and same with sdb. results are:

sda: test=Short offline, status="Completed without error", remaining=00%, lifetime=19230, firsterror=- 
sdb: test=Short offline, status="Completed: read failure", remaining=90%, lifetime=19256, firsterror=67031516 

and the detail of sdb:

root@sysresccd /root % smartctl -A /dev/sdb
smartctl 5.42 2011-10-20 r3458 [i686-linux-3.0.21-std250-i586] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   180   021    Pre-fail  Always       -       5975
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19256
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       32
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       27
193 Load_Cycle_Count        0x0032   183   183   000    Old_age   Always       -       51128
194 Temperature_Celsius     0x0022   111   093   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       17
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

the Current_Pending_Sector 17 might be a problem.

then the further steps: 1. bought 3x 2tb hdds 2. booted with memory stick 3. copied the 2 old 1.5tb disks to 2 of the new ones one after another: dd if=/dev/sda of=dev/sdc bs=32M dd if=/dev/sdb of=dev/sdc bs=32M 4. removed the 2 old disks (to not make matters worse) 5. attached the 3 new disks. rebooted.

the output then goes like this:

Memory for crash kernel (0x0 to 0x0) notwithin permissible range
PCI: BIOS Bug: MCFC area at e0000000 is not E820-reserved
PCI: Not using MMCONFIG.
Red Hat nash version 5.1.19.6 starting
insmod: error inserting '/lib/raid456.ko': -1 File exists
md: invalid raid superblock magic on sdb3
md: md2: raid array is not clean -- starting background reconstruction
raid5: not enough operational devices for md2 (2/3 failed)
raid5: failed to run raid set md2
md: pers->run() failed ...
md: md2: raid array is not clean -- starting background reconstruction
raid5: not enough operational devices for md2 (2/3 failed)
raid5: failed to run raid set md2
md: pers->run() failed ...
EXT3-fs: unable to read superblock
mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
setuproot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

so i booted with the new disks and sysresccd from memory stick. here is some information:

fdisk -l
shows the two full disks exactly like the output was on the old disks

Device  Boot    Start   End Blocks  Id  System
/dev/sda1   *   63  610469  305203+ fd  Linux raid autodetect
/dev/sda2       610470  8803619 4096575 fd  Linux raid autodetect
/dev/sda3       8803620 2930272064  1460734222+ fd  Linux raid autodetect

/dev/sdb1   *   63  610469  305203+ fd  Linux raid autodetect
/dev/sdb2       610470  8803619 4096575 fd  Linux raid autodetect
/dev/sdb3       8803620 2930272064  1460734222+ fd  Linux raid autodetect

sdc doens't contain a valid partition table (that's the empty 3rd disk)

smartctl -t short /dev/sda
smartctl -X /dev/sda
smartctl -l selftest /dev/sda
sda: test=Short offline, status="Completed without error", remaining=00%, lifetime=19230, firsterror=-
sdb: test=Short offline, status="Completed: read failure", remaining=90%, lifetime=19256, firsterror=67031516

smartctl -A /dev/sdb
offline_uncorrectable: 0

then:

root@sysresccd /root % cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md125 : inactive sda3[0](S)
      1460734144 blocks

md126 : active raid1 sda1[0] sdb1[1]
      305088 blocks [2/2] [UU]

md127 : active raid1 sda2[0] sdb2[1]
      4096448 blocks [2/2] [UU]

unused devices: <none>

note: the raid5 is shown as md125 there.

detail for 127:

root@sysresccd /root % mdadm --detail /dev/md127
/dev/md127:
        Version : 0.90
  Creation Time : Sun Dec 13 18:45:15 2009
     Raid Level : raid1
     Array Size : 4096448 (3.91 GiB 4.19 GB)
  Used Dev Size : 4096448 (3.91 GiB 4.19 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar  8 00:40:45 2012
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 939f1a92:590d4172:2414ef47:5e2b15cb
         Events : 0.236

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2

for 126:

root@sysresccd /root % mdadm --detail /dev/md126
/dev/md126:
        Version : 0.90
  Creation Time : Sun Dec 13 19:21:09 2009
     Raid Level : raid1
     Array Size : 305088 (297.99 MiB 312.41 MB)
  Used Dev Size : 305088 (297.99 MiB 312.41 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 126
    Persistence : Superblock is persistent

    Update Time : Wed Mar  7 23:34:02 2012
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : bde56644:86d3e3a4:1128f4fe:0f47f21f
         Events : 0.242

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

detail for 125:

root@sysresccd /root % mdadm --detail /dev/md125
mdadm: md device /dev/md125 does not appear to be active.

sda3:

root@sysresccd /root % mdadm --examine /dev/sda3
/dev/sda3:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 062f3190:b9337fc1:0b38f5df:7ec7c53b
  Creation Time : Sun Dec 13 18:45:15 2009
     Raid Level : raid5
  Used Dev Size : 1460733952 (1393.06 GiB 1495.79 GB)
     Array Size : 2921467904 (2786.13 GiB 2991.58 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 2

    Update Time : Sat Mar  3 22:48:34 2012
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e5ac0d6c - correct
         Events : 26243911

         Layout : left-symmetric
     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     0       8        3        0      active sync   /dev/sda3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       0        0        2      faulty removed

sdb3: root@sysresccd /root % mdadm --examine /dev/sdb3 mdadm: No md superblock detected on /dev/sdb3.

then:

root@sysresccd /root % mdadm --examine /dev/sd[ab]3 | egrep 'dev|Update|Role|State|Chunk Size'
mdadm: No md superblock detected on /dev/sdb3.
/dev/sda3:
    Update Time : Sat Mar  3 22:48:34 2012
          State : active
     Chunk Size : 256K
      Number   Major   Minor   RaidDevice State
this     0       8        3        0      active sync   /dev/sda3
   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3

more:

root@sysresccd /root % mdadm --verbose --examine --scan
ARRAY /dev/md2 level=raid5 num-devices=3 UUID=062f3190:b9337fc1:0b38f5df:7ec7c53b
   devices=/dev/sda3
ARRAY /dev/md126 level=raid1 num-devices=2 UUID=bde56644:86d3e3a4:1128f4fe:0f47f21f
   devices=/dev/sdb1,/dev/sda1
ARRAY /dev/md127 level=raid1 num-devices=2 UUID=939f1a92:590d4172:2414ef47:5e2b15cb
   devices=/dev/sdb2,/dev/sda2

(note: here it's listed as md125 and not md2)

root@sysresccd /root % mdadm --verbose --create --assume-clean /dev/md2 --level=5 --raid-devices=3 /dev/sda3 /dev/sdb3 missing
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: super1.x cannot open /dev/sda3: Device or resource busy
mdadm: failed container membership check
mdadm: device /dev/sda3 not suitable for any style of array

UPDATE: it could be that the dd copy of disk sdb did not succeed. the copy of sdb looked suspicious, and so i run this:

root@sysresccd /root % dd if=/dev/sda3 of=/dev/sdc3 bs=128M
11144+1 records in
11144+1 records out
1495791843840 bytes (1.5 TB) copied, 42354.9 s, 35.3 MB/s
root@sysresccd /root % dd if=/dev/sdb3 of=/dev/sdd3 bs=128M
dd: reading `/dev/sdb3': Input/output error
222+1 records in
222+1 records out
29813932032 bytes (30 GB) copied, 676.459 s, 44.1 MB/s
root@sysresccd /root %

this time only copying the sdb3 partition since sdb1 and sdb2 were fine. as you can see it aborts. thus i am now running:

ddrescue -S -c 20480 -f /dev/sdb3 /dev/sdd3 /tmp/log3

to copy it again, this time with ddrescue. it will take many more hours, so far there were errsize=17928 kB and errors=3.

i'll update this post when the copy finishes and i find out more.

share|improve this question
    
What are the model numbers of the disks? Some disks aren't really RAID suitable (WD Blues and WD Greens to name a pair.) –  Tom O'Connor Mar 7 '12 at 12:41
    
the old ones are wd caviar greens (was working fine for 2 years) –  gonfi Mar 7 '12 at 12:49
    
Do they pass SMART (short and extended) tests? –  Tom O'Connor Mar 7 '12 at 12:53
    
new disks: seagate barracuda green (adv. format), device model ST2000DL003-9VT166, has smart capability –  gonfi Mar 7 '12 at 13:04
    
old disks: wd caviar gren, device model = WDC WD5EADS-00P8B0, has smart capability. will check smart tests now. –  gonfi Mar 7 '12 at 13:05
show 9 more comments

1 Answer

(answering myself)

ddrescue solved the problem, afterwards it was possible to re-assemble the raid5 array.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.