Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

One of my four Linux software raid arrays drops one of its two devices when I reboot my system. The other three arrays work fine. I am running RAID1 on kernel version 2.6.32-5-amd64. Every time I reboot, /dev/md2 comes up with only one device. I can manually add the device by saying $ sudo mdadm /dev/md2 --add /dev/sdc1. This works fine, and mdadm confirms that the device has been re-added as follows:

mdadm: re-added /dev/sdc1

After adding the device and and allowing the array time to resynch, this is what the output of $ cat /proc/mdstat looks like:

Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdc1[0] sdd1[1]
      732574464 blocks [2/2] [UU]

md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]

unused devices: <none>

Then after I reboot, this is what the output of $ cat /proc/mdstat looks like:

Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdd1[1]
      732574464 blocks [2/1] [_U]

md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]

unused devices: <none>

During reboot, here is the output of $ sudo cat /var/log/syslog | grep mdadm :

Jun 22 19:00:08 rook mdadm[1709]: RebuildFinished event detected on md device /dev/md2
Jun 22 19:00:08 rook mdadm[1709]: SpareActive event detected on md device /dev/md2, component device /dev/sdc1
Jun 22 19:00:20 rook kernel: [ 7819.446412] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446415] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446782] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446785] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515844] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515847] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606829] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606832] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855616] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855620] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855950] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855952] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962169] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962171] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054365] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054368] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588662] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588664] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601990] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601991] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602693] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602695] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605981] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605983] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606138] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606139] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:48 rook mdadm[1737]: DegradedArray event detected on md device /dev/md2

Here is the mdadm.conf file:

ARRAY /dev/md0 metadata=0.90 UUID=92121d42:37f46b82:926983e9:7d8aad9b
ARRAY /dev/md1 metadata=0.90 UUID=9c1bafc3:1762d51d:c1ae3c29:66348110
ARRAY /dev/md2 metadata=0.90 UUID=98cea6ca:25b5f305:49e8ec88:e84bc7f0
ARRAY /dev/md3 metadata=1.2 name=rook:3 UUID=ca3fce37:95d49a09:badd0ddc:b63a4792

I also ran $ sudo smartctl -t long /dev/sdc and no hardware issues were detected. As long as I do not reboot, /dev/md2 seems to work fine. Does anyone have any suggestions?

Here is the output of $ sudo mdadm -E /dev/sdc1 after re-adding the device and letting it resync:

/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 5fd6cc13 - correct
         Events : 180998


      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1

Here is the output of $ sudo mdadm -D /dev/md2 after re-adding the device and letting it resync:

/dev/md2:
        Version : 0.90
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
     Array Size : 732574464 (698.64 GiB 750.16 GB)
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
         Events : 0.180998

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
share|improve this question
    
What is the output of mdadm -E /dev/sdc1 and mdadm -D /dev/md2? –  Andrew Jun 24 '13 at 23:55
    
Andrew - I added the output of mdadm -E /dev/sdc1 and mdadm -D /dev/md2 at the end of the original post. –  user1389890 Jun 25 '13 at 2:29
2  
I wonder if there is a metadata discrepancy on /dev/sdc1 somewhere. Have you considered wiping /dev/sdc1 out completely (dd if=/dev/zero of=/dev/sdc1 when it is not in the RAID). –  chutz Jun 26 '13 at 15:11
1  
I am considering removing /dev/sdc1 from the array and zeroing the superblock by doing # mdadm --zero-superblock /dev/sdc1 and then adding it back into the array. This would be a last resort. As far as I can tell, the output of # mdadm -E /dev/sdc1 looks fine. –  user1389890 Jun 26 '13 at 18:58
add comment

2 Answers

Do you have partition type 'fd' set on /dev/sdc1? It should be set as autodetect for boot time, but you can still manually add it with partition type 83.

share|improve this answer
add comment

try to check disk with smartctl like

short test

smartctl --test=short /dev/your_disk

then check results

smartctl -a /dev/your_disk

And long one (takes long time)

smartctl --test=long /dev/your_disk

Also looks like sdc was not connected after reboot:

md2 : active raid1 sdc1[0] sdd1[1]
      732574464 blocks [2/2] [UU]


md2 : active raid1 sdd1[1]
      732574464 blocks [2/1] [_U]

Something wrong with hardware (port, etc)?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.