0

My server had RAID-1 setup and a few days ago one drive failed. I replaced it and added it back to the array. The problem is that the 3TB drive had 2 partitions:

root@s01 [~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_server01-LogVol01
                      2.7T  419G  2.2T  17% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/md0              477M  149M  303M  33% /boot
/dev/md2               32G  6.5G   24G  22% /mysql
/usr/tmpDSK           4.0G  223M  3.6G   6% /tmp
root@s01 [~]#

So basically 2.7TB for /, 7.8GB for tmpfs and 477MB for boot.

So both md0 and md1 were on the same drive. The failed drive was then re-added with mdadm --manage /dev/md1 --add /dev/sdc2

The problem now as you can see is that md1 was synced up but md0 was left out:

root@s01 [~]# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 sdd1[1]
      511988 blocks super 1.0 [2/1] [_U]

md2 : active raid1 sda1[0] sdb1[1]
      33558657 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdc1[2] sdd2[1]
      2929751932 blocks super 1.1 [2/2] [UU]
      bitmap: 6/22 pages [24KB], 65536KB chunk

unused devices: <none>
root@s01 [~]#

Is there any way to safely re-add md0 without re-formatting everything?

md0 now looks like:

root@s01 [~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.0
  Creation Time : Sat Jul  7 18:23:23 2012
     Raid Level : raid1
     Array Size : 511988 (499.99 MiB 524.28 MB)
  Used Dev Size : 511988 (499.99 MiB 524.28 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Jul  7 00:59:54 2016
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : server01.domain.com:0
           UUID : 2ca3a4dc:db3ef0ca:b3d37d49:b43e826f
         Events : 726

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       49        1      active sync   /dev/sdd1
root@s01 [~]#

and md1 looks like:

root@s01 [~]# mdadm --detail /dev/md1
/dev/md1:
        Version : 1.1
  Creation Time : Sat Jul  7 18:23:24 2012
     Raid Level : raid1
     Array Size : 2929751932 (2794.03 GiB 3000.07 GB)
  Used Dev Size : 2929751932 (2794.03 GiB 3000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Jul  7 20:33:31 2016
          State : active 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : server01.domain.com:1
           UUID : 58600fc5:5348d92c:a7d25465:20d42940
         Events : 2359606

    Number   Major   Minor   RaidDevice State
       2       8       33        0      active sync   /dev/sdc1
       1       8       50        1      active sync   /dev/sdd2
root@s01 [~]#
5

So both md0 and md1 were on the same drive. The failed drive was then re-added with mdadm --manage /dev/md1 --add /dev/sdc2

The problem now as you can see is that md1 was synced up but md0 was left out:

This is because you readded missing drive only to /dev/md1, you need to do the same with /dev/md0, like mdadm --manage /dev/md0 --add /dev/sdcX

sdcX - you need to now what partition to use, we cannot know that.

Is there any way to safely re-add md0 without re-formatting everything?

You can use parted, and find out what is where and what partition is not used. Mdadm is quite safe for that, it won't allow you to add device currently in use.

I can bet that you didn't create 2nd partition on that new drive. Mdadm allowed you to readd partition with diffrent size (bigger in your case). You should create same partition layout as on /dev/sdd. Parted will help you with that as well. You can check /dev/sdd layout like this: parted /dev/sdd unit MiB print, then compare it with /dev/sdc.

Make sure that you align partitions correctly, parted will help you with that too, after you create first partition on the new disk, you should notice that it start from 2048 sector, 1st MiB. Misalignment in general is bad for disk's performance.

You'll also need, before making partitions:

mdadm --manage /dev/mdX --fail /dev/sdX - to mark disk as failed (non-destructive)

mdadm --manage /dev/mdX --remove /dev/sdX - to remove disk from array (non-destructive)

Partition the disk accordingly (destructive to some point) and readd it again (non-destructive, to the array of course not for added disk/parition).

  • initramfs probably also needs to be updated, if he expects his RAID to start up at next boot – asdmin Jul 8 '16 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.