Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

one of our huge production servers crashed this morning due to a HDD failure on port 0 (the bootable one).

That server was running CentOS 6.5 in RAID 1 with mdadm. Although, a trainee changed the HDD (which is good) and tried to rebuild the RAID array, but failed miserably. It looks like he deleted the RAID configuration.

Now, I'm in rescue mode (the datacenter netboot).

Here is the output of fdisk -l: root@rescue:~# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sda doesn't contain a valid partition table

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000a5c6d

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        4096    41947135    20971520   fd  Linux raid autodetect
/dev/sdb2        41947136  1952468991   955260928   fd  Linux raid autodetect
/dev/sdb3      1952468992  1953519615      525312   82  Linux swap / Solaris

Disk /dev/md2: 978.2 GB, 978187124736 bytes
2 heads, 4 sectors/track, 238815216 cylinders, total 1910521728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md1: 21.5 GB, 21474770944 bytes
2 heads, 4 sectors/track, 5242864 cylinders, total 41942912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table
root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md1 : active raid1 sdb1[1]
      20971456 blocks [2/1] [_U]

md2 : active raid1 sdb2[1]
      955260864 blocks [2/1] [_U]

unused devices: <none>

sda is the changed disk, while sdb is the RAIDed disk.

Here is the output of cat /proc/mdstat

root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md1 : active raid1 sdb1[1]
      20971456 blocks [2/1] [_U]

md2 : active raid1 sdb2[1]
      955260864 blocks [2/1] [_U]

unused devices: <none>

How can I rebuild the RAID? Like, recopy the content of sdb to sda and keep the RAID working afterwards?

share|improve this question
1  
By the way, why aren't you RAIDing swap as well? If you'd had RAIDed swap, the server would very likely not have crashed when it lost sda; if half the swap space suddenly vanished, I can't blame it for getting a little tetchy! –  MadHatter Mar 24 at 15:15

1 Answer 1

up vote 1 down vote accepted

First you have to create a similar partition table on /dev/sda than /dev/sdb has using fdisk.

Then, you issue:

mdadm --manage /dev/md1 --fail /dev/sda1
mdadm --manage /dev/md2 --fail /dev/sda2

commands to mark the /dev/sdaX mirrors as failed. The MD layer might already regard these as failed, so these could give error messages.

Then, remove the sdaX from the RAID:

mdadm --manage /dev/md1 --remove /dev/sda1
mdadm --manage /dev/md2 --remove /dev/sda2

Again, MD might consider those as removed already, so there could be an error message.

Finally, add the replaced hard disk to the arrays:

mdadm --manage /dev/md1 --add /dev/sda1
mdadm --manage /dev/md2 --add /dev/sda2

After this you should see from /proc/mdstat that the system is rebuilding the array.

share|improve this answer
    
I don't think the --fail and --remove steps are needed, as the /proc/mdstat output already shows no sda devices left in the MDs. But apart from that, +1 from me - this is what I'd do (and have often done, and it works). Don't forget to tag the partitions on sda as being type fd (RAID autodetect). –  MadHatter Mar 24 at 15:12
    
I think so too, but I just included those to cover all bases. And yes, type fd is is important. –  Tero Kilkanen Mar 24 at 15:15
    
@TeroKilkanen I did everything you showed. It was syncronizing... Then I booted and the grub was weird... Here is the output of cat /proc/mdstat: i.imgur.com/hTx8PBe.png and here is the output of fdisk -l: i.imgur.com/UKNC8YU.png –  Felix Lebel Mar 24 at 19:26
    
The output of those looks just fine. How was the grub weird? You might want to install grub to the /dev/sda drive too. –  Tero Kilkanen Mar 25 at 1:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.