First of all, it needs to be understood that this is not about changing a failed disk in a RAID-1 array but on how to completely remove the RAID array, clone the disk which is about to fail and then boot the system from the good disk without any RAID configuration.
The reason of this approach is because the first disk (sda), which is the good disk, is completely empty, no partitions and file systems and when I add it to the array it fails to sync (rebuilt) probably because there are read errors on the second disk (sdb). Moreover, the second disk (the bad one) is from where the system boots.
The steps needed to complete this task are fairly simple if you know linux and its commands, however, the procedure involves partitioning, cloning and MBR installation tasks that are going to destroy your data if you are not careful enough. Finally, the following procedure is specific for the disks' configuration described in the question but it should work in other systems if you carefully substitute the necessary devices and partition names.
Here is the procedure:
1. Boot the system in rescue mode.
Since we are going to clone data from the system's boot disk we don't need to worry about any locked files or things like that. The best way to do this is to boot in to rescue mode. Fortunately, my server hosting company supports that in a very easy way. In a server that you have direct access to (no remote), it would be something like booting from a live CD or selecting "Rescue mode" from your boot menu.
2. Prepare the first disk (/dev/sda) for cloning the data.
In order to prepare sda we need to get partitioning information from sdb. Actually this step isn't mandatory because our aim is to boot the system without any RAID configuration or without any relation with sdb, if you prefer, so if the current partition scheme isn't what we want this is the opportunity to change it. The only thing that we need to have in mind is that the new partitions need to have enough space to hold the data from sdb. However, partitioning in linux requires some deep knowledge about aligning and stuff like that and since sdb is already properly partitioned I found it easier to do the same in sda.
To do that we first need to view the partitions in sdb, I used parted for that.
Note: The most dificult task when you are researching into how to do things with linux is to find which command (of the many) is best to use. For example, there are quite a bit partitioning commands and utilities and deciding which one to use requires a significant amount of time reading and comparing information. I chose to use parted here because it shows the file system along with the partitions and because it has better support for large disks? (I am not sure).
# parted /dev/sdb
(parted) unit s
(parted) print
output:
Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdb: 5860533168s
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Number Start End Size Type File system Flags
1 2048s 25167872s 25165825s primary
2 25169920s 26218496s 1048577s primary ext3 raid
3 26220544s 2173704192s 2147483649s primary ext4 raid
4 2173706240s 5860533167s 3686826928s extended lba
5 2173708288s 5860533167s 3686824880s logical ext4 raid
(parted) quit
As we can see here, sdb has a partition table of msdos type, that means that we can create up to 4 primary partitions on the disk. In this case it has 3 primary partitions (1,2,3) and one extended (4) which holds a logical partition (5).
Next step is to create the same partitions on sda:
# parted /dev/sda
(parted) unit s
(parted) mkpart primary 2048 25167872
(parted) mkpart primary 25169920 26218496
(parted) mkpart primary 26220544 2173704192
(parted) mkpart extended 2173706240 5860533167
(parted) mkpart logical 2173708288 5860533167
(parted) quit
As you can see I used the same start and end sectors from sdb, of course the disks are the same otherwise you need to arrange the sectors accordingly.
Now we need to tell the system about the partition changes in sda:
# partprobe /dev/sda
Finally, we need to create the file systems on sda. Probably this step isn't required at all because we are going to clone the partitions from sdb so the cloning procedure will also copy the file system information however it won't hurt.
# mkfs -t ext3 /dev/sda2
# mkfs -t ext4 /dev/sda3
# mkfs -t ext4 /dev/sda5
Notice that we didn't create a file system in partition 4 (sda4). This is because it is an extended partition which holds a logical partition (sda5), we only have to create the file system on the logical one.
Now we have sda partitioned and ready to hold our data.
3. Copying the data from sdb.
This step required the most research mainly because there are many ways to copy data. Probably cp -a would be sufficient, however, I didn't manage to find any solid information on how to handle any hidden files, links e.t.c. in order to boot the system correctly. So I decided to go with a byte to byte copying utility (cloning) instead of a file copy command. The utility I used was dd:
# dd if=/dev/md1 of=/dev/sda2 bs=512 conv=noerror,sync,notrunc
# dd if=/dev/md2 of=/dev/sda3 bs=512 conv=noerror,sync,notrunc
# dd if=/dev/md3 of=/dev/sda5 bs=512 conv=noerror,sync,notrunc
Notes: First of all notice that we are copying from /dev/md* and not from /dev/sdb*. This is because our sdb disk is actually part of a RAID-1 array. That was another part that I didn't manage to find any solid information for. For example, the system 'sees' /dev/sdb2, /dev/sdb3 and /dev/sdb5 which are the partitions that hold our data but according to /etc/fstab file it mounts /dev/md1, /dev/md2 and /dev/md3 so I assumed that it is better to copy from /dev/md*. Another thing that you need to notice is where on sda you are going to copy the data. In other words, where on sda /dev/md1,2,3 needs to be copied. Well, in this case it's easy to find that out based on the partition sizes and file systems, however in a different system df -k can show this but not from inside rescue mode; you need to boot normally for this to work.
Finally, with the above commands we instruct dd to clone each partition separately byte by byte without stopping in any read errors (noerror parameter). This of course may lead to an unbootable system if there is any read data error on sdb (which there is in this case) however, before I end up using dd I did use some techniques to find which files were affected and if it was safe to proceed with the cloning. This task is beyond the scope of this answer but a good place to start is here.
Another significant point of attention is the block size parameter (bs). According to dd's documentation you should set the bs to 512 bytes because if there is a read error, and it will, it will only "destroy" 512 bytes in the target disk instead of a bigger area however the process will be slower. Again, I didn't manage to find any solid information on this and probably a block size of 4096 byte would have the same results.
Finally, dd doesn't produce any output during its operation and it will take quite some time to finish due to the size of the partitions. If you want to see where it is, you should open a new console session (in my case a new ssh remote session since I am doing that remotely) and issue the command:
# kill -USR1 $(pidof dd)
This will force dd to print its current progress in the first console session.
4. Make sda bootable.
This is pretty straightforward to do. First of all you need to mount the necessary mount points on sda:
# mount -t ext4 /dev/sda3 /mnt
# mount -t proc proc /mnt/proc
# mount -t sysfs sys /mnt/sys
# mount -o bind /dev /mnt/dev
# mount -t ext3 /dev/sda2 /mnt/boot
Note: the last command is not required if you do not have a separate partition for /boot in your system.
Next you need to become a root on /dev/sda disk, right now you are root in the rescue system.
# chroot /mnt /bin/bash
Next we need to update our etc/mtab file. I don't really know why we should do that, I found it in another rescue tutorial without an explanation.
# grep -v rootfs /proc/mounts > /etc/mtab
Next we need to install GRUB in sda.
# grub-install --recheck /dev/sda
output:
Probing devices to guess BIOS drives. This may take a long time.
Installation finished. No error reported.
Right now we have sda ready to boot, however there are a few more steps in order to let the system know where to find the necessary file after it boots from sda.
First thing is to edit /boot/grub/grub.conf file which is the file that GRUB will read in order to find out how it will proceed. In my case I had to replace every "root (hd0,0)" instance with "root (hd0,1)" and every "/dev/md2" instance with "/dev/sda3".
grub.conf can be very confusing especially for someone who hasn't used it before. The most confusing part is the root parameters, in grub there are two different meanings of the word root. When it appears as "root (hd0,1)" it tells grub where its root directory is in order to find the required linux kernels and instructions on how to boot them. Many people confuse it with the root "/" directory on the system when it is fully booted but what grub needs here is where the /boot directory is. In this case, I have a separate partition (/dev/sda2) which holds the /boot directory and because /dev/sda2 is the second partion of the first disk I need to tell grub exactly that but starting with zero (the first partition is 0, the second is 1 and so on), so the command "root (hd0,1)" tells grub that it will find the files it needs on the second partition of the first disk.
When root appears as "/dev/" it tells to the kernel where the root of the file system will be, that is the "/" when the system is fully booted. So in this case we are entering the partition name (as linux sees it) where the "/" is, in my case /dev/sda3.
Finally we need to edit /etc/fstab file in order to tell the system what to mount and where when it starts.
In my case I had to replace all the "/dev/md*" entries with the coressponding "/dev/sda*" partitions which were:
/dev/md1 --> /dev/sda2
/dev/md2 --> /dev/sda3
/dev/md3 --> /dev/sda5
5. Verify and disable boot from sdb.
If you are compulsive like me then you may want to run
# file -s /dev/sda
to see if grub is installed on sda. If you get an output like:
/dev/sda: sticky x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 address 0x2000, stage2 segment 0x200, GRUB version 0.94;
then grub (GRand Unified Bootloader) is installed on sda.
Although it isn't necessary you may want to remove grub from sdd. I did that because if something with the above configuration is wrong then the system will boot from sdb and that may be confusing. In order to delete grub from sdb I used dd:
# dd if=/dev/zero of=/dev/sdb bs=446 count=1
That will write zero to first 446 bytes of the disk which is where grub resides in disks with msdos partition tables.
$ file -s /dev/sdb
That will check that grub has been removed.
6. Exit from chroot and unmount everything.
# exit
# umount /mnt/boot
# umount /mnt/dev
# umount /mnt/sys
# umount /mnt/proc
# umount /mnt
If any error appears during the execution of the commands above, it means that something isn't finished or went wrong on our chroot session and we probably need to start over.
7. Reboot.
Now cross your fingers and reboot
# reboot
If you manage to login to the server with your original root credentials and everything seems to work OK (databases, websites, e.t.c.) then you have successfully boot from sda.
8. Prepare sdb for replacement.
In my case I wanted to erase all the data from sdb before ask support to replace it. Furthermore, I needed to completely remove the RAID configuration from the system in order to build a new one from scratch when the new disk is in place. I started by running parted to delete the partitions:
# parted /dev/sdb
# (parted) rm 5
# (parted) rm 4
# (parted) rm 2
# (parted) rm 2
# (parted) rm 1
# (parted) quit
Then I noticed that every RAID configuration was removed as well, so I assumed that the RAID definition was inside the partitions so I didn't proceed to any further action for this. Then I secured erased sdb. There are many ways to do it. I chose the /dev/urandom method from here
