-2

I am completely new to Linux. I have inherited a dedicated server hosted somewhere in Germany. The OS is CentOS release 6.7 (Final) 64bit. The server has two 3TB disks in software RAID-1 configuration. The situation I am dealing with is that, according to SMART, the second disk is about to fail, however the previous admin had some problems with the RAID and he completely remove /dev/sda components from it so now the system boots from a single disk (degraded) RAID-1 array with all the files be on /dev/sdb (the disk which is about to fail). Moreover he completely erased /dev/sda. In order to solve the problem I need to perform the following tasks:

  1. Re-partition /dev/sda to be the same with /dev/sdb.
  2. Copy (clone) all the data from /dev/sdb to /dev/sda.
  3. Completely remove the RAID-1 configuration.
  4. Configure the system to boot from /dev/sda as a native disk (not /dev/md*).
  5. Make sure that everything is working OK.
  6. Secure erase /dev/sdb.
  7. Ask the support to replace the failing disk (/dev/sdb) with a new one.
  8. Re-Create and sync the RAID-1 array.

The problem is that I don't know how to perform these tasks in Linux. Please note that I am accessing the system remotely so I cannot afford any mistake that render the system un-bootable. However I do have access to a rescue system (a small linux bootable from DHCP with access to the underlying system).

Here are some (maybe useful) commands outputs and config files:

fdisk -l

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000e76a6

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1567    12582912+  83  Linux
/dev/sdb2            1567        1633      524288+  fd  Linux raid autodetect
/dev/sdb3            1633      135307  1073741824+  fd  Linux raid autodetect
/dev/sdb4          135307      364802  1843413464    f  W95 Ext'd (LBA)
/dev/sdb5          135308      364802  1843412440   fd  Linux raid autodetect

Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00074207

Device Boot      Start         End      Blocks   Id  System

Disk /dev/md3: 1887.7 GB, 1887654199296 bytes
2 heads, 4 sectors/track, 460853076 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1: 536 MB, 536858624 bytes
2 heads, 4 sectors/track, 131069 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md2: 1099.5 GB, 1099511488512 bytes
2 heads, 4 sectors/track, 268435422 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

file -s /dev/sdb

/dev/sdb: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3,     stage2 address 0x2000, stage2 segment 0x200, GRUB version 0.94; partition 1: ID=0x83, starthead 32, startsector 2048, 25165825 sectors; partition 2: ID=0xfd, starthead 254, startsector 25169920, 1048577 sectors; partition 3: ID=0xfd, starthead 254, startsector 26220544, 2147483649 sectors; partition 4: ID=0xf, starthead 254, startsector 2173706240, 3686826928 sectors, code offset 0x48

file -s /dev/sda

/dev/sda: x86 boot sector, code offset 0xb8

cat /etc/fstab

proc /proc proc defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0
/dev/md0 none swap sw 0 0
/dev/md1 /boot ext3 defaults 0 0
/dev/md2 / ext4 defaults 0 0
/dev/md3 /home ext4 defaults 0 0

cat /boot/grub/grub.conf

timeout 5
default 0

title CentOS (2.6.32-573.7.1.el6.x86_64)
root (hd0,1)
kernel /vmlinuz-2.6.32-573.7.1.el6.x86_64 ro root=/dev/md2 rd_NO_LUKS rd_NO_DM nomodeset crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de
initrd /initramfs-2.6.32-573.7.1.el6.x86_64.img

title CentOS (2.6.32-504.1.3.el6.x86_64)
root (hd0,1)
kernel /vmlinuz-2.6.32-504.1.3.el6.x86_64 ro root=/dev/md2 rd_NO_LUKS rd_NO_DM nomodeset crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de
initrd /initramfs-2.6.32-504.1.3.el6.x86_64.img

cat /proc/mdstat

Personalities : [raid1]
md2 : active raid1 sdb3[1]
      1073741688 blocks super 1.0 [2/1] [_U]

md1 : active raid1 sdb2[1]
      524276 blocks super 1.0 [2/1] [_U]

md3 : active raid1 sdb5[1]
      1843412304 blocks super 1.0 [2/1] [_U]

unused devices: <none>

Any help on this will be highly appreciated.

Thank you.

closed as too broad by Ward, womble Oct 9 '15 at 3:33

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

1

First of all, it needs to be understood that this is not about changing a failed disk in a RAID-1 array but on how to completely remove the RAID array, clone the disk which is about to fail and then boot the system from the good disk without any RAID configuration. The reason of this approach is because the first disk (sda), which is the good disk, is completely empty, no partitions and file systems and when I add it to the array it fails to sync (rebuilt) probably because there are read errors on the second disk (sdb). Moreover, the second disk (the bad one) is from where the system boots. The steps needed to complete this task are fairly simple if you know linux and its commands, however, the procedure involves partitioning, cloning and MBR installation tasks that are going to destroy your data if you are not careful enough. Finally, the following procedure is specific for the disks' configuration described in the question but it should work in other systems if you carefully substitute the necessary devices and partition names.

Here is the procedure:

1. Boot the system in rescue mode.

Since we are going to clone data from the system's boot disk we don't need to worry about any locked files or things like that. The best way to do this is to boot in to rescue mode. Fortunately, my server hosting company supports that in a very easy way. In a server that you have direct access to (no remote), it would be something like booting from a live CD or selecting "Rescue mode" from your boot menu.

2. Prepare the first disk (/dev/sda) for cloning the data.

In order to prepare sda we need to get partitioning information from sdb. Actually this step isn't mandatory because our aim is to boot the system without any RAID configuration or without any relation with sdb, if you prefer, so if the current partition scheme isn't what we want this is the opportunity to change it. The only thing that we need to have in mind is that the new partitions need to have enough space to hold the data from sdb. However, partitioning in linux requires some deep knowledge about aligning and stuff like that and since sdb is already properly partitioned I found it easier to do the same in sda. To do that we first need to view the partitions in sdb, I used parted for that.

Note: The most dificult task when you are researching into how to do things with linux is to find which command (of the many) is best to use. For example, there are quite a bit partitioning commands and utilities and deciding which one to use requires a significant amount of time reading and comparing information. I chose to use parted here because it shows the file system along with the partitions and because it has better support for large disks? (I am not sure).

# parted /dev/sdb
(parted) unit s
(parted) print

output:

Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdb: 5860533168s
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number  Start        End          Size         Type      File system  Flags
 1      2048s        25167872s    25165825s    primary
 2      25169920s    26218496s    1048577s     primary   ext3         raid
 3      26220544s    2173704192s  2147483649s  primary   ext4         raid
 4      2173706240s  5860533167s  3686826928s  extended               lba
 5      2173708288s  5860533167s  3686824880s  logical   ext4         raid

(parted) quit

As we can see here, sdb has a partition table of msdos type, that means that we can create up to 4 primary partitions on the disk. In this case it has 3 primary partitions (1,2,3) and one extended (4) which holds a logical partition (5).

Next step is to create the same partitions on sda:

# parted /dev/sda
(parted) unit s
(parted) mkpart primary 2048 25167872
(parted) mkpart primary 25169920 26218496
(parted) mkpart primary 26220544 2173704192
(parted) mkpart extended 2173706240 5860533167
(parted) mkpart logical 2173708288 5860533167
(parted) quit

As you can see I used the same start and end sectors from sdb, of course the disks are the same otherwise you need to arrange the sectors accordingly.

Now we need to tell the system about the partition changes in sda:

# partprobe /dev/sda

Finally, we need to create the file systems on sda. Probably this step isn't required at all because we are going to clone the partitions from sdb so the cloning procedure will also copy the file system information however it won't hurt.

# mkfs -t ext3 /dev/sda2
# mkfs -t ext4 /dev/sda3
# mkfs -t ext4 /dev/sda5

Notice that we didn't create a file system in partition 4 (sda4). This is because it is an extended partition which holds a logical partition (sda5), we only have to create the file system on the logical one.

Now we have sda partitioned and ready to hold our data.

3. Copying the data from sdb.

This step required the most research mainly because there are many ways to copy data. Probably cp -a would be sufficient, however, I didn't manage to find any solid information on how to handle any hidden files, links e.t.c. in order to boot the system correctly. So I decided to go with a byte to byte copying utility (cloning) instead of a file copy command. The utility I used was dd:

# dd if=/dev/md1 of=/dev/sda2 bs=512 conv=noerror,sync,notrunc
# dd if=/dev/md2 of=/dev/sda3 bs=512 conv=noerror,sync,notrunc
# dd if=/dev/md3 of=/dev/sda5 bs=512 conv=noerror,sync,notrunc

Notes: First of all notice that we are copying from /dev/md* and not from /dev/sdb*. This is because our sdb disk is actually part of a RAID-1 array. That was another part that I didn't manage to find any solid information for. For example, the system 'sees' /dev/sdb2, /dev/sdb3 and /dev/sdb5 which are the partitions that hold our data but according to /etc/fstab file it mounts /dev/md1, /dev/md2 and /dev/md3 so I assumed that it is better to copy from /dev/md*. Another thing that you need to notice is where on sda you are going to copy the data. In other words, where on sda /dev/md1,2,3 needs to be copied. Well, in this case it's easy to find that out based on the partition sizes and file systems, however in a different system df -k can show this but not from inside rescue mode; you need to boot normally for this to work.

Finally, with the above commands we instruct dd to clone each partition separately byte by byte without stopping in any read errors (noerror parameter). This of course may lead to an unbootable system if there is any read data error on sdb (which there is in this case) however, before I end up using dd I did use some techniques to find which files were affected and if it was safe to proceed with the cloning. This task is beyond the scope of this answer but a good place to start is here.

Another significant point of attention is the block size parameter (bs). According to dd's documentation you should set the bs to 512 bytes because if there is a read error, and it will, it will only "destroy" 512 bytes in the target disk instead of a bigger area however the process will be slower. Again, I didn't manage to find any solid information on this and probably a block size of 4096 byte would have the same results.

Finally, dd doesn't produce any output during its operation and it will take quite some time to finish due to the size of the partitions. If you want to see where it is, you should open a new console session (in my case a new ssh remote session since I am doing that remotely) and issue the command:

# kill -USR1 $(pidof dd)

This will force dd to print its current progress in the first console session.

4. Make sda bootable.

This is pretty straightforward to do. First of all you need to mount the necessary mount points on sda:

# mount -t ext4 /dev/sda3 /mnt
# mount -t proc proc /mnt/proc
# mount -t sysfs sys /mnt/sys
# mount -o bind /dev /mnt/dev
# mount -t ext3 /dev/sda2 /mnt/boot

Note: the last command is not required if you do not have a separate partition for /boot in your system.

Next you need to become a root on /dev/sda disk, right now you are root in the rescue system.

# chroot /mnt /bin/bash

Next we need to update our etc/mtab file. I don't really know why we should do that, I found it in another rescue tutorial without an explanation.

# grep -v rootfs /proc/mounts > /etc/mtab

Next we need to install GRUB in sda.

# grub-install --recheck /dev/sda

output:

Probing devices to guess BIOS drives. This may take a long time.
Installation finished. No error reported.

Right now we have sda ready to boot, however there are a few more steps in order to let the system know where to find the necessary file after it boots from sda.

First thing is to edit /boot/grub/grub.conf file which is the file that GRUB will read in order to find out how it will proceed. In my case I had to replace every "root (hd0,0)" instance with "root (hd0,1)" and every "/dev/md2" instance with "/dev/sda3".

grub.conf can be very confusing especially for someone who hasn't used it before. The most confusing part is the root parameters, in grub there are two different meanings of the word root. When it appears as "root (hd0,1)" it tells grub where its root directory is in order to find the required linux kernels and instructions on how to boot them. Many people confuse it with the root "/" directory on the system when it is fully booted but what grub needs here is where the /boot directory is. In this case, I have a separate partition (/dev/sda2) which holds the /boot directory and because /dev/sda2 is the second partion of the first disk I need to tell grub exactly that but starting with zero (the first partition is 0, the second is 1 and so on), so the command "root (hd0,1)" tells grub that it will find the files it needs on the second partition of the first disk.

When root appears as "/dev/" it tells to the kernel where the root of the file system will be, that is the "/" when the system is fully booted. So in this case we are entering the partition name (as linux sees it) where the "/" is, in my case /dev/sda3.

Finally we need to edit /etc/fstab file in order to tell the system what to mount and where when it starts.

In my case I had to replace all the "/dev/md*" entries with the coressponding "/dev/sda*" partitions which were:

/dev/md1 --> /dev/sda2 /dev/md2 --> /dev/sda3 /dev/md3 --> /dev/sda5

5. Verify and disable boot from sdb.

If you are compulsive like me then you may want to run

# file -s /dev/sda

to see if grub is installed on sda. If you get an output like:

/dev/sda: sticky x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 address 0x2000, stage2 segment 0x200, GRUB version 0.94;

then grub (GRand Unified Bootloader) is installed on sda.

Although it isn't necessary you may want to remove grub from sdd. I did that because if something with the above configuration is wrong then the system will boot from sdb and that may be confusing. In order to delete grub from sdb I used dd:

# dd if=/dev/zero of=/dev/sdb bs=446 count=1

That will write zero to first 446 bytes of the disk which is where grub resides in disks with msdos partition tables.

$ file -s /dev/sdb

That will check that grub has been removed.

6. Exit from chroot and unmount everything.

# exit
# umount /mnt/boot
# umount /mnt/dev
# umount /mnt/sys
# umount /mnt/proc
# umount /mnt

If any error appears during the execution of the commands above, it means that something isn't finished or went wrong on our chroot session and we probably need to start over.

7. Reboot.

Now cross your fingers and reboot

# reboot

If you manage to login to the server with your original root credentials and everything seems to work OK (databases, websites, e.t.c.) then you have successfully boot from sda.

8. Prepare sdb for replacement.

In my case I wanted to erase all the data from sdb before ask support to replace it. Furthermore, I needed to completely remove the RAID configuration from the system in order to build a new one from scratch when the new disk is in place. I started by running parted to delete the partitions:

# parted /dev/sdb
# (parted) rm 5
# (parted) rm 4
# (parted) rm 2
# (parted) rm 2
# (parted) rm 1
# (parted) quit

Then I noticed that every RAID configuration was removed as well, so I assumed that the RAID definition was inside the partitions so I didn't proceed to any further action for this. Then I secured erased sdb. There are many ways to do it. I chose the /dev/urandom method from here

0

It might be best not to attempt this, and instead start fresh or something. I would have not attempted this when I was new to Linux.

That being said, all you need to do is partition and add /dev/sda back to the raid array and once it is rebuilt, replace /dev/sdb. Make sure grub is on both disks and that the bios will try both disks when booting.

  • Thank you for your answer, actually that was my first attempt to fix the issue so I partitioned /dev/sda and add it back to the array but, unfortunately, only md1 and md2 were synced with md2 to degrade from time to time and md3 was never synced. After a lot of research (I don't know linux) I manage to find that the issue was due read errors on sdb. So I decided to go through it manually that's why I posted the question. Anyway I manage to resolve it and I will post the answer later. – ChD Computers Oct 5 '15 at 23:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.