0

My CentOS 6 server is running a software RAID 1 configuration with two 1 TB disks. Yesterday trouble started happening when the server was very irresponsive. I checked the logs and noticed a whole bunch of continuous error messages having to do with disk IO. I also noticed that the second disk /dev/sdb got 'kicked' out of the RAID 1 setup. It said it was degraded and showed [U_] instead of [UU] when I did a cat /proc/mdstat. The output looks like this:

root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] 
md1 : active raid1 sda1[0]
      20971456 blocks [2/1] [U_]

md2 : active raid1 sda2[0]
      955260864 blocks [2/1] [U_]

unused devices: <none>

After contacting my hosting company where the server is running and running smartctl to check the drives it turns out both disks need to be replaced.

At this point I can only but up in rescue mode where I can still mount my partitions (/dev/md1 and /dev/md2) and access the data. I already downloaded the most recent Plesk backups using scp.

Now please bear with me as I'm not a server administrator, but my plan is to first replace the second disk, /dev/sdb and re-add it to the RAID 1 setup. That way the new second disk will be synched with the old disk 1. After that's done I can repeat the process with disk 1 and sync the new disk 1 with the new disk 2. Is that possible?

Also what steps should I take to do this?

  1. Replace faulty disk 2 with new disk 2
  2. Partition disk 2 the same as disk 1: sfdisk -d /dev/sda | sfdisk /dev/sdb
  3. Add the new disk to the partitions /dev/md1 and /dev/md2: mdadm --manage /dev/md1 --add /dev/sdb1 and mdadm --manage /dev/md2 --add /dev/sdb2
  4. Wait for synch to complete keeping an eye on /proc/mdstat to verify
  5. Repeat above steps for disk 1

Does this look okay? And will the synching start right after point 3? Did I forget anything in these steps?

Improve this question
2
  • 1
    In the future, install and configure smartd (/etc/smartd.conf) with a single line like (tracks all disks, does a short test each day and a long test once per week, sends mail to 'root' when errors are seen): DEVICESCAN -a -S on -o on -i 190 -I 190 -s (S/../.././06|L/../../6/06) -M diminishing -m root
    – tgharold
    Sep 28, 2013 at 4:25
  • Awesome that's exactly what I needed. Both disks have been replaced last month and I am now monitoring it manually. I just edited my smartd.conf and started smartd. Thanks for the advice.
    – Ruben
    Sep 28, 2013 at 20:02

1 Answer 1

Reset to default
1

If both drives need replacing, then chances are you'll lose the first drive as the array rebuilds. I'd just save yourself the headache and restore from backups.

Improve this answer
5
  • So you're saying even if I replace the drives one by one there's still a chance it will fail? If it's really impossible I will restore from backups of course, but I'd rather not because it's a lot of data. Just re-uploading it will probably take about a day that's why I'm trying so desperately to fix it this way. :)
    – Ruben
    Aug 13, 2013 at 19:08
  • 1
    If smart is showing "your drive is bad, replace!" then yes, the stress of doing a full disk copy on a bad drive can cause problems. It can also corrupt the array if it hits a bad block.
    – Nathan C
    Aug 13, 2013 at 19:12
  • SMART actually didn't say that. It passed on both drives and the data is still accessible. Both disks are showing signs that they'r e dying though I guess because smartctl showed both disks had a number of bad sectors shown by Offline_Uncorrectable. Disk 1 showed a number of 128 and disk 2 (worse) 1208.
    – Ruben
    Aug 13, 2013 at 19:47
  • I'd say if you're willing to take the chance of the drive failing then try to recover it. Worst case is the drive fails and you have to rebuild and restore from backup anyway.
    – Nathan C
    Aug 14, 2013 at 11:24
  • What happened was that after changing the worst disk first and re adding it to the array it started to rebuild from faulty disk 1. However during the process disk 1 got read errors again, so it was unable to fully rebuild. I changed out the second disk too and restored everything from backup.
    – Ruben
    Aug 14, 2013 at 20:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .