3

I have a Centos 5.4 server in production with 2 drives in software RAID1.

The last days /var/log/messages has a lot of messages, showing that one of the drives is ready to fail:

Sep 23 00:48:38 milkyway kernel: SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB)
Sep 23 00:48:39 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:48:39 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:48:39 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:48:40 milkyway kernel:          res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:48:40 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:48:40 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:48:40 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:48:40 milkyway kernel: ata1: EH complete
Sep 23 00:48:41 milkyway kernel: sda: Write Protect is off
Sep 23 00:48:41 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:48:58 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:00 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:03 milkyway kernel:          res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:03 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:04 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:04 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:04 milkyway kernel: ata1: EH complete
Sep 23 00:49:04 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:49:04 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:04 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:04 milkyway kernel:          res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:04 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:04 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:04 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:05 milkyway kernel: ata1: EH complete
Sep 23 00:49:05 milkyway kernel: SCSI device sda: drive cache: write back
Sep 23 00:49:06 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:49:06 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:06 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:06 milkyway kernel:          res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:06 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:06 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:06 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:08 milkyway kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002

However in /proc/mdstat none of the hard drives is showing as degraded:

Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4] 
md0 : active raid1 sdb1[1] sda1[0]
      4200896 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      2104448 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[0]
      726266432 blocks [2/2] [UU]

unused devices: <none>

I have started migrating all data to a new server. But the result is that right now this is pretty slow, due to the failing hard disk, making almost impossible to transfer them all. Furthermore due to the harddisk bottleneck the load sky rockets, resulting in making the server unusable.

Is it possible to remove the failing drive without losing the data and without any downtime? I don't mind even if the RAID1 remains with 1 drive temporarily, in order to complete the transfer as soon as possible without delays.

Improve this question

1 Answer 1

Reset to default
3

You can manually mark a drive as failed via mdadm thus:

mdadm --manage /dev/md0 --fail /dev/sda1

which will then allow you to remove the drive from the array:

mdadm --manage /dev/md0 --remove /dev/sda1

Repeat for all the arrays.

This will leave the arrays with only one drive, which should hopefully allow you to backup the data on the other drive

OR

Replace the failing/failed drive with a spare and rebuild the array with it by mirroring the disk partitioning from the good drive and then add these partitions to the md devices for array rebuilding.

However, the normal mantra of "RAID is not backup" applies, i.e. it would have been prescient to have had backups of the contents of the array long before an imminent disk failure, though this isn't particularly helpful to you just now.

Improve this answer
3
  • I'll offer an alternative to the gentle approach: just unplug the failing drive. Whichever method, do it soon; you don't want for bad data from the bad drive to be what's used and copied.
    – Shane Madden
    Sep 22, 2011 at 22:37
  • 1
    Aslong as you can positively identify which drive is which... and that nothing has caused what was previously labeled sda by the OS to swap with sdb, thus inverting whatever the drives were physically labeled as (or remembered as), e.g. the move to UUID for disk identification.
    – gsreynolds
    Sep 22, 2011 at 22:51
  • Thank for your detailed response. I followed the 1st solution and everything went fine! Ps. You are absolutely right about the rule "RAID is not backup". I have offsite backups but are 1-day old so its better to try and save the data if possible and the go to the backups if everything has failed.
    – Hik
    Sep 26, 2011 at 1:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .