I have a Centos 5.4 server in production with 2 drives in software RAID1.
The last days /var/log/messages has a lot of messages, showing that one of the drives is ready to fail:
Sep 23 00:48:38 milkyway kernel: SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB)
Sep 23 00:48:39 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:48:39 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:48:39 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:48:40 milkyway kernel: res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:48:40 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:48:40 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:48:40 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:48:40 milkyway kernel: ata1: EH complete
Sep 23 00:48:41 milkyway kernel: sda: Write Protect is off
Sep 23 00:48:41 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:48:58 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:00 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:03 milkyway kernel: res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:03 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:04 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:04 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:04 milkyway kernel: ata1: EH complete
Sep 23 00:49:04 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:49:04 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:04 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:04 milkyway kernel: res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:04 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:04 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:04 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:05 milkyway kernel: ata1: EH complete
Sep 23 00:49:05 milkyway kernel: SCSI device sda: drive cache: write back
Sep 23 00:49:06 milkyway kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 23 00:49:06 milkyway kernel: ata1.00: irq_stat 0x40000001
Sep 23 00:49:06 milkyway kernel: ata1.00: cmd 25/00:10:31:21:8c/00:00:28:00:00/e0 tag 0 dma 8192 in
Sep 23 00:49:06 milkyway kernel: res 51/40:00:35:21:8c/00:00:28:00:00/e0 Emask 0x9 (media error)
Sep 23 00:49:06 milkyway kernel: ata1.00: status: { DRDY ERR }
Sep 23 00:49:06 milkyway kernel: ata1.00: error: { UNC }
Sep 23 00:49:06 milkyway kernel: ata1.00: configured for UDMA/133
Sep 23 00:49:08 milkyway kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
However in /proc/mdstat none of the hard drives is showing as degraded:
Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb1[1] sda1[0]
4200896 blocks [2/2] [UU]
md1 : active raid1 sdb2[1] sda2[0]
2104448 blocks [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]
726266432 blocks [2/2] [UU]
unused devices: <none>
I have started migrating all data to a new server. But the result is that right now this is pretty slow, due to the failing hard disk, making almost impossible to transfer them all. Furthermore due to the harddisk bottleneck the load sky rockets, resulting in making the server unusable.
Is it possible to remove the failing drive without losing the data and without any downtime? I don't mind even if the RAID1 remains with 1 drive temporarily, in order to complete the transfer as soon as possible without delays.