Basically I have two identical hard disks that are in perfect working order, they just don't have the vibration tolerance they need for the environment they are in. (I am making an educated guess that this is the cause). The disks are in RAID 1 configuration via md raid/linux software raid. The RAID is then partitioned with LVM.
The issue is that every few days, on one or other of the disks, 1-5 contiguous sectors will start throwing I/O errors, but hdparm --write-sector puts them back in working order without issue. The disk does not clock up reallocated or pending sectors in SMART, so I can only assume that it manages to successfully fix the original sectors.
What I would like to do, is find some way to have the raid automatically fall back to the other disk for a sector it can't read. Currently the errors end up reaching the file system level and corrupting it, sometimes very severely. It tends to snowball the longer it is left.
I am confident the disks are not on the brink of failure, as they have been operating in this scenario for just under 2 years. The issue appeared after approx 6 months.
I have tried the flag which prioritises writes to one disk (usually used on SSDs) on the more reliable disk to no avail. Is there anything else I can try? I'm prepared to try alternative filesystems such as btrfs if the built in raid is more robust. Intel onboard 'fake' raid is also available to me (H67 chipset), but I'm guessing it is poor in comparison. Replacing the disks is very infeasible, as the box is colocated in what may as well be a different country