We're finally trying to start doing things right by getting proper servers. We've got a HP server (With B110i but it's not being used as RAID controller) and it's running a CentOS 5 derivative with a few software RAID1 partitions setup.
My question is, now that we've setup the RAID, how should we maintain it? What do we do when a hard drive fails? What do we need to do to replace it OS wise? What's the best way to test it? Basically, how do we recover from typical failure scenarios? We don't have experience in managing a linux software RAID and whilst there's plenty of information on configuring them I haven't found general information about how to recover from an incident and we want to write scripts and documentation so when it does happen we know what to do.
Can any provide some information on the tools available and how to use them?