** Disclaimer, I just recently became an administrator to this system and realized that the backups are unusable. As well, the state of administration software is terrible **
The system (Ubuntu 14.04) was running a two 146GB 10k SAS drives in RAID 1 (A, and B). The enclosures are hot-swappable so the server was and is still running through this process.
- Failed drive A was replaced with drive C, flashing green status confirmed that array was rebuilding
Come back to C with solid green status (online) but drive B solid amber (offline/critical failure)
However, there are large patches of filesystem that were clearly not synced, indicated by Input/output errors, and the filesystem reverting to Read-Only
My goal is to determine the source of the drive B failure, and if it's something small such as an Unreadable Block Error, to either restart the system using drive B, or to force a rebuild of the array despite the errors. The main thing is determining how to get the array controller report the failure mode, and treat the failed drive as good.
I'm only looking to recover a few small config files to make my life easier when reinstalling.
The server is currently on in a limited state, but definitely won't boot from drive C if restarted, as portions of /bin/ were lost. Surprisingly it's still serving it's function as it is only used regularly for dhcp and ssh.