My current diagnosis is that there is a hardware fault, either related to the memory of the hardware RAID controller or something related to the extra juice the hard drives requires when running on full.
Tried replacing the PSU, no dice. Tried live booting another OS, no dice. High disk IO ensures a hard shutdown, whatever the OS. I have established that a new server is definitely needed.
Now, however, I am facing the challenge of taking a back-up of the content of the drives, without causing a high IO. I have configured cgroups with a IOps limit which I have played around with to see how high it can be without causing a shutdown. The results seem to vary a bit from each disk, but an average of about 100 iops seems to work for a longer while before crashing. However this is not really feasible considering there is about 120GB of data. These are SAS 15K drives by the way.
Throttling the IO seems to work, but is a tiredsome process since I have to set it all up again when the server shuts down. I am copying the data over from the drives to an external drive using Runtime Live CD (Knoppix fork).
The server is about 7 years old, and I do not happen to have an extra connector for the drives.
When facing such a scenario, what is an reliable way of getting the data from the drives?
Just for reference, this is the link which helped me setup the cgroups limiting the IO: http://fritshoogland.wordpress.com/2012/12/15/throttling-io-with-linux/