1

My current diagnosis is that there is a hardware fault, either related to the memory of the hardware RAID controller or something related to the extra juice the hard drives requires when running on full.

Tried replacing the PSU, no dice. Tried live booting another OS, no dice. High disk IO ensures a hard shutdown, whatever the OS. I have established that a new server is definitely needed.

Now, however, I am facing the challenge of taking a back-up of the content of the drives, without causing a high IO. I have configured cgroups with a IOps limit which I have played around with to see how high it can be without causing a shutdown. The results seem to vary a bit from each disk, but an average of about 100 iops seems to work for a longer while before crashing. However this is not really feasible considering there is about 120GB of data. These are SAS 15K drives by the way.

Throttling the IO seems to work, but is a tiredsome process since I have to set it all up again when the server shuts down. I am copying the data over from the drives to an external drive using Runtime Live CD (Knoppix fork).

The server is about 7 years old, and I do not happen to have an extra connector for the drives.

When facing such a scenario, what is an reliable way of getting the data from the drives?

Just for reference, this is the link which helped me setup the cgroups limiting the IO: http://fritshoogland.wordpress.com/2012/12/15/throttling-io-with-linux/

1
  • Why not just automate the IO throttling code?
    – EEAA
    Aug 14, 2014 at 13:55

1 Answer 1

1

I have seen something similar myself, although it was a few years ago.

In my case it was problem memory and when copying data, there was some form of caching going on which I suspected was slowly using memory until it got to the problem memory and... boom! crashed computer. A memory test should identify this easily enough, or perhaps removing some memory to see if it resolves or worsens the problem?

If not, then I doubt its data throughput related, and that the throughput is simply delaying the problem whilst hiding the actual cause.

it might be worth checking CPU temps also just in case the data copy is causing enough CPU work to raise it to crashing point? i.e. with a failed fan or slipped heat sink? slowing the data copy is just reducing the burden enough to delay the temp rise.

Lastly, you don't say how your disks are configured? i.e. RAID, or JBOD? are you able to move the drives to another server individually, or as a group? obviously don't just move them if they are RAID drives except as a last, last resort! unless you know what you are doing, of course!

HTH

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .