I have a IBM System x3650 server with a ServeRaid controller and two RAID5 arrays, each consisting of 3 disks.

Yesterday, one disk failed (It was the Raid array that holds the data, the system is located on the sound array). I naively trusted the RAID controller in rebuilding the array. I shut down the server, replaced the failed disk with a new similar. I booted in the controller bios, where I could see that it recognized the new disk and was ready to rebuild (I had nothing to do, everything was automatic). I started the server and it rebuilt the array.

This morning everything seemed OK. The rebuild was finished, the array seemed sound. Only a few hours later, the mysql service crashed with a corrupted database. I managed to dump the data partially and restored the rest from backup. I thought I was OK.

But then I found that some active logfiles were corrupt: they included blocks from different random files. If I appreciate the situation correctly, only files modified since the rebuild has started are corrupted, but I'm not yet 100% sure for this. Somehow, the rebuild must have corrupted the data.

I ask this question to learn out of error. I hope the next time will be never...

What can be the reason that the rebuild failed ? What can I do better next time ?
Is it compulsary to cut the server from the network during rebuild ? I thought, the controller should manage concurrently rebuild and make ordinary reads and writes.
Or shouldn't this never happen, and maybe the controller is faulty ?

  • What model of disks do you use in raid? Apr 27 '17 at 20:16

From your description, it seem that the rebuild did not fail, in the sense that the array was up and running. However, it seems that the rebuild process caused some blocks to be wrongly placed/remapped, which is an extraordinary rare but dangerous thing.

I suggest you to take the time to examine the situation. Did you read/follow the RAID card manual? Are you 100% sure that you did the right things? If the reply to both question is "yes", you should immediately open a support case with your server vendor/consultant.

  • Thanks for your precious advice. For the moment pass the night restoring backups and copying some Teras around, hoping that tomorrow morning everything is up and running. The server is no longer covered by warranty, but I will still call their support to see what they say. Apr 27 '17 at 21:27

I learnt a lesson again I already new: Read the event logs. I would have seen errors that state that the file system is corrupted. I could have planned for the downtime an hour or two ahead of time. I also knew that only one logical volume was affected, and not the whole physical raid volume.

What I can do better next time:

A raid rebuild is an important event that requires a complete server checkup once it is finished.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.