On Friday I had an absolute disaster. I went to the datacenter in order to rack a new server and at the same time, took a new disk with me to replace a drive that had failed a couple of weeks back in a HP P2000 G3 10Gbit iSCSI array.
The P2000 is loaded with 12 x 2TB 7.2k MDL SAS disks and configured as 2 RAID10 arrays, each with 4 drives + 2 hot spares. I had already removed the failed disk a week earlier, its place in the array had been taken automatically by one of the hot spares as expected.
So, the task in hand was a simple drive replacement to give me back a hot spare. Simple right? Slide in the disk, make sure its visible in the GUI and configure as a hot spare....
I opened the brand new disk from its packaging and slid it into place, instantly all the drives in the array went orange. I checked the GUI and every single disk was showing 'Invalid metadata', a quick check of running services using the two arrays confirmed that everything had lost visibility of the LUN's.
I rescanned the channels, rebooted the controllers all to no effect. Drives started disappearing from the GUI and my VD's (LUN's) were now missing from the GUI as well. I removed the new disk as well, still nothing.
In an act of desperation and confusion I pulled the power to the P2000 and let it fully reboot. It came back online and I could see my VD's again... However both arrays had lost all redundancy, like each half of the 2 RAID10's had lost their mirror disks.
All the other disks that were once part of the array were now showing as available. I was able to configure them as hot spares and the 2 RAID10's began reconstruction. It is now running again, all be it without my new disk since I am too scared to put it in again.
Does anyone have a clue about what happened here?
The only thing I can think of is that the new disk must have contained metadata of its own and confused the P2000. However it was a new sealed disk from our usual supplier. However even if this was the case, I wouldn't expect the array to do anything with that disk that effects the existing RAID configuration!