9

We have an old HP DL380G5 server, with 5 300GB SCSI 3.5'' disks in a RAID5 array, in an external bay, formatted as a logical volume with ext3 filesystem, that hosts 1.2 TB of sensitive clinical patients data.

Two disks showed predictive failure in hpacucli, so I replaced one of them first, and saw that it was OK, but I didn't see that it also states "Ready for rebuilding". I totally carelessly changed the second one as well, and now it says that the RAID has FAILED.

I returned the old disk back, tried rebooting the server but it now puts me into recovery mode during boot, and says it cannot find the logical volume.

Anything I can do to try to restore this? We do not have a backup, unfortunately. Any help would really be appreciated!

I was thinking about returning BOTH old drives back, is there any chance this would revive the RAID?

  • Comments are not for extended discussion; this conversation has been moved to chat. – Michael Hampton Mar 16 '18 at 20:43
  • 4
    I imagine your group will start making backups now. If it was ever a question of necessity or cost, this should be a fairly clear warning shot. – Jonathon Reinhart Mar 17 '18 at 18:13
25

I'm sorry. But this is operator error.

You had two failing disks on a RAID5 array, and you removed more disks than the array could sustain.

Doing this without any backups is the bigger mistake.

You should contact a data recovery firm to attempt to retrieve the data from the broken Logical Drive.

  • 1
    Sounds like he's knows it's operator error … That's not a reason to not ask what he can do now – StarWeaver Mar 17 '18 at 0:07
  • @StarWeaver Yes... contacting a data recovery firm is the appropriate next step. – ewwhite Mar 17 '18 at 0:44
11

Do not power the system back on again. Shut it down, call a data recovery service. There are a number of services that allow for remote recovery of this type of failure. At this point, all you can do is make it worse.

This often involves connecting all drives directly to a known-good HBA (not a RAID card or other controller!) and starting a specific downloadable linux image with remote management tools. The company then remotely accesses the system, assesses the disk status, and recovers any RAID metadata left. Using proprietary software, they can re-assemble a virtual RAID disk (technical detail: often something that plugs into the standard Linux device-mapper system). This then exposes the RAID read-only in-software (with no RAID SoC accelerator). Next steps are verifying the data is not corrupted beyond use and cloning the virtual disk to a new disk to complete the data recovery. After that you can worry about getting the system back up and running.

While I'm not going to name any services here, most of them are easy to find, and for the ones with remote services (saving you the roundtrip of shipping the RAID drives + recovery drive to them and waiting for the recovery + clone and then them sending it back) you get the benefit of the data never actually leaving your facility.


A small amount of good news: as long as the RAID controller (or you) didn't write any new data to any of the disks, and the pre-fail warning is not a fail-warning, there is practically a 99.9999% chance a good data recovery team can restore all of it, and reasonably quick too.

5

Re: restoring the old drives.

Since your RAID is completely dead as it stands, you have little to lose by refitting the two pre-fail drives.

Do install them in the original bays.

Remember they're pre-fail not failed outright, so there is a fair chance they will run for long enough to rescue your data.

There is a chance the raid simply won't come up, and a small chance the controller will ask to "reset" the raid (choose NO/CANCEL) and a tiny chance the raid controller might automatically reset the raid which would negate any value added by a data recovery firm.

So your utter top priority if the RAID comes up, is to get the data off. That means having at least 1.2 TB space available and ready to copy data off, and a tool like robocopy or xcopy32 or in your linux case rsync ready to run. You don't want to waste time reading man pages and figuring out the syntax if your drives are wasting their last minutes.


Once your data is safe, then recreate the raid as a raid6 with the new drives. You'll drop 300GB of capacity, but gain a two-drive tolerance. Or add an additional drive and consider a raid10 over 6 drives. Or consider retiring this machine completely; the G5 is over 10 years old and is really not suitable for important production tasks any more.

And not trying to put the boot in, but do set up a proper backup solution too. There will be a next time.

  • 2
    Just to clarify - there is a small but non-zero chance that doing this will make it harder for any commercial data recovery firm to succeed in assisting you. Personally I'd try to reassemble the raid, and if that doesn't work then take stock. – Criggie Mar 16 '18 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.