Hot answers tagged

38

You have a double disk failure. This means your data is gone, and you will have to restore from a backup. This is why we aren't supposed to use raid 5 on large disks. You want to set up your raid so you always have the ability to withstand two disk failures, especially with large slow disks.


37

Your options are: Restoring from backups. You do have backups, don't you? RAID is not a backup. Professional data recovery It's possible, though very expensive and not guaranteed, that a professional recovery service will be able to recover your data. Accepting your data loss and learning from the experience. As noted in the comments, large SATA disks ...


25

I'm sorry. But this is operator error. You had two failing disks on a RAID5 array, and you removed more disks than the array could sustain. Doing this without any backups is the bigger mistake. You should contact a data recovery firm to attempt to retrieve the data from the broken Logical Drive.


24

After you accepted a bad answer, I am really sorry for my heretic opinion (which saved such arrays multiple times already). Your second failed disk has probably a minor problem, maybe a block failure. This is the cause, why the bad sync tool of your bad raid5 firmware crashed on it. You could easily make a sector-level copy with a lowlevel disk cloning ...


19

If you lose more than a single disk in a RAID 5, your array has been irreperably damaged in some way. In most cases, the data is entirely destroyed in your case if you're not an expert at recovery, or if you are unwilling to ship it off to a recovery outfit. If you DO want to recover the data from this array, take it offline immediately and either recover it ...


18

Designing the reliability of a disk array: Find the URE Rate of your drive (manufacturers don't like to talk about their drives failing, so you might have to dig to find this. It should be 1/10^X where X is around 12-18 commonly). Decide what is an acceptable risk rate for your storage needs†. Typically this is <0.5% chance of failure, but could be ...


17

Why would you ask the internet about this? There's so much WTF here, that I don't understand where to start!! This question shows a fundamental lack of understanding of hardware, RAID arrays, storage, monitoring, and general IT best-practices. I read this question and can't help but think: Who is actually responsible for this server hardware? Where is ...


15

Your system I assume is still up, so the best thing to do is make an immediate backup, dump the disks/array, rebuild, and restore from the backup. Bad blocks don't always mean your backups are also bad. If you haven't experienced any performance problems or damaged files, then your backups should still be complete enough to finish a restore. To test, take ...


12

Please use RAID 1+0 with your controller and drive setup. If you need more capacity, a nested RAID level like RAID 50/60 could work. You can get away with RAID 5 on a small number of enterprise SAS disks (8 drives or fewer) because the rebuild times aren't bad. However, 24 drives is a terrible mistake. (Oh, and disable the individual disk caching feature... ...


11

Your reasoning is correct, though you're missing the scale of the problem. Enterprise SSDs are being made with higher endurance MLC cells, and can tolerate very high write-rates. SLC still blows high-endurance MLC out of the water, but in most cases the lifetime write-endurance of HE-MLC exceed the expected operational lifetime of a SSD. These days, ...


11

Do not power the system back on again. Shut it down, call a data recovery service. There are a number of services that allow for remote recovery of this type of failure. At this point, all you can do is make it worse. This often involves connecting all drives directly to a known-good HBA (not a RAID card or other controller!) and starting a specific ...


10

RAID 5 should never exist with a hot spare (warm spare.) RAID 6 is always a better use of the same drive count. http://www.smbitjournal.com/2012/07/hot-spare-or-a-hot-mess/ There is no space/capacity advantage or cost advantage to the RAID 5 solution (but some small performance advantage) but it does a ton for mitigating things like URE risk.


10

With RAID 5 you can only lose 1 disk and have your data remain available. You have lost 3 so you will need to rebuild the RAID and then restore the data from backup. We have a canonical Q&A about RAID levels that may help your understanding.


10

You'd be left with only half the capacity, so if the file system was more than half full already, it would be impossible. Even if there is sufficiently free space, the operation involves resizing the file system, which requires intimate knowledge of the file system. Moreover applying ordinary resizing tools to a degraded RAID5 before converting it to RAID1 ...


10

You probably have an heavily punctured array, which cause an early "planned death" of the replacement disk due to failed stripe reconstruction. You can read more information here and here The solution is to backup, destroy the array, recreate it and restore from backup. Next time avoid using a RAID5 array with such big drives. I strongly suggest using ...


9

I found 2 research papers about this topic: Parity update increases write workload and space utilization Introduction [...] The results from our analytical model show that RAID5 is less reliable than striping with a small number of devices because of write amplification. Conclusion [...] Different factors such as the number of devices ...


9

The reason that article exists is to draw attention to Unrecoverable Bit Error Rates on HDDs. Specifically, your cheap 'home PC' disks. They typically have a factory spec of 1 / 10^14. This is about 12.5TB of data, which if you are doing a RAID-5 with 2TB disks ... you hit quite quickly. This means you should either: use smaller RAID groups, and accept ...


9

You've had a double drive failure, with one of the drives being dead for six months. With RAID5, this is irrecoverable. Replace the failed hardware and restore from backup. Going forward, consider RAID6 with large drives like this and make sure you have monitoring in place to catch device failures so you can respond to them ASAP.


9

This is a bit crazy... A Smart Array P600 PCI-X RAID controller (circa 2005)?!? 25 disks? RAID 5? Is this an HP MSA70 enclosure? Is probably not the HP D2700? "Ready for Rebuild" is about the worst array status message you can receive on an HP ProLiant system. This indicates that the logicaldrive can't finish its rebuild because there's trouble reading from ...


8

Right this instant, do the following: Stop rotating backups or deleting old ones for this system. You want to keep all of the backups you currently have. Take a full backup of the server. Hopefully the disks are still good enough that your data is intact, and you won't encounter any problems running the new full backup. Then scrap those disks, and build ...


8

OK, the reason Windows Server Backup is failing is because of the cluster size you're using on the volume. (And I'll explain exactly why that is at the end, after the important issue of your RAID array being a time bomb.) But before addressing the backup issue, we need to address the issue with your RAID setup. Don't use RAID5 with large disks. And don't ...


8

If you make a mistake, you can lose all your data. Backup first. Then continue. Use storcli /c0 show to see what drives and volumes you have. The TOPOLOGY table is a good start: TOPOLOGY : ======== ---------------------------------------------------------------------------- DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR ------...


8

So just to clarify, you had a 4-disk R5 array, you replaced 2 disks at once - is that right? will the data get rebuild automatically or do I need to do some other changes If what I'm reading you've done is correct then no, no it won't get rebuilt, ever and you've destroyed your data and yes you will have to do some other changes in that you'll have to ...


7

Parity RAID will thrash your $300 desktop SATA SSD. It will not even put a dent in a $3000 enterprise grade SSD. It's all about what you're shopping for and what your use case is. SSD is a much more mature technology than it used to be. On the high end, their MTBF and max writes are approaching the same sort of reliability as mechanical HDDs. One reason ...


7

You should acquire new hardware and install a modern operating system like Windows Server 2012 R2 and then migrate your applications and data to it. Windows Server 2003 is quickly approaching end of support and you should be focusing your effort on getting off of it, not on rigging it to keep on running.


7

It's not possible to say precisely what the odds of X drives going out in Y amount of time are, but it is safe to say that drive failures are not completely independent, as commonly assumed. Multiple disk failures in the same array within close temporal proximity are actually a fairly common occurrence. Less than a month ago, we had 4 drives fail over ...


7

This is a bit nuts... That's too many disks for a RAID5. It's offset by the fact that you have two hot-spare drives, but damn!! However, you've already expanded the underlying Array, but not the Logical Drive. Remember, the "Array" is the physical grouping of disks. The "Logical Drive" is what you assign RAID level to. You can have multiple Logical Drives ...


6

Yes... This is possible. So you're saying you have 7 disks in the RAID 5 set, and want to take disk #8, which is currently a spare, and move it to become the 8th member of the RAID 5 array? If you have the HP Array Configuration Utility (HP ACU) installed, this is very easy. There is a web-based GUI and a command-line option to handle this change. It can ...


6

I had a similar problem: after a failure of a software RAID5 array I fired mdadm --create without giving it --assume-clean, and could not mount the array anymore. After two weeks of digging I finally restored all data. I hope the procedure below will save someone's time. Long Story Short The problem was caused by the fact that mdadm --create made a new ...


6

It's actually rather common, and the main reason it is frequently advised to buy hard drives from different batches in a single RAID set. Identical batches often have identical flaws or thresholds. Also, failures don't always result from just simple old age of the drive, they can also be triggered by minimal power surges, unexpected load for a few minutes, ...


Only top voted, non community-wiki answers of a minimum length are eligible