Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have an Ubuntu 13.04 server. Today I found the box had crashed. I restarted it, and now every single hard drive's partition table is missing. (1 SSD for /boot, /, and 3 2TB drives for RAID).

I have the SSD connected to a laptop VIA USB->SATA cable, and sure enough, the partition table is missing. This tells me that the Motherboard / SATA controller / software actually broke the drives, not that they just can't be read correctly.

Something similar happened to only the SSD a few months ago, and I was forced to just re-partition it.

How the heck could his have happened? Bad Motherboard or SATA controller?

share|improve this question

closed as too broad by Scott Pack, Ward, mdpc, Jenny D, Tim Brigham Jul 1 '13 at 17:03

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Depending on how you're doing the RAID, it's quite possible that the drives are not supposed to have a partition table. Can you elaborate more on the type of RAID (hardware/software, etc.)? Be sure to consider the possibility that the drives are not being assembled within the array correctly. (i.e. drives got jumbled between channels somehow) –  Andrew B Jun 29 '13 at 20:01
    
What were you using for the RAID and why wasn't the SSD in a fault tolerant config? Were you using md or ZFS or a RAID controller or something like the crappy Intel Storage Manager baked into cheapo motherboards? –  MDMarra Jun 29 '13 at 20:44
3  
Cosmic rays ... –  Iain Jun 29 '13 at 21:39
    
I had it some days ago. I've installed older PCI Promise raid controller with 4 HDs. After creation of two raid partitions and formatting everything looked OK (e.g after 2 days). However, when I started to move data from old server I found that most of directories were empty. Also after reboot mdadm started to re-synchronize RAID partitions. Fortunately for me these were not the boot and root so I could check the messages. I found the info that the IRQ 185 was "removed". I don't need to say that this was IRQ of Promise. The most funny was info from kernel "deleyed allocation failed.... This sh –  tatus2 Jun 29 '13 at 21:43
    
What type of drive are they? If they're WD Green (or equivalent), then it's entirely plausible. –  Tom O'Connor Jun 30 '13 at 6:47

3 Answers 3

This could have been bad/corrupted memory. Considering running memtest86+ to find out for sure.

share|improve this answer

It doesn't need to be the disks per se but it could be faulty memory, faulty raid controller or a bug. Make sure you are on the latest firmware.

share|improve this answer

There could be a lot of reasons. Data corruption doesn't need to be caused by just the things you've outlined, and doesn't imply a broken drive or that anything broke it.

The SSD's firmware could be bad. The controller (old, new, or both) could be bad. There could be a root or kernel process which was running in bad memory and overwrote the beginnings and ends of the drives. The CPU might even be bad. It's also possible that all the drives are actually bad (this doesn't happen often, but it does happen sometimes). If you are using software RAID with LVM, you might have upgraded to a buggy version or something, or just encountered a random bug.

The best thing to do is to take a bytewise image of any drive you need to recover data from, and manipulate that. Write it onto another drive, write a partition table exactly the same as the one that you expected to see there, and try mounting. Copy where you expect the filesystem to be on the drive, and mount it using a loop device. Use data recovery software of some kind. However, the easiest thing to do is restore from backup.

It isn't immediately clear what kind of hardware you are using. I would nonetheless run a full hardware test on the server (at least memtest, but do HDD tests and a CPU test if you have a capable test suite). Test the drives on the controller and on another controller if you can, and check their SMART status. Update everything related to the drives (the filesystem drivers, kernel, and LVM if it is in use, particularly). If you have a hardware RAID device, consider upgrading its firmware.

I have had this issue caused by several faulty RAID controllers in the past too. If it has blanked parts of multiple drives, get an RMA for it and put in a new one.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.