Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a 64GB SSD drive in a tower server with a local colocation company. This drive and the enter system was built about six months ago, brand new parts.

Until this weekend the SSD/system were working perfectly. We're running CentOS 6.2

After booting perfectly, the system can be used about 20-30 minutes (no real consistency with time) before the drive starts acting funny.

Libraries start saying they can't load, ssh starts denying public key logins. Shutdown starts saying "input/outout error". Some programs start indicating the drive is read-only.

Only 25GB of the 64GB are used.

I can't find any errors that indicate what happened. I tried running fsck from a live cd on the drive and it showed no problems and most of the time boot works fine. There was one boot that said "couldn't find os" but that's not happening anymore.

Where can I look to find logs about what happens? Are there any other disk checks I should do? It seems like a repairable problem, and not that I need a new drive.

Update:

I enabled SMART after rebooting the server. After 1 hour of uptime and normal system operation (running services are httpd, mysql, but very little to no traffic), suddenly things just stop working. During the hour of uptime it responded with a PASS for the smart health check. After the hour I tried it again (through webmin) and it now says SMART is disabled.

The hard drive is now showing the same issues I've seen before - trying most commands show "input/output error".

Running a smart health check now shows:

Log Sense failed, IE page [scsi response fails sanity test]

What can I do to figure out what's causing this to fail after a random period of time? It runs perfectly for 30-60 minutes and then it starts acting odd like this.

Update 2

Some requested that I try dmesg and this was the result: http://www.pastie.org/private/hk7jfhxilj7ypy828irna. Someone else recommended that I not assume it's the drive, but possibly the drive controller. I don't understand how to determine whether the error is the controller versus the drive - aside from trying a different drive. If I have to buy a replacement motherboard or drive I need to know which is failing first.

Running fsck shows:

fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/mapper/vg_192-lv_root

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
share|improve this question
    
When something starts to 'fail', are you seeing lots of errors on the console? What types of errors? –  Zoredache Apr 23 '12 at 19:51
    
No errors, but I may not be looking in the right place –  BotskoNet Apr 23 '12 at 19:58
    
Just look at the console, or run dmesg. If you have drive errors or filesystem corruption you should be getting errors displayed. Unless you modified theprintk settings anyway. –  Zoredache Apr 23 '12 at 20:03
    
Assuming the drive is failed and therefore in read-only mode, maybe its caching a few of the writes until a buffer fills up? –  djangofan Apr 23 '12 at 21:04
    
Here is a bit of what dmesg shows: pastie.org/private/hk7jfhxilj7ypy828irna –  BotskoNet Apr 23 '12 at 21:37
show 2 more comments

3 Answers

SSDs are notoriously fragile. Jeff Atwood outlines some failure rates here. They will fail without any warning and turn your data into a distant memory.

Looks like it's time to RMA and restore from backup. It shouldn't be a problem though, because you're not running a production server on a single, non-RAIDed disk, right? And you definitely have recent backups you can use to get back on your feet, right?

Right?

share|improve this answer
2  
.........right? –  JohnThePro Apr 23 '12 at 19:55
    
lol right, we do have night backups and no, it's not a production server. –  BotskoNet Apr 23 '12 at 19:58
1  
"turn your data into a distant memory" is an exaggeration. When SSD drives fail they just revert to read-only mode and so you can still get your data. Losing your data is very unlikely, as you suggest. –  djangofan Apr 23 '12 at 20:54
    
My point is that it shouldn't matter if one disk dies, because you should have RAID in place. If you do have RAID in place, dumping data off of one failed disk is pointless. –  Joel E Salas Apr 24 '12 at 0:28
    
Depends on the nature of the failure and the kind of RAID you have. I do hope that RAID controllers handle it properly. –  Yuhong Bao Jun 27 at 6:41
add comment

If your hard drive has SMART statistics (and it is almost guaranteed to have them) use a SMART utility to cull all of the available messages and statistics. The answer likely lies there, or at least some hint as to where to look next.


EDIT

Consider that you might be misdirecting your suspicions. Your drive controller could be part of the problem. Look into what metrics that it collects as well as what logs it creates. Keep it in the circle of suspects for now. Everything in IT is guilty until proven innocent.

share|improve this answer
    
I enabled smart after rebooting the server. After 1 hour of uptime and normal system operation (running services are httpd, mysql, but very little to no traffic). –  BotskoNet Apr 23 '12 at 20:40
    
@BotskoNet I updated my answer with another possibility. –  Wesley Apr 23 '12 at 20:43
    
How can I find logs or run checks on the drive controller? –  BotskoNet Apr 23 '12 at 20:49
    
@BotskoNet That is 100% dependent upon the server and controller manufacturer. You'll have to find some documentation and peruse through it. –  Wesley Apr 23 '12 at 20:50
1  
"Everything in IT is guilty until proven innocent" - nice. And even then, it can still be guilty. ****ing computers. –  Mark Sowul May 5 '12 at 18:12
show 1 more comment

I had exact the same fault with my home PC running an EXT-4 filesystem on a 64Gb Crucial/ Micron M4 SSD. I ran smartctl -a on my device and it passed all tests satisfactorily. I booted my server from a systemrescue cd and reran smartctl and this detected old firmware, v 0009 known to cause issues and provided the fix. My firmware is now at release 070H and the problems have now vanished. So the soluton in my case was to visit the crucial website and download a smal bootable iso image to update my SSD firmware. No more Input/ Output errors

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.