Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Note: this question was previously closed as off-topic. You can read the discussion. My reasons for asking it here are:

  • This drive is in an offline content cache server for schools in rural Zambia.
  • The servers are created from disk images, and all the content is replaceable.
  • It has to be cheap because Zambian schools are budget-limited and there will be a lot of them.
  • It also has to be reliable because it might be 8 hours each way on bad roads to replace.
  • I'm not allowed to ask here what drives are not "ultra-cheap crap".
  • So we're doing our own research and experimentation on drives that meet these criteria.
  • My inability to repair the bad sectors by overwriting them (auto reallocate) defied my assumptions and I wanted to know why.
  • I thought maybe a SECURITY ERASE might fix the bad sectors, but wanted opinions of others before I trash the drive.
  • I thought I might have missed something in the SMART data that could have predicted the failure.

This is a Kingston 240GB SSD disk that was working fine on site for about 3 months, and has suddenly developed bad sectors:

smartctl 5.41 2011-06-09 r3365 [i686-linux-3.2.20-net6501-121115-1cw] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     KINGSTON SVP200S3240G
Serial Number:    50026B7228010E5C
LU WWN Device Id: 5 0026b7 228010e5c
Firmware Version: 502ABBF0
User Capacity:    240,057,409,536 bytes [240 GB]
Sector Size:      512 bytes logical/physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-2 revision 3
Local Time is:    Tue Mar  5 17:10:24 2013 CAT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (  48) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x0021) SCT Status supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   084   050    Pre-fail  Always       -       10965286670575
  5 Reallocated_Sector_Ct   0x0033   100   100   003    Pre-fail  Always       -       16
  9 Power_On_Hours          0x0032   000   000   000    Old_age   Always       -       46823733462185
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       127
171 Unknown_Attribute       0x0032   000   000   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   000   000   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0030   000   000   000    Old_age   Offline      -       131
177 Wear_Leveling_Count     0x0000   000   000   000    Old_age   Offline      -       1
181 Program_Fail_Cnt_Total  0x0032   000   000   000    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   000   000   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   000   000   000    Old_age   Always       -       49900
194 Temperature_Celsius     0x0022   033   078   000    Old_age   Always       -       33 (Min/Max 21/78)
195 Hardware_ECC_Recovered  0x001c   120   120   000    Old_age   Offline      -       235163887
196 Reallocated_Event_Count 0x0033   100   100   003    Pre-fail  Always       -       16
201 Soft_Read_Error_Rate    0x001c   120   120   000    Old_age   Offline      -       235163887
204 Soft_ECC_Correction     0x001c   120   120   000    Old_age   Offline      -       235163887
230 Head_Amplitude          0x0013   100   100   000    Pre-fail  Always       -       100
231 Temperature_Celsius     0x0013   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0000   000   000   000    Old_age   Offline      -       363
234 Unknown_Attribute       0x0032   000   000   000    Old_age   Always       -       208
241 Total_LBAs_Written      0x0032   000   000   000    Old_age   Always       -       208
242 Total_LBAs_Read         0x0032   000   000   000    Old_age   Always       -       1001

SMART Error Log not supported
SMART Self-test Log not supported
SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Now I get bad blocks in certain places on the disk:

root@iPad2:~# badblocks /dev/sda -v
Checking blocks 0 to 234431063
Checking for bad blocks (read-only test): 8394752 done, 1:15 elapsed
8394756 done, 1:21 elapsed
8394757 done, 1:23 elapsed
8394758 done, 1:24 elapsed
8394759 done, 1:27 elapsed
...
190882871one, 29:49 elapsed
190882888one, 29:53 elapsed
190882889one, 29:54 elapsed
190882890one, 29:56 elapsed
190882891one, 29:58 elapsed
done                                
Pass completed, 80 bad blocks found.

They appear to be repeatable, and auto reallocation fails, so they can't be fixed by writing to them:

root@iPad2:~# badblocks /dev/sda -wvf 8394756 8394756
/dev/sda is apparently in use by the system; badblocks forced anyway.
Checking for bad blocks in read-write mode
From block 8394756 to 8394756
Testing with pattern 0xaa: 8394756
done                                
Reading and comparing: done                                
Testing with pattern 0x55: done                                
Reading and comparing: done                                
Testing with pattern 0xff: done                                
Reading and comparing: done                                
Testing with pattern 0x00: done                                
Reading and comparing: done                                
Pass completed, 1 bad blocks found.

And I get errors like this in the system logs:

ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
ata1.00: irq_stat 0x40000000
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:00:08:30:00/00:00:01:00:00/40 tag 0 ncq 4096 in
         res 51/40:08:08:30:00/00:00:01:00:00/40 Emask 0x409 (media error) <F>
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/133
sd 0:0:0:0: [sda] Unhandled sense code
sd 0:0:0:0: [sda]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda]  Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
        01 00 30 08 
sd 0:0:0:0: [sda]  Add. Sense: Unrecovered read error - auto reallocate failed
sd 0:0:0:0: [sda] CDB: Read(10): 28 00 01 00 30 08 00 00 08 00
end_request: I/O error, dev sda, sector 16789512
Buffer I/O error on device sda, logical block 2098689
ata1: EH complete

Now I don't understand why auto reallocate is failing on this disk. The smartctl output all looks fine to me. Only 16 sectors have been reallocated, that's not many at all. I can't see any legitimate reason why this drive refuses to reallocate sectors. Is this model of SSD just broken or badly designed?

Notes:

  • attribute 174 is "Unexpected Power Loss" according to Kingston's docs.
  • 131 unexpected power losses is quite bad.
  • attribute 187 (Reported_Uncorrect) is 49900 out of a possible maximum of 65535
  • highest temperature ever is quite high at 78'C

The most interesting SMART counters are hidden by Kingston on this drive. But we can infer the number of spare sectors from from attribute 196. Reallocated_Event_Count, which has the following formula for the normalised value:

100 -(100* RBC / MRC)
RBC = Retired Block Count (Grown)
MRE = Maximum reallocation count

Since the normalised value is 100, this implies that RBC << MRE, so we are nowhere close to having exhausted all available sectors for reallocation.

share|improve this question
3  
Your SMART data does not indicate any significant problems. Have you tested this disk on a different controller with a different cable to eliminate the other possibilities of hardware issues? –  Michael Hampton Mar 5 '13 at 15:15
1  
I have not yet tried another controller/cable, because some sectors read fine repeatably and others error repeatably. I'm deducing from this that the fault is in the drive, not with the controller or the cable. –  qris Mar 5 '13 at 15:52
1  
Your deduction is wrong because the exact same thing could happen with a bad controller or cable (albeit cable should be a bit more erratic, but it's worth trying anyway). –  gparent Mar 5 '13 at 16:12
3  
@gparent The deduction is not necessarily wrong; just the logic that brought about the conclusion. The drive may well be the culprit, but his reasons for thinking so have little basis. In any case, he should test on another controller/cable as suggested. –  Chris S Mar 6 '13 at 16:29
    
Yes, that's what I thought I had just said. I think you can safely consider any differences as part of a language barrier because I'm in complete agreement with your comment. –  gparent Mar 6 '13 at 19:53

1 Answer 1

up vote 1 down vote accepted

Cheap SSDs seem to have serious quality issues. You will find lots of users that have issues with your particular drive. However I think that the vendors also sell different drives (e.g. with other NAND chips/controllers) under the same label. So each drive may behave differently.

The SMART values give no indication that the drive would fail soon. In my experience its the same: Suddenly drive errors occur and then the disk fails.

What are your reasons why you use SSDs? I see the advantages of SSDs as there are no mechanical parts and as they are dustproof and produce less heat. However I also see a lot of disadvantages.

e.g. the number of writes to a single memory cell that even with wear-leveling may be reached quickly on a busy volume e.g. when you are using a filesystem with journaling.

And the electronics is also affected by high humidity or high temperatures - the same as with conventional hard drives.

Why not use cheaper conventional hard drives instead and (if raid is not required) ship the server with spare drives that are not connected until they are needed as a replacement (already present in the server case or mounted in a hot-swap cage so that the disk may be used in different servers). Then they could get prepared by a script in the field or from remote (if possible).

As long as conventional hard drives are not powered up, the transport to their destination can be rough...

If there are multiple school servers / permanent clients and a reliable/redundant network, maybe a distributed filesystem could also help to create a failsafe cache server (e.g. by using glusterfs).

share|improve this answer
    
Thanks Brian, I'm accepting this answer as I have no better ones. My experience with hard disks is that sectors can be reallocated and hence recovery is possible. You are right about the reasons for using SSDs. Dust proofing is a big issue for Zambia. The drives will not be written often, most of the content is static. Solid state electronics are less affected by heat and humidity than spinning rust. RAIDing hard disks had occurred to us, but we were hoping SSDs would be more reliable and thus save money. Perhaps we were wrong. And schools have very poor connectivity so glusterfs is out. –  qris May 9 '13 at 14:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.