0

I got a mail to root, (same error in syslog)

The following warning/error was logged by the smartd daemon:

Device: /dev/bus/0 [megaraid_disk_12] [SAT], ATA error count increased from 0 to 9

Device info: ST13000NM0005-2A1201, S/N:ZVJ2GZSP, WWN:5-000c50-0b3bg13c3, FW:SN02, 12.0 TB

After running two tests on it,

$ smartctl -l selftest /dev/bus/0 -d megaraid,12

/dev/bus/0 [megaraid_disk_12] [SAT]: Device open changed type from 'megaraid,12' to 'sat+megaraid,12'
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     11445         -
# 2  Short offline       Completed without error       00%     11427         -

IN storecli,

$ sudo storcli /call show
Generating detailed summary of the adapter, it may take a while to complete.

Controller = 0
Status = Success
Description = None


Virtual Drives = 3

VD LIST :
=======

------------------------------------------------------------
DG/VD TYPE   State Access Consist Cache sCC       Size Name
------------------------------------------------------------
0/0   RAID1  Optl  RW     Yes     RWBD  -   223.062 GB
1/1   RAID1  Optl  RW     Yes     RWTD  -     3.492 TB
2/2   RAID10 Optl  RW     Yes     RWBD  -    32.740 TB
------------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 12

PD LIST :
=======

-----------------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model                      Sp
-----------------------------------------------------------------------------------
0:0       1 Onln   0 223.062 GB SATA SSD N   N  512B INTEL SSDSC2*******        U
0:1       2 Onln   0 223.062 GB SATA SSD N   N  512B INTEL SSDSC2*******        U
0:2       6 Onln   1   3.492 TB SATA SSD N   N  512B Samsung SSD 883 DCT 3.84TB U
0:3      10 Onln   1   3.492 TB SATA SSD N   N  512B Samsung SSD 883 DCT 3.84TB U
0:4       8 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:5      11 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:6       3 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:7       4 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:8       7 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:9      12 Onln   2  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       U
0:10      9 GHS    -  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       D
0:11      5 GHS    -  10.913 TB SATA HDD N   N  512B ST13000NM0005-2A1201       D
-----------------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded


=== START OF READ SMART DATA SECTION ===
SMART Status not supported: ATA return descriptor not supported by controller firmware
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (  575) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (1125) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x70bd) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   064   044    Pre-fail  Always       -       157163520
  3 Spin_Up_Time            0x0003   090   090   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       11
  5 Reallocated_Sector_Ct   0x0033   098   098   010    Pre-fail  Always       -       9208
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       920073154
  9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       11564
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       11
187 Reported_Uncorrect      0x0032   091   091   000    Old_age   Always       -       9
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   071   048   040    Old_age   Always       -       29 (Min/Max 24/52)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   093   093   000    Old_age   Always       -       15951
194 Temperature_Celsius     0x0022   029   052   000    Old_age   Always       -       29 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a   082   064   000    Old_age   Always       -       157163520
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       2824 (96 42 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3015256
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3036621419213

SMART Error Log Version: 1
ATA Error Count: 9 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 9 occurred at disk power-on lifetime: 11377 hours (474 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 ff ff ff 4f 00  39d+08:08:45.869  READ VERIFY SECTOR(S) EXT
  42 00 01 ff ff ff 4f 00  39d+08:08:45.176  READ VERIFY SECTOR(S) EXT
  35 00 01 ff ff ff 4f 00  39d+08:08:45.175  WRITE DMA EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:29.406  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:29.190  READ VERIFY SECTOR(S) EXT

Error 8 occurred at disk power-on lifetime: 11377 hours (474 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 ff ff ff 4f 00  39d+08:08:29.406  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:29.190  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:29.119  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:28.814  READ VERIFY SECTOR(S) EXT
  42 00 01 ff ff ff 4f 00  39d+08:08:28.143  READ VERIFY SECTOR(S) EXT

Error 7 occurred at disk power-on lifetime: 11377 hours (474 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 ff ff ff 4f 00  39d+08:08:18.534  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:10.326  READ VERIFY SECTOR(S) EXT
  42 00 01 ff ff ff 4f 00  39d+08:08:09.645  READ VERIFY SECTOR(S) EXT
  35 00 01 ff ff ff 4f 00  39d+08:08:09.645  WRITE DMA EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:00.404  READ VERIFY SECTOR(S) EXT

Error 6 occurred at disk power-on lifetime: 11377 hours (474 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 ff ff ff 4f 00  39d+08:08:00.404  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:00.226  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:08:00.083  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:07:57.845  READ VERIFY SECTOR(S) EXT
  42 00 01 ff ff ff 4f 00  39d+08:07:57.215  READ VERIFY SECTOR(S) EXT

Error 5 occurred at disk power-on lifetime: 11377 hours (474 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 ff ff ff 4f 00  39d+08:07:37.571  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:07:37.554  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:07:37.536  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:07:37.519  READ VERIFY SECTOR(S) EXT
  42 00 00 ff ff ff 4f 00  39d+08:07:32.546  READ VERIFY SECTOR(S) EXT

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     11445         -
# 2  Short offline       Completed without error       00%     11427         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Can you please help in understanding and getting the failed disk if any. Thanks

3

Yes, this 10 Tb disk, with serial no. ZVJ2GZSP, is dying. I wonder why initial report said it is 16 Tb.

It already used over 9000 sectors in reserved area:

5 Reallocated_Sector_Ct   0x0033   098   098   010    Pre-fail  Always       -       9208

I think you can remove it safely, because all storage on this RAID controller is redundant (RAID1 or RAID 10) and all is Optimal. You have to set this disk to Offline. Once disk is set offline one of GHS (global hot spares) should get up and replace this disk in its array.

This disk is in the enclosure bay no. 9 (remember, these are 0-based, so it is 10th one). However, to be certain, you might use locate function of your enclosure.

For details see controller manual.

4
  • Thank you @Nikita Kipriyanov. How you got it's dying? The smart tests are saying no issue, in storecli it's saying online and spun up. – gokam Jan 12 at 19:46
  • HDD has some extra space, which isn't visible to the user. If firmware finds a "bad enough" element, it gets "remapped" to this reserved area next write. Usually drive starts remapping blocks rapidly, i.e. many blocks get bad in a short time, due to some mechanical problem, like dust appeared inside. Now things will only get worse over time. Over 9K remapped blocks for a disk which is less than 2 years old is very large number, in my opinion that must be in the order of units, not thousands. Also your drive has at least 9 blocks already "bad" but not yet remapped (no write was done into them). – Nikita Kipriyanov Jan 12 at 20:17
  • For me that drive is unreliable enough to get rid of it. Of course you can rely on its internal "smart status". I've seen plenty of obviously failed drives (in particular, that were kicked from RAID after repeated failed reads) which smart status was still "good" by some reason. BTW, do your controller have patrol reads? That helps to find and kick out unreliable drives early. That's a must have. – Nikita Kipriyanov Jan 12 at 20:23
  • Ok Thank you. Replaced the disk. – gokam Jan 15 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.