0

I received some alerts from mdadm. Further checking seem to indicate that some HDD's may be damaged.

Could you please verify that?

There is two SCSI drives which should work in RAID1

This is an automatically generated mail message from mdadm
running on host...

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1]
md0 : active raid1 sda2[0](F) sdb2[1]
972597112 blocks super 1.2 [2/1] [_U]

unused devices:

Output of mdadm --detail /dev/md0

root@data:/# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Apr 12 06:43:28 2013
Raid Level : raid1
Array Size : 972597112 (927.54 GiB 995.94 GB)
Used Dev Size : 972597112 (927.54 GiB 995.94 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Tue Jul 16 11:02:38 2013
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0

Name : grml:0
UUID : ac43ac06:67d7f5e8:147f9c57:f087c896
Events : 768474

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 18 1 active sync /dev/sdb2

0 8 2 - faulty spare /dev/sda2

I just want to add, i booted it in rescue mode and this is the output i get with smartctl tests:

The authenticity of host '78.31.69.128 (78.31.69.128)' can't be established.
RSA key fingerprint is 6b:10:c9:74:d5:50:4e:48:6f:e9:54:54:fd:b3:b4:82.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '78.31.69.128' (RSA) to the list of known hosts.
root@78.31.69.128's password:
Grml - Linux for geeks

root@grml ~ # parted -l
Model: ATA Hitachi HUA72201 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 134kB 4000MB 4000MB primary linux-swap(v1)
2 4000MB 1000GB 996GB primary boot, raid


Model: ATA Hitachi HUA72201 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 134kB 4000MB 4000MB primary
2 4000MB 1000GB 996GB primary boot, raid


root@grml ~ # smartctl -s on -t short -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.7-1-grml-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Hitachi Ultrastar A7K2000
Device Model: Hitachi HUA722010CLA330
Serial Number: JPW9K0N01PS0SL
LU WWN Device Id: 5 000cca 39ad7f9ba
Firmware Version: JP4OA3MA
User Capacity: 1.000.204.886.016 bytes [1,00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Tue Jul 16 15:42:09 2013 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.

General SMART Values:
Offline data collection status: (0x85) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9572) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 160) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 096 096 016 Pre-fail Always - 524288
2 Throughput_Performance 0x0005 137 137 054 Pre-fail Offline - 90
3 Spin_Up_Time 0x0007 128 128 024 Pre-fail Always - 294 (Average 296)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 70
5 Reallocated_Sector_Ct 0x0033 001 001 005 Pre-fail Always FAILING_NOW 2015
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 144 144 020 Pre-fail Offline - 28
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 6009
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 70
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 82
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 82
194 Temperature_Celsius 0x0002 142 142 000 Old_age Always - 42 (Min/Max 15/52)
196 Reallocated_Event_Count 0x0032 001 001 000 Old_age Always - 2033
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
ATA Error Count: 326 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 326 occurred at disk power-on lifetime: 4758 hours (198 days + 6 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 dc 64 87 3e 00 Error: UNC at LBA = 0x003e8764 = 4097892

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 18 08 40 88 3e 40 08 16d+23:38:55.512 READ FPDMA QUEUED
60 f8 00 48 87 3e 40 08 16d+23:38:55.511 READ FPDMA QUEUED
61 02 00 08 38 77 40 08 16d+23:38:50.104 WRITE FPDMA QUEUED
61 20 00 90 45 bc 40 08 16d+23:38:48.761 WRITE FPDMA QUEUED
61 02 00 08 38 77 40 08 16d+23:38:48.694 WRITE FPDMA QUEUED

Error 325 occurred at disk power-on lifetime: 3541 hours (147 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 03 85 01 fb 09 Error: UNC at LBA = 0x09fb0185 = 167444869

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 00 80 01 fb 40 08 5d+19:53:07.617 READ FPDMA QUEUED
ef 10 02 00 00 00 a0 08 5d+19:53:07.616 SET FEATURES [Reserved for Serial ATA]
27 00 00 00 00 00 e0 08 5d+19:53:07.615 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 08 5d+19:53:07.609 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 08 5d+19:53:07.608 SET FEATURES [Set transfer mode]

Error 324 occurred at disk power-on lifetime: 3541 hours (147 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 03 85 01 fb 09 Error: UNC at LBA = 0x09fb0185 = 167444869

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 00 80 01 fb 40 08 5d+19:52:49.411 READ FPDMA QUEUED
ef 10 02 00 00 00 a0 08 5d+19:52:49.410 SET FEATURES [Reserved for Serial ATA]
27 00 00 00 00 00 e0 08 5d+19:52:49.409 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 08 5d+19:52:49.403 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 08 5d+19:52:49.402 SET FEATURES [Set transfer mode]

Error 323 occurred at disk power-on lifetime: 3541 hours (147 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 03 85 01 fb 09 Error: UNC at LBA = 0x09fb0185 = 167444869

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 00 80 01 fb 40 08 5d+19:52:31.204 READ FPDMA QUEUED
ef 10 02 00 00 00 a0 08 5d+19:52:31.203 SET FEATURES [Reserved for Serial ATA]
27 00 00 00 00 00 e0 08 5d+19:52:31.202 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 08 5d+19:52:31.196 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 08 5d+19:52:31.195 SET FEATURES [Set transfer mode]

Error 322 occurred at disk power-on lifetime: 3541 hours (147 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 03 85 01 fb 09 Error: UNC at LBA = 0x09fb0185 = 167444869

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 00 80 01 fb 40 08 5d+19:52:12.792 READ FPDMA QUEUED
ef 10 02 00 00 00 a0 08 5d+19:52:12.791 SET FEATURES [Reserved for Serial ATA]
27 00 00 00 00 00 e0 08 5d+19:52:12.790 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 a0 08 5d+19:52:12.784 IDENTIFY DEVICE
ef 03 46 00 00 00 a0 08 5d+19:52:12.783 SET FEATURES [Set transfer mode]

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 6009 -
# 2 Extended offline Completed: read failure 90% 3370 161277656
# 3 Short offline Completed without error 00% 3369 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Tue Jul 16 15:43:11 2013

Use smartctl -X to abort test.
216 root@grml ~ # ls -la :(
total 43
drwxr-xr-x 7 root root 120 Jul 16 15:42 ./
drwxr-xr-x 32 root root 240 Jul 16 13:35 ../
drwx------ 3 root root 25 Mai 3 14:24 .cache/
drwx------ 4 root root 60 Jul 16 13:35 .config/
drwxr-xr-x 2 root root 60 Jul 16 15:35 .kbd/
drwx------ 3 root root 28 Mai 3 14:24 .local/
-rw------- 1 root root 748 Mai 3 14:27 .viminfo
-rw-r--r-- 1 root root 33992 Mai 3 14:18 .zcompdump
-rw-r--r-- 1 root root 8 Jul 16 15:41 .zdirs
-rw------- 1 root root 531 Jul 16 15:42 .zsh_history
root@grml ~ # smartctl -s on -t short -a /dev/sdb
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.7-1-grml-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Hitachi Ultrastar A7K2000
Device Model: Hitachi HUA722010CLA330
Serial Number: JPW9K0N01JNXSL
LU WWN Device Id: 5 000cca 39ad5ac00
Firmware Version: JP4OA3MA
User Capacity: 1.000.204.886.016 bytes [1,00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Tue Jul 16 15:43:39 2013 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x85) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9753) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 163) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 136 136 054 Pre-fail Offline - 93
3 Spin_Up_Time 0x0007 128 128 024 Pre-fail Always - 295 (Average 295)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 70
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 140 140 020 Pre-fail Offline - 30
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 6011
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 70
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 80
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 80
194 Temperature_Celsius 0x0002 150 150 000 Old_age Always - 40 (Min/Max 14/49)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 0
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 6010 -
# 2 Extended offline Completed without error 00% 3373 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Tue Jul 16 15:44:41 2013

Use smartctl -X to abort test.
root@grml ~ # 
  • The (F) means it's failed. And each drive is a position in [xxx]. So a normal stat is [UU]. – Halfgaar Jul 16 '13 at 11:26
3

You have a mirror (RAID1) of two partitions (sda2 and sdb2 form md0).
Sda2 has failed.

  1. If there is nothing else on sda (e.g. sda2 is the size of the whole disk) then replace it.
  2. If sda has multiple partitions than first make sure that there is no active data on those.

Array Size and Used Dev Size seem the same or at least very close, which means that it probably is the first and it is safe to replace.

If you do not feel comfortable with that then first make a backup (and test that backup).

SCSI is usually hot plugable, but do check this (e.g. does the machine use SCSI with a 80 pins SCSI connector or is this a homebrewn setup with molex connectors for power and 50/68 pins for data. In the latter case powering it down before replacing the drive might be safer.

  • 1
    He writes "SCSI" but probably means SAS - there were no 1 TB SCSI drives in the market. – the-wabbit Jul 16 '13 at 11:19
  • 1
    Always run backups.. It's too late when hardware is malfunctioning, as heavy IO could potentially trigger another disk failure in the RAID (URE, common when people use disks from the same production batch) – pauska Jul 16 '13 at 16:11
1

Yes, you have a failing drive. Replace it. Looks like sda2 is the culprit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.