SATA hdd errors

Question

I have one server with WDC WD3202ABYS ... There are 100 virtual hosts. Server is working about 5 years and in this time period I have change 4 disks. All with same reason: sata error. The last one:

ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:60:57:7b:b6/00:01:06:00:00/e0 tag 0 dma 180224 out
         res 51/10:60:57:7b:b6/10:01:06:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found

Descriptor sense data with sense descriptors (in hex):
        72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        06 b6 7b 57 
end_request: I/O error, dev sdb, sector 112622423
Buffer I/O error on device dm-8, logical block 14077747
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077748
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077749
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077750
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077751
lost page write due to I/O error on dm-8
Buffer I/O error on device dm-8, logical block 14077756
lost page write due to I/O error on dm-8
ata2: EH complete
SCSI device sdb: 625142448 512-byte hdwr sectors (320073 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:90:17:30:b7/00:02:08:00:00/e0 tag 0 dma 335872 out
         res 51/10:90:17:30:b7/10:02:08:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found

Descriptor sense data with sense descriptors (in hex):
        72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        08 b7 30 17 
end_request: I/O error, dev sdb, sector 146223127
printk: 34 messages suppressed.
Buffer I/O error on device dm-8, logical block 18277835

Looks like some software error...

but in short time after that (maybe when I started fsck) following error:

EXT3-fs error (device dm-8): ext3_put_super: Couldn't clean up the journal
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/01:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x1 (device error)
ata2.00: status: { DRDY ERR }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)

Is it possible that this errors are also "software" ... I mean this HDD is only 9000 hours old ... where are no extra load on HDD... temperature is 29 Celsius... Do I need to replace hdd? or check disk is enough?

EXT3-fs error (device dm-8): ext3_put_super: Couldn't clean up the journal
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/01:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x1 (device error)
ata2.00: status: { DRDY ERR }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x4
ata2.00: cmd c8/00:00:8f:0d:84/00:00:00:00:00/e1 tag 0 dma 131072 in
         res 51/40:00:a8:0d:84/10:02:08:00:00/e1 Emask 0x9 (media error)

How to find out reason?

---

Here are errors from smart:

Error 36 occurred at disk power-on lifetime: 9160 hours (381 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 22 09 80 e3  Error: UNC at LBA = 0x03800922 = 58722594

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 1f 09 80 03 0a  47d+13:38:13.534  READ DMA
  ec 00 00 00 00 00 00 0a  47d+13:38:13.530  IDENTIFY DEVICE
  ef 03 46 00 00 00 00 0a  47d+13:38:13.528  SET FEATURES [Set transfer mode]

Ok. Is it possible the following scenario: 1. Disk was on 9000 without fsck. 2. There are some errors 3. In dmesg there started errors like:

ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x5
ata2.00: cmd 35/00:60:57:7b:b6/00:01:06:00:00/e0 tag 0 dma 180224 out
         res 51/10:60:57:7b:b6/10:01:06:00:00/e0 Emask 0x81 (invalid argument)
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: SCSI error: return code = 0x08000002
sdb: Current [descriptor]: sense key: Aborted Command
    Add. Sense: Recorded entity not found

4. And errors like inode error and so on...
5. I tried to umount this parition, and error come from hdd like it cannot find such inode and so on... ?

If so.. I do not understand. Do I need to change disk every year? Just to prevent this error? Does some one have same problem? Not only with one disk...

Answer 1

In my experience the errors you're seeing are actually hardware errors reflected in software. The 'lost page write due to I/O error' message is one I've seen with bad hard-drives, and it behaves similar to how you describe when attempting to fsck it. This is almost definitely a true hardware fault.

You should check the output of smartctl to see what it says could be problem.

smartctl --attributes /dev/sdb

It'll give you output similar to this:

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED     RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   212   186   021    Pre-fail  Always       -       4358
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       97
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   200   200   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   066   066   000    Old_age   Always       -       25420
 10 Spin_Retry_Count        0x0013   100   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0013   100   253   051    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       86
194 Temperature_Celsius     0x0022   104   001   000    Old_age   Always       -       46
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      -       0

The output can be arcane, but the one I'd pay close attention to would be Reallocated_Sector_Ct, since that tells you what the HD has for known bad sectors. The command 'smartctl -a' will give a lot more data. On the bad HD I had a while back, the bottom of that output is the 'SMART Error Log' which had a few entries.

Answer 2

You had an uncorrectable read error.

Error: UNC at LBA = 0x03800922 = 58722594

The data that was on that block is now lost.

You should:

• be using a mirror in the first place. Enterprise disks are actually intended to be behind a mirror and they would rather return a read error than try really hard to get the data.
• recover the lost data from backups

You have NO EXCUSE to not be using RAID (especially if you host website for clients!) - the OS is not that large, you don't need a dedicated disk for it on a 2-disk system.

Answer 3

Are you using a RAID controller? What kind of controller are you using?

One of the things (that's both frustrating and enlightening) is the increasing segmentation HDD manufacturers are introducing into the SATA market place. There are now drives for 'small-enterprise/RAID usage' and 'single/desktop usage'. SAS seems to be pushed for the 'high-end enterprise' market.

Your Model # is WD's RE3 series of drives, designed for RAID setups. I'm told this means among other things) the drive will 'give up' sooner (ie, within 3-4 seconds) when trying to repair errors, instead of trying over and over for a longer period of time. Giving up sooner reports the error to the RAID controller, so it can use another drive to recover from. If instead, the drive waits longer, the RAID controller would kick the drive out of the array for being unresponsive.

Failures should still be rare, and not once every year. Perhaps it's another aspect of your setup? (I had a frustrating fight with a SATA cable once, It's now mounted over my door as a warning to other cables ...)

Answer 4

I've had a very bad experience with Western Digital drives. More than half of my drives have had to be replaced under warranty due to complete failure or bad sectors.

After buying only WD drives for about 8 years, I no longer want to spend any money with them. I have no idea which WD drive I can trust; my experience so far says "none of them".

You've replaced the original drive 4 times; did you buy all five drives at the same time? Buy new to replace each one when it failed? Return the drive under warranty for a replacement? How and when did you acquire the five drives and what models were they? In my experience batches of WD drives are often bad and fail at the same time.