I found out one of our servers was down today due to a HDD failure. The only thing I could see on console the was:
scsi 0:0:0:0: rejecting I/O to dead device.
This server has 2 HDDs with a software RAID 1. After rebooting and running a manual fsck it booted fine. I need to identify which HDD is failing and usually I would use smartctl. But here is what I get:
# smartctl -i /dev/sdb
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
Probable ATA device behind a SAT layer
Try an additional '-d ata' or '-d sat' argument.
Both -d ata and -d sat results in:
# smartctl -i -d ata /dev/sdb
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
Adding -T permissive also results in an error.
Strangely hdparm also fails:
# hdparm -i /dev/sdb
/dev/sdb:
HDIO_GET_IDENTITY failed: Invalid argument
Also, to make things even more strange:
Aug 15 03:14:14 server smartd[3426]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Aug 15 03:14:14 server smartd[3426]: Device: /dev/hda, opened
Aug 15 03:14:15 server smartd[3426]: Device: /dev/hda, packet devices [this device CD/DVD] not SMART capable
Aug 15 03:14:15 server smartd[3426]: Device: /dev/sda, opened
Aug 15 03:14:15 server smartd[3426]: Device: /dev/sda, is SMART capable. Adding to "monitor" list.
Aug 15 03:14:15 server smartd[3426]: Device: /dev/sdb, opened
Aug 15 03:14:16 server smartd[3426]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list.
Aug 15 03:14:16 server smartd[3426]: Monitoring 0 ATA and 2 SCSI devices
Aug 15 03:14:17 server smartd[3428]: smartd has fork()ed into background mode. New PID=3428.
During boot it is smart capable?
This happens to both sdb and sda which have different sizes and are from different vendors. System is using /dev/sda currently but RAID array is being rebuilt with /dev/sdb. So far no errors regarding this.
I know hdparm and smartctl doesn't work for all HDD models/vendors. But smartctl was working before on this server.
My question: what is happening? :)
And how can I identify which HDD is failing before this server freezes again?
