Harddrive / Raid1 issues

Question

I'm having problems with the hard-drives in my root-server.

It's running with 2 hdds in a software Raid1.

After having performance issues with a large MySQL DB, that writes via O_DIRECT and innodb_flush_log_at_trx_commit=0, I replaced one of the disks (sda), because SMART values showed high error rates.

A week ago, there were performance issues again and following messages showed up in the syslog:

Sep 25 15:09:41 server02 kernel: ata1.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen
Sep 25 15:09:41 server02 kernel: ata1.00: failed command: READ FPDMA QUEUED
Sep 25 15:09:41 server02 kernel: ata1.00: cmd 60/08:08:58:ed:1e/00:00:79:00:00/40 tag 1 ncq 4096 in
Sep 25 15:09:41 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 25 15:09:41 server02 kernel: ata1.00: status: { DRDY }
Sep 25 15:09:41 server02 kernel: ata1: hard resetting link
Sep 25 15:09:41 server02 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 25 15:09:41 server02 kernel: ata1.00: configured for UDMA/133
Sep 25 15:09:41 server02 kernel: ata1.00: device reported invalid CHS sector 0
Sep 25 15:09:41 server02 kernel: ata1: EH complete 

/dev/sdb3 was automatically deactivated by mdadm. I then replaced the second harddrive (sdb) and changed cables and ports.

Now, I'm getting the same errors again:

Oct  2 21:44:46 server02 kernel: ata2.00: exception Emask 0x0 SAct 0x6 SErr 0x0 action 0x6 frozen
Oct  2 21:44:46 server02 kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct  2 21:44:46 server02 kernel: ata2.00: cmd 60/18:08:10:08:24/00:00:79:00:00/40 tag 1 ncq 12288 in
Oct  2 21:44:46 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct  2 21:44:46 server02 kernel: ata2.00: status: { DRDY }
Oct  2 21:44:46 server02 kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct  2 21:44:46 server02 kernel: ata2.00: cmd 60/10:10:30:08:24/00:00:79:00:00/40 tag 2 ncq 8192 in
Oct  2 21:44:46 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct  2 21:44:46 server02 kernel: ata2.00: status: { DRDY }
Oct  2 21:44:46 server02 kernel: ata2: hard resetting link
Oct  2 21:44:46 server02 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Oct  2 21:44:46 server02 kernel: ata2.00: configured for UDMA/133
Oct  2 21:44:46 server02 kernel: ata2.00: device reported invalid CHS sector 0
Oct  2 21:44:46 server02 kernel: ata2.00: device reported invalid CHS sector 0
Oct  2 21:44:46 server02 kernel: ata2: EH complete

And /dev/sdb3 was deactivated again.

Does anyone have any ideas, what might be wrong with my server?

Thanks! Christian

Answer 1

After replacing two HDDs you again have errors indicating a hardware problem with a disk. This can mean that you are just plain unlucky, or that there's some other problem, which exhibits itself as a disk failure. This could be:

1. Problem with the main board -- maybe the SATA controller is faulty.
2. Problem with cabling -- loose, frayed, bent, etc.
3. Environment -- temperature, vibrations, humidity.
4. Problem with the SATA chip drivers.
5. Unstable AC.

Can you reproduce the problem on a test/development machine? Is there something, that triggers the failure (backups, load spike, particular query)? Is the problem intermittent (you log a couple of errors and then the server runs OK), or persistent (once it starts showing up, it keeps showing up)?

Personally, I wouldn't trust the machine and I'd migrate the data to some other server while looking for the cause of this errors. 3 disk failures in short time are possible (people win lottery sometimes), but they aren't very probable.