Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having problems with the hard-drives in my root-server.

It's running with 2 hdds in a software Raid1.

After having performance issues with a large MySQL DB, that writes via O_DIRECT and innodb_flush_log_at_trx_commit=0, I replaced one of the disks (sda), because SMART values showed high error rates.

A week ago, there were performance issues again and following messages showed up in the syslog:

Sep 25 15:09:41 server02 kernel: ata1.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen
Sep 25 15:09:41 server02 kernel: ata1.00: failed command: READ FPDMA QUEUED
Sep 25 15:09:41 server02 kernel: ata1.00: cmd 60/08:08:58:ed:1e/00:00:79:00:00/40 tag 1 ncq 4096 in
Sep 25 15:09:41 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 25 15:09:41 server02 kernel: ata1.00: status: { DRDY }
Sep 25 15:09:41 server02 kernel: ata1: hard resetting link
Sep 25 15:09:41 server02 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 25 15:09:41 server02 kernel: ata1.00: configured for UDMA/133
Sep 25 15:09:41 server02 kernel: ata1.00: device reported invalid CHS sector 0
Sep 25 15:09:41 server02 kernel: ata1: EH complete 

/dev/sdb3 was automatically deactivated by mdadm. I then replaced the second harddrive (sdb) and changed cables and ports.

Now, I'm getting the same errors again:

Oct  2 21:44:46 server02 kernel: ata2.00: exception Emask 0x0 SAct 0x6 SErr 0x0 action 0x6 frozen
Oct  2 21:44:46 server02 kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct  2 21:44:46 server02 kernel: ata2.00: cmd 60/18:08:10:08:24/00:00:79:00:00/40 tag 1 ncq 12288 in
Oct  2 21:44:46 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct  2 21:44:46 server02 kernel: ata2.00: status: { DRDY }
Oct  2 21:44:46 server02 kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct  2 21:44:46 server02 kernel: ata2.00: cmd 60/10:10:30:08:24/00:00:79:00:00/40 tag 2 ncq 8192 in
Oct  2 21:44:46 server02 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct  2 21:44:46 server02 kernel: ata2.00: status: { DRDY }
Oct  2 21:44:46 server02 kernel: ata2: hard resetting link
Oct  2 21:44:46 server02 kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Oct  2 21:44:46 server02 kernel: ata2.00: configured for UDMA/133
Oct  2 21:44:46 server02 kernel: ata2.00: device reported invalid CHS sector 0
Oct  2 21:44:46 server02 kernel: ata2.00: device reported invalid CHS sector 0
Oct  2 21:44:46 server02 kernel: ata2: EH complete

And /dev/sdb3 was deactivated again.

Does anyone have any ideas, what might be wrong with my server?

Thanks! Christian

share|improve this question
    
Does your controller card have diagnostics you can run. Sounds like it's having problems if it's failed 3 hard drives now... –  Chris S Oct 3 '11 at 13:37

1 Answer 1

After replacing two HDDs you again have errors indicating a hardware problem with a disk. This can mean that you are just plain unlucky, or that there's some other problem, which exhibits itself as a disk failure. This could be:

  1. Problem with the main board -- maybe the SATA controller is faulty.
  2. Problem with cabling -- loose, frayed, bent, etc.
  3. Environment -- temperature, vibrations, humidity.
  4. Problem with the SATA chip drivers.
  5. Unstable AC.

Can you reproduce the problem on a test/development machine? Is there something, that triggers the failure (backups, load spike, particular query)? Is the problem intermittent (you log a couple of errors and then the server runs OK), or persistent (once it starts showing up, it keeps showing up)?

Personally, I wouldn't trust the machine and I'd migrate the data to some other server while looking for the cause of this errors. 3 disk failures in short time are possible (people win lottery sometimes), but they aren't very probable.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.