Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a problem that has been keeping me somewhat busy for about a year. In random intervals, sata harddiscs simply get lost after several unsuccessful hard resets. This is not related with thermal issues (I keep a complete log of all temperature sensors), nor with the load on the system (in fact it seems more likely to happen on an idle system). I've recently switched from 2.6.26 to 2.6.32 and the problem has gone noticeably worse to about biweekly crashes (before: average once every three months).

A typical log entry of such an event looks like this (from 2.6.32 kernel)

ata1: exception Emask 0x10 SAct 0x0 SErr 0x90200 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { Persist PHYRdyChg 10B8B }
ata1: hard resetting link
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1.00: qc timeout (cmd 0xec)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata1.00: revalidation failed (errno=-5)
ata1: hard resetting link
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1.00: qc timeout (cmd 0xec)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata1.00: revalidation failed (errno=-5)
ata1: hard resetting link
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1.00: qc timeout (cmd 0xec)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata1.00: revalidation failed (errno=-5)
ata1.00: disabled
ata1: hard resetting link
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: [sda] Unhandled error code
sd 0:0:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 01 06 9b 8f 00 00 10 00
end_request: I/O error, dev sda, sector 17210255
Buffer I/O error on device sda1, logical block 2151274
lost page write due to I/O error on sda1
Buffer I/O error on device sda1, logical block 2151275
lost page write due to I/O error on sda1
sd 0:0:0:0: rejecting I/O to offline device
Aborting journal on device sda1.
sd 0:0:0:0: rejecting I/O to offline device
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1: link online but 1 devices misclassified, retrying
ext3_abort called.
EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
ata1: reset failed (errno=-11), retrying in 5 secs
ata1: hard resetting link
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1: link online but 1 devices misclassified, retrying
ata1: reset failed (errno=-11), retrying in 5 secs
sd 0:0:0:0: rejecting I/O to offline device
ata1: hard resetting link
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: link online but device misclassifed
ata1: link online but 1 devices misclassified, retrying
ata1: reset failed (errno=-11), retrying in 30 secs
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device

Kernel 2.6.26 gives slightly different logs (last log entries before losting root partition = where /var/log is stored... I'm thinking of using a software raid on all my harddiscs for /var/log :->>)

Jun  3 15:49:57 athlon64 kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x90200 action 0xe frozen
Jun  3 15:50:10 athlon64 kernel: ata1: irq_stat 0x00400000, PHY RDY changed
Jun  3 15:50:10 athlon64 kernel: ata1: SError: { Persist PHYRdyChg 10B8B }
Jun  3 15:50:10 athlon64 kernel: ata1: hard resetting link
Jun  3 15:50:10 athlon64 kernel: ata1: SATA link down (SStatus 0 SControl 300)
Jun  3 15:50:10 athlon64 kernel: ata1: failed to recover some devices, retrying in 5 secs

I've tested multiple harddiscs on multiple ports of the same controller (including one SSD), reduced the SATA bandwidth to 1.5Gbps, switched off NCQ for all devices, several BIOS updates (including SSD firmware), switching from pure AHCI to IDE mode in the BIOS - no effect. I'm using the M3A79-T Deluxe Motherboard by Asus which has a AMD SB750 SATA controller...

00:11.0 SATA controller [0106]: ATI Technologies Inc Unknown device [1002:4390] (prog-if 01 [AHCI 1.0])
        Subsystem: ASUSTeK Computer Inc. Unknown device [1043:81ef]
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
        I/O ports at 9000 [size=8]
        I/O ports at 8000 [size=4]
        I/O ports at 7000 [size=8]
        I/O ports at 6000 [size=4]
        I/O ports at 5000 [size=16]
        Memory at f6fff800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [60] Power Management version 2
        Capabilities: [70] #12 [0010]

... and seems to need the SB600 PMP workaround during startup and sometimes during operation. In some cases the SB600 PMP SRST workaround appears before a device becomes dead, but in most cases it does not. This is how it looks like in dmesg during boot:

ata3: softreset failed (device not ready)
ata2: softreset failed (device not ready)
ata2: applying SB600 PMP SRST workaround and retrying
ata1: softreset failed (device not ready)
ata1: applying SB600 PMP SRST workaround and retrying
ata3: applying SB600 PMP SRST workaround and retrying
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: FORCE: horkage modified (noncq)
ata1.00: ATA-8: OCZ-VERTEX, 1.41, max UDMA/133
ata1.00: 62533296 sectors, multi 1: LBA48 NCQ (not used)
ata1.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      OCZ-VERTEX       1.41 PQ: 0 ANSI: 5
ata2.00: FORCE: horkage modified (noncq)
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

I'm really stumped by this problem, any ideas? I'm even thinking of getting a PCIe SATA controller which is very well supported and very stable by 2.6.32 (any suggestions?) and stop using the on-board controller entirely.

UPDATE: Adding irqpool to the kernel boot command line seems to improve stability (82 days uptime until now) but severely impacts performance. There are still similar log entries about once a month (for various harddiscs, not only ata1)

 ata1: exception Emask 0x10 SAct 0x0 SErr 0x90200 action 0xe frozen
 ata1: irq_stat 0x00400000, PHY RDY changed
 ata1: SError: { Persist PHYRdyChg 10B8B }
 ata1: hard resetting link
 ata1: softreset failed (device not ready)
 ata1: applying SB600 PMP SRST workaround and retrying
 ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
 ata1.00: configured for UDMA/133
 ata1: EH complete

So I think it's the SB600 PMP SRST workaround/reset combined with a "hiccup" in irq handling...

share|improve this question
    
If anyone knows of recent changes (kernel > 2.6.32) in libata (specifically SB600/700 PMP SRST workaround handling), I would be happy to patch my kernel and check them out. irqpoll is not the way to go for me. –  user47544 Nov 9 '10 at 8:40
    
Sorry to say so, but irqpoll does not solve the problem. The system has broken down last night with exactly the same log entries as above. –  user47544 Nov 28 '10 at 10:13

6 Answers 6

We're now having this issue as well on hardware that's been stably running Linux (Debian then Ubuntu) for years. Suddenly in the last weeks, 5 identical machines exhibit this problem. And these are the simplest imaginable servers: no hardware raid, Xeon, SATA.

-It smacks of a newly introduced kernel bug.-

Actually, scratch that. This problem is already pretty well documented on launchpad.net for a while now. It's also be summarily ignored:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/480665 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/665163

I've rebuilt my problem machines with EXT3 and they've been fine for over a week. Not bad considering they could hardly make it a day before.

Remind me again what's so special about EXT4?

/p

share|improve this answer
    
I think so, too. For a workaround, adding irqpool to the kernel boot command line seems to resolve the problem somewhat (82 days uptime until for with 2.6.32 kernel) but has still similar log messages about once a month. –  user47544 Nov 9 '10 at 8:28
    
However, irqpoll severely impacts performance - my QuadCore is now about as fast as a single core machine. :-( –  user47544 Nov 9 '10 at 8:36
    
irqpoll ameliorates but does not resolve the problem. My system broke down last night with the same log entries... –  user47544 Nov 28 '10 at 10:14

We've seen similar issues in our servers (drive disappers, only a power cycle will recover them) with a variety of Seagate and Hitachi drives.

In our case, there's a strong correlation between drive vibration and these drive lockups - do your systems have large cooling fans?

share|improve this answer

I just went through this myself and eventually realized every single drive I hooked up had this problem under load. They were all Seagate's and 7200.12 so I thought they were all bad (this is no joke as we know the problem with these defective drives).

At first I suspected it to be a bad SATA cable, but it wasn't (I swapped it) and eventually I realized I daisy chained 3 drives off one MOLEX connector. I changed it so only the two were that way (the third one is the problem one which wasn't able to get enough power).

So my answer is really that maybe you overloaded your PSU or a single MOLEX, so try an individual/alternate one and you may find the same thing as me, that it's simply a power issue.

share|improve this answer

What drives? Velociraptors have a known issue with some firmware versions that ersult in malfunctions every around 50 days - for some seconds. If that happens at a wrong time.... well.... they drop out. Others sometimes also have problems. Seeking firmware updates sounds like a decent idea.

share|improve this answer
1  
This problem appears for all kinds of drives, including... Western Digital (WD15EADS-00R6B0) Seagate (ST32000542AS rev. CC32, ST31500341AS rev. CC1H) Hitachi (HTS541680J9SA00) OCZ Vertex SSD (rev. 1.41) - I've initially had the problem with the OCZ Vertex - removed, checked, replaced by Notebook harddisc (Hitachi) - then the same problem appeared for the first Seagate, this was also replaced by a 2TB Hitachi, etc.. There was no single HD that was in the system all the time, and none of the HDs showed any problem in another system with extensive read/write tests. –  user47544 Jul 6 '10 at 7:05

I see this using 2.6.32 in Debian Squeeze, only on a 10TB Drobo-S via eSATA on an Intel Ibex Peak AHCI controller. None of the other disks attached to the controller exhibit the issue, I don't have another eSATA drive to test with.

This Drobo will not appear through this controller if hot-plugged. It has to be present & fully initialized at POST to be present at all.

I'm not able to install Windows to this controller to see if it hot-plugs or drops-out on that kernel.

The same Drobo on a Windows 7 machine through an Intel ICH9 controller does hot-plug, thus does not have to be fully initialized at POST, and the drop-out issue does not occur. I have yet to test that controller with this kernel.

I suspected the Ibex Peak controller wasn't forwarding the Drobo's presence to the OS until I started Googling & see this is a relatively common error.

I'll post back to this thread in case I come across the appropriate bug report.

share|improve this answer
    
Do you also have a SB600/700 AHCI controller? Do you also get SB600 PMP SRST workaround messages in the log? If no, yours is probably a different bug or perhaps related to the specific harddisc. –  user47544 Nov 9 '10 at 8:38
    
Ok, thanks for clarifying- there is that subtle difference. –  NginUS Nov 10 '10 at 6:20

We've essentially abandoned using SATA disks for all but client machines simply because almost none of them have a >30% 'duty cycle'. This is hard to spot in the documentation but they're designed to be active for less than a third of a day, which is great for a desktop or laptop but their MTBF figure drop significantly if you exceed this. And I'm not talking being busy 24 hours a day, I mean actual use, however infrequent, affects their reliability.

share|improve this answer
    
hmmm. Been using sata discs in raids 24x7 here for years, litrally. –  Sirex Nov 10 '10 at 11:02
    
That's good Sirex, I'm genuinely happy for you but we had something like 8000-9000 1TB SATA/FATA disks in a variety of arrays from HP, HDS and NetApp and saw a >20% first year failure rate and >15% second year rate. These were in different data centres with lots of different uses, we gave up and went back to SAS/FC even though it cost us. –  Chopper3 Nov 10 '10 at 16:53
    
It's a different problem - the SATA disks are completely ok, down to the most severe read/write tests (which in the meantime I've done on all of my discs). These are all transient errors which completely disappear after reboot. Have you checked cooling on your systems? SAS are typically rated for higher temperature, perhaps that explains your different findings. –  user47544 Nov 28 '10 at 10:12

protected by Chris S May 16 '12 at 17:21

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?