In May we had what we believe was a hard drive failure on our centOS server in the Hetzner DC. Hetzner replaced the faulty drive and after a fsck we were back up and running.
I've just done a tw_cli >> show alarms and it shows that on August 14th the same drive encountered an error. Here's is the log:
c0 [Tue Aug 14 23:13:48 2012] ERROR Drive timeout detected: port=1
c0 [Tue Aug 14 23:13:51 2012] ERROR Degraded unit: unit=0, port=1
c0 [Tue Aug 14 23:14:03 2012] ERROR Drive timeout detected: port=1
It appears that I have a verify schedule running every 7 days which hasn't run since, for whatever reason. The RAID is showing as DEGRADED however we've noticed no problems and no filesystem errors like before, when the server wouldn't boot into the OS properly without going read-only.
My concern is we're now coming up to our busiest window for business online and so anything I do to the server needs to be as non-invasive and cause as little downtime as possible. Of course we have backups and such like so there's no real concern of data loss, just stability.
What action(s) can I take to try and remedy the situation? If I go to Hetzner they will insist on 12hrs+ downtime whilst they perform diagnostics and I'm trying to avoid that if possible.
I've also attached the output from /c0 show all incase it helps:
/c0 Driver Version = 2.26.08.003-2.6.18RH
/c0 Model = 9650SE-2LP
/c0 Memory Installed = 112MB
/c0 Firmware Version = FE9X 4.10.00.007
/c0 Bios Version = BE9X 4.08.00.002
/c0 Monitor Version = BL9X 3.08.00.001
/c0 Serial Number = L325030A9520128
/c0 PCB Version = Rev 035
/c0 PCHIP Version = 2.00
/c0 ACHIP Version = 1.90
/c0 Number of Ports = 2
/c0 Number of Units = 1
/c0 Number of Drives = 2
/c0 Total Optimal Units = 0
/c0 Not Optimal Units = 1
/c0 JBOD Export Policy = off
/c0 Disk Spinup Policy = 1
/c0 Spinup Stagger Time Policy (sec) = 1
/c0 Auto-Carving Policy = off
/c0 Auto-Carving Size = 2048 GB
/c0 Auto-Rebuild Policy = on
/c0 Controller Bus Type = PCIe
/c0 Controller Bus Width = 1 lane
/c0 Controller Bus Speed = 2.5 Gbps/lane
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-1 DEGRADED - - - 1396.97 ON ON
Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 1.36 TB 2930277168 9VS4CWQV
p1 DEVICE-ERROR u0 1.36 TB 2930277168 9VS3FLNM
Any advice is appreciated.
Cheers.
