Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In May we had what we believe was a hard drive failure on our centOS server in the Hetzner DC. Hetzner replaced the faulty drive and after a fsck we were back up and running.

I've just done a tw_cli >> show alarms and it shows that on August 14th the same drive encountered an error. Here's is the log:

c0   [Tue Aug 14 23:13:48 2012]  ERROR     Drive timeout detected: port=1
c0   [Tue Aug 14 23:13:51 2012]  ERROR     Degraded unit: unit=0, port=1
c0   [Tue Aug 14 23:14:03 2012]  ERROR     Drive timeout detected: port=1

It appears that I have a verify schedule running every 7 days which hasn't run since, for whatever reason. The RAID is showing as DEGRADED however we've noticed no problems and no filesystem errors like before, when the server wouldn't boot into the OS properly without going read-only.

My concern is we're now coming up to our busiest window for business online and so anything I do to the server needs to be as non-invasive and cause as little downtime as possible. Of course we have backups and such like so there's no real concern of data loss, just stability.

What action(s) can I take to try and remedy the situation? If I go to Hetzner they will insist on 12hrs+ downtime whilst they perform diagnostics and I'm trying to avoid that if possible.

I've also attached the output from /c0 show all incase it helps:

/c0 Driver Version = 2.26.08.003-2.6.18RH
/c0 Model = 9650SE-2LP
/c0 Memory Installed  = 112MB
/c0 Firmware Version = FE9X 4.10.00.007
/c0 Bios Version = BE9X 4.08.00.002
/c0 Monitor Version = BL9X 3.08.00.001
/c0 Serial Number = L325030A9520128
/c0 PCB Version = Rev 035
/c0 PCHIP Version = 2.00
/c0 ACHIP Version = 1.90
/c0 Number of Ports = 2
/c0 Number of Units = 1
/c0 Number of Drives = 2
/c0 Total Optimal Units = 0
/c0 Not Optimal Units = 1 
/c0 JBOD Export Policy = off
/c0 Disk Spinup Policy = 1
/c0 Spinup Stagger Time Policy (sec) = 1
/c0 Auto-Carving Policy = off
/c0 Auto-Carving Size = 2048 GB
/c0 Auto-Rebuild Policy = on
/c0 Controller Bus Type = PCIe
/c0 Controller Bus Width = 1 lane
/c0 Controller Bus Speed = 2.5 Gbps/lane

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       1396.97   ON     ON     

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     1.36 TB     2930277168    9VS4CWQV            
p1     DEVICE-ERROR     u0     1.36 TB     2930277168    9VS3FLNM 

Any advice is appreciated.

Cheers.

share|improve this question
    
I'd have them replace the faulty drive and the controller, just to be sure. –  Michael Hampton Aug 28 '12 at 7:05
    
Hi Michael, Obviously this will result in significant downtime, correct, however if installing a new controller and drive, will it then be able to rebuild the array from the "good" drive that exists? –  fRAiLtY- Aug 28 '12 at 7:21
    
Yes, you would be down for as long as it takes to replace the hardware (and possibly rebuild the array). Replacing the controller isn't a problem as long as it's the same or similar model. And, now is a really good time to start thinking about moving into "the cloud." –  Michael Hampton Aug 28 '12 at 7:33
    
Hi Michael, but it would rebuild the array from the good disk? What is the uncertainty over whether I'd be down during the rebuild process as this took over 16hrs last time! –  fRAiLtY- Aug 28 '12 at 7:45
    
It only takes a few minutes to swap a card, so count on 17 hours. And did I mention the cloud yet? :) –  Michael Hampton Aug 28 '12 at 7:47
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.