0

Yesterday one of the disks in RAID 1 mirror attached to Adaptec 5405 died and was replaced by new one but after about 2 hours of rebuilding array went offline. Hoster staff updated firmware on controller and forced array online back. System booted normally and arcconf getconfig 1 showed following output:

Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status                        : Optimal
Channel description                      : SAS/SATA
Controller Model                         : Adaptec 5405
Controller Serial Number                 : 1D091194BC6
Physical Slot                            : 18
Temperature                              : 85 C/ 185 F (Normal)
Installed memory                         : 256 MB
Copyback                                 : Disabled
Background consistency check             : Disabled
Automatic Failover                       : Enabled
Global task priority                     : High
Performance Mode                         : Default/Dynamic
 Stayawake period                         : Disabled
Spinup limit internal drives             : 0
Spinup limit external drives             : 0
Defunct disk drive count                 : 0
Logical devices/Failed/Degraded          : 1/0/1
SSDs assigned to MaxCache pool           : 0
Maximum SSDs allowed in MaxCache pool    : 8
MaxCache Read Cache Pool Size            : 0.000 GB
MaxCache flush and fetch rate            : 0
MaxCache Read, Write Balance Factor      : 3,1
NCQ status                               : Enabled
Statistics data collection mode          : Enabled
--------------------------------------------------------
Controller Version Information
--------------------------------------------------------
BIOS                                     : 5.2-0 (18948)
Firmware                                 : 5.2-0 (18948)
Driver                                   : 1.1-7 (28000)
Boot Flash                               : 5.2-0 (18948)
--------------------------------------------------------
Controller Battery Information
--------------------------------------------------------
Status                                   : Not Installed 

----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical device number 0
Logical device name                      : 
RAID level                               : 1
Status of logical device                 : Degraded
Size                                     : 953334 MB
Read-cache mode                          : Enabled
MaxCache preferred read cache setting    : Enabled
Write-cache mode                         : Disabled (write-through)
Write-cache setting                      : Disabled (write-through)
Partitioned                              : Yes
Protected by Hot-Spare                   : No
Bootable                                 : Yes
Failed stripes                           : No
Power settings                           : Disabled
--------------------------------------------------------
Logical device segment information
--------------------------------------------------------
Segment 0                                : Present (Controller:1,Connector:0,Device:1)             9VP3AGB1
Segment 1                                : Rebuilding (Controller:1,Connector:0,Device:0)             Z1D49LKS


----------------------------------------------------------------------
Physical Device information
----------------------------------------------------------------------
  Device #0
     Device is a Hard drive
     State                              : Rebuilding
     Supported                          : Yes
     Transfer Speed                     : SATA 3.0 Gb/s
     Reported Channel,Device(T:L)       : 0,0(0:0)
     Reported Location                  : Connector 0, Device 0
     Vendor                             : 
     Model                              : ST1000DM003-9YN1
     Firmware                           : CC4H
     Serial number                      : Z1D49LKS
     Size                               : 953869 MB
     Write Cache                        : Enabled (write-back)
     FRU                                : None
     S.M.A.R.T.                         : No
     S.M.A.R.T. warnings                : 0
     Power State                        : Full rpm
     Supported Power States             : Full rpm,Powered off,Reduced rpm
     SSD                                : No
     MaxCache Capable                   : No
     MaxCache Assigned                  : No
     NCQ status                         : Enabled
  Device #1
     Device is a Hard drive
     State                              : Online
     Supported                          : Yes
     Transfer Speed                     : SATA 3.0 Gb/s
     Reported Channel,Device(T:L)       : 0,1(1:0)
     Reported Location                  : Connector 0, Device 1
     Vendor                             : 
     Model                              : ST31000528AS
     Firmware                           : CC38
     Serial number                      : 9VP3AGB1
     Size                               : 953869 MB
     Write Cache                        : Enabled (write-back)
     FRU                                : None
     S.M.A.R.T.                         : No
     S.M.A.R.T. warnings                : 0
     Power State                        : Full rpm
     Supported Power States             : Full rpm,Powered off
     SSD                                : No
     MaxCache Capable                   : No
     MaxCache Assigned                  : No
     NCQ status                         : Enabled

and aacraid-status

-- Controller informations --
-- ID | Model | Status
c0 | Adaptec 5405 | Optimal

-- Arrays informations --
-- ID | Type | Size | Status | Task | Progress
c0u0 | RAID1 | 953G | Degraded | Rebuild | 44%

-- Disks informations
-- ID | Model | Status

There is at least one disk/array in a NOT OPTIMAL state.

smartctl for drive #1 showed that it is in pre-failure state:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   099   006    Pre-fail  Always       -       69962026
  3 Spin_Up_Time            0x0003   095   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -        93
  5 Reallocated_Sector_Ct   0x0033   002   002   036    Pre-fail  Always   FAILING_NOW  4015
  7 Seek_Error_Rate         0x000f   083   060   030    Pre-fail  Always       -       222073391
  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       24485
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       72
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   006   006   000    Old_age   Always       -       94
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       1507556589923
189 High_Fly_Writes         0x003a   095   095   000    Old_age   Always       -       5
190 Airflow_Temperature_Cel 0x0022   059   051   045    Old_age   Always       -       41 (Min/Max 40/41)
194 Temperature_Celsius     0x0022   041   049   000    Old_age   Always       -       41 (0 19 0 0)
195 Hardware_ECC_Recovered  0x001a   045   028   000    Old_age   Always       -       69962026
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       202082506249469
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1020514404
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       590957029

and array failed to rebuild again and went offline.

What is the best way to proceed now with getting this system back online?

1 Answer 1

1

According to smart - drive might be in failing status, try to rebuild again and if reallocated sector count grows - it's definitely bad.

ST1000DM003 is not supported drive - see compatibility report, also, according to my experience, these drives have some firmware/compatibility problems.

Globally, Adaptec 5 Series are very problematic from the point of view of compatibility, in some cases workaround is to connect drives directly, without backplane, in some cases they stop failing when drives are switched to 1.5 gbps (drive jumpers).

Use drives from compatibility list and don't forget to upgrade drive firmwares.

p.s. you've got write cache enabled on drives, but disabled on controller.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .