I've got an interesting problem I'm hoping you all can help me with. I have an IBM 5014 (the LSI 9260-8i equivalent) running two RAID10 virtual drives. The first is 4 WD RE4's, each 2TB, for a 4TB total drive - let's call this VD1. The other is 4 WD RE4-GP's, each 2TB, for another 4TB total drive - let's call this VD0. In case it matters, The card runs in a Norco case w/ 3 fans (1 on each bank of 4 drives + 1 on the Gigabyte MB, 16GB RAM, and IBM card. There is an IBM5015 running 4 256GB SSD's in RAID10 as well). I'm virtualized using ESXi5.5 with a series of VM's. The 5014 card runs in passthrough mode to a WHS2011 host, while the 5015 contains the VM's themselves.
VD0 runs fine, and has no issues. It is my primary document storage.
VD1, however, which contains all my videos, periodically drops a drive, causing a degraded state, and then almost instantaneously (typically with the exact same time stamp, but occasionally on a 1 second lag) drops the remainder of the drives as well causing it to go offline.
The controller itself has been running fine for nearly 6 months, so while it could be controller related, that feels like it would cause problems with both virtual drives, not just one of them.
The challenge I have is that the drives don't consistently drop out (at least according to the log) in the same order - so I don't know which drive is causing the issue. I've included an excerpt from the log below. As you'll see, it is dropping the drives then re-adding them.
Any advice on how to troubleshoot which drive would be very welcome - I can't believe they all went bad together, nor can I believe how little information is contained in the MSM log itself.
Thank you all in advance!
Doug
ID = 248
SEQUENCE NUMBER = 382617
TIME = 07-07-2015 08:14:46
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 8
ID = 112
SEQUENCE NUMBER = 382616
TIME = 07-07-2015 08:14:46
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:1
ID = 248
SEQUENCE NUMBER = 382615
TIME = 07-07-2015 08:14:45
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 13
ID = 112
SEQUENCE NUMBER = 382614
TIME = 07-07-2015 08:14:45
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:3
ID = 248
SEQUENCE NUMBER = 382613
TIME = 07-07-2015 08:14:44
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 9
ID = 112
SEQUENCE NUMBER = 382612
TIME = 07-07-2015 08:14:44
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:0
ID = 248
SEQUENCE NUMBER = 382611
TIME = 07-07-2015 08:14:44
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 14
ID = 112
SEQUENCE NUMBER = 382610
TIME = 07-07-2015 08:14:44
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:2
ID = 247
SEQUENCE NUMBER = 382609
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 14
ID = 91
SEQUENCE NUMBER = 382608
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:2
ID = 247
SEQUENCE NUMBER = 382607
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 9
ID = 91
SEQUENCE NUMBER = 382606
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:0
ID = 247
SEQUENCE NUMBER = 382605
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8
ID = 91
SEQUENCE NUMBER = 382604
TIME = 07-07-2015 07:53:09
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:1
ID = 247
SEQUENCE NUMBER = 382603
TIME = 07-07-2015 07:53:04
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 13
ID = 91
SEQUENCE NUMBER = 382602
TIME = 07-07-2015 07:53:04
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:3
ID = 248
SEQUENCE NUMBER = 382601
TIME = 07-07-2015 07:52:44
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 9
ID = 112
SEQUENCE NUMBER = 382600
TIME = 07-07-2015 07:52:44
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:0
ID = 248
SEQUENCE NUMBER = 382599
TIME = 07-07-2015 07:52:42
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 13
ID = 112
SEQUENCE NUMBER = 382598
TIME = 07-07-2015 07:52:42
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:3
ID = 248
SEQUENCE NUMBER = 382597
TIME = 07-07-2015 07:52:41
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 8
ID = 112
SEQUENCE NUMBER = 382596
TIME = 07-07-2015 07:52:41
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:1
ID = 248
SEQUENCE NUMBER = 382595
TIME = 07-07-2015 07:52:40
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 14
ID = 112
SEQUENCE NUMBER = 382594
TIME = 07-07-2015 07:52:40
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:2
ID = 145
SEQUENCE NUMBER = 382593
TIME = 07-07-2015 07:10:59
LOCALIZED MESSAGE = Controller ID: 0 Battery temperature is high
ID = 149
SEQUENCE NUMBER = 382592
TIME = 07-07-2015 06:56:54
LOCALIZED MESSAGE = Controller ID: 0 Battery temperature is normal
ID = 247
SEQUENCE NUMBER = 382591
TIME = 07-07-2015 04:08:56
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 14
ID = 91
SEQUENCE NUMBER = 382590
TIME = 07-07-2015 04:08:56
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:2
ID = 247
SEQUENCE NUMBER = 382589
TIME = 07-07-2015 04:08:56
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 9
ID = 91
SEQUENCE NUMBER = 382588
TIME = 07-07-2015 04:08:56
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:0
ID = 247
SEQUENCE NUMBER = 382587
TIME = 07-07-2015 04:08:55
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8
ID = 91
SEQUENCE NUMBER = 382586
TIME = 07-07-2015 04:08:55
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:1
ID = 248
SEQUENCE NUMBER = 382585
TIME = 07-07-2015 04:08:49
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 8
ID = 112
SEQUENCE NUMBER = 382584
TIME = 07-07-2015 04:08:49
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:1
ID = 248
SEQUENCE NUMBER = 382583
TIME = 07-07-2015 04:08:47
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 9
ID = 112
SEQUENCE NUMBER = 382582
TIME = 07-07-2015 04:08:47
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:0
ID = 248
SEQUENCE NUMBER = 382581
TIME = 07-07-2015 04:08:47
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 14
ID = 112
SEQUENCE NUMBER = 382580
TIME = 07-07-2015 04:08:47
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:2
ID = 247
SEQUENCE NUMBER = 382579
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 14
ID = 91
SEQUENCE NUMBER = 382578
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:2
ID = 247
SEQUENCE NUMBER = 382577
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 13
ID = 91
SEQUENCE NUMBER = 382576
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:3
ID = 247
SEQUENCE NUMBER = 382575
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 8
ID = 91
SEQUENCE NUMBER = 382574
TIME = 07-07-2015 03:24:32
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:1
ID = 247
SEQUENCE NUMBER = 382573
TIME = 07-07-2015 03:24:27
LOCALIZED MESSAGE = Controller ID: 0 Device inserted Device Type: Disk Device Id: 9
ID = 91
SEQUENCE NUMBER = 382572
TIME = 07-07-2015 03:24:27
LOCALIZED MESSAGE = Controller ID: 0 PD inserted: -:-:0
ID = 248
SEQUENCE NUMBER = 382571
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 9
ID = 112
SEQUENCE NUMBER = 382570
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:0
ID = 248
SEQUENCE NUMBER = 382569
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 14
ID = 112
SEQUENCE NUMBER = 382568
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:2
ID = 248
SEQUENCE NUMBER = 382567
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 8
ID = 112
SEQUENCE NUMBER = 382566
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:1
ID = 248
SEQUENCE NUMBER = 382565
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 13
ID = 112
SEQUENCE NUMBER = 382564
TIME = 07-07-2015 03:23:36
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:3
ID = 139
SEQUENCE NUMBER = 382435
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 Deleted VD: 1
ID = 114
SEQUENCE NUMBER = 382434
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:0 Previous = Failed Current = Unconfigured Bad
ID = 114
SEQUENCE NUMBER = 382433
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:2 Previous = Failed Current = Unconfigured Bad
ID = 114
SEQUENCE NUMBER = 382432
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:1 Previous = Failed Current = Unconfigured Bad
ID = 114
SEQUENCE NUMBER = 382431
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:3 Previous = Failed Current = Unconfigured Bad
ID = 114
SEQUENCE NUMBER = 382430
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:0 Previous = Online Current = Failed
ID = 248
SEQUENCE NUMBER = 382429
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 9
ID = 112
SEQUENCE NUMBER = 382428
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:0
ID = 252
SEQUENCE NUMBER = 382427
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 VD is now OFFLINE VD 1
ID = 81
SEQUENCE NUMBER = 382426
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change on VD: 1 Previous = Degraded Current = Offline
ID = 114
SEQUENCE NUMBER = 382425
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:2 Previous = Online Current = Failed
ID = 248
SEQUENCE NUMBER = 382424
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 14
ID = 112
SEQUENCE NUMBER = 382423
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:2
ID = 114
SEQUENCE NUMBER = 382422
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:1 Previous = Online Current = Failed
ID = 248
SEQUENCE NUMBER = 382421
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 8
ID = 112
SEQUENCE NUMBER = 382420
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:1
ID = 251
SEQUENCE NUMBER = 382419
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 VD is now DEGRADED VD 1
ID = 81
SEQUENCE NUMBER = 382418
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change on VD: 1 Previous = Optimal Current = Degraded
ID = 114
SEQUENCE NUMBER = 382417
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 State change: PD = -:-:3 Previous = Online Current = Failed
ID = 248
SEQUENCE NUMBER = 382416
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 Device removed Device Type: Disk Device Id: 13
ID = 112
SEQUENCE NUMBER = 382415
TIME = 04-07-2015 08:27:32
LOCALIZED MESSAGE = Controller ID: 0 PD removed: -:-:3
