We have 3 hosts which are under oracle RAC cluster, recently we have seen lot of multipath path missing on all the 3 hosts and sometimes these hosts reboot automatically.
current scenario is on host 1 all the paths are showing active and ready & rest 2 nodes its showing failed.
what can be the issue here? should i manually rescan the luns? does re-scanning the luns disrupt any service? This is RHEL5 box and some are ASM disks.
Any help would be appreciated. Storage guys said everything is ok at their end.
Host1
mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 4:0:1:94 sdhh 133:112 [active][ready]
\_ 3:0:1:94 sdhi 133:128 [active][ready]
host2:
mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 3:0:1:94 sdhh 133:112 [failed][faulty]
\_ 4:0:1:94 sdhi 133:128 [active][ready]
Host3
mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 3:0:1:94 sdhh 133:112 [failed][faulty]
\_ 4:0:1:94 sdhi 133:128 [active][ready]
F--From the vmcore of host 1 which rebooted on its own
sd 4:0:0:17: Unhandled error code
sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK sd 4:0:0:17: Unhandled error code sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK sd 4:0:0:17: Unhandled error code sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK qla2xxx 0000:06:00.0: Mailbox command timeout occured, cmd=0x54 mb[0]=0x54. Issuing ISP abort.
qla2xxx 0000:06:00.0: Performing ISP error recovery - ha= ffff81127ffb44f8.
qla2xxx 0000:06:00.0: LIP reset occured (f7f7).
qla2xxx 0000:06:00.0: LOOP UP detected (4 Gbps).
qla2xxx 0000:06:00.0: scsi(4:1:138): Abort command issued -- 0 13ce28adc 2002.
sd 4:0:1:8: timing out command, waited 60s
device-mapper: multipath: Failing path 133:192.
SysRq : Trigger a crashdump