0

We have 3 hosts which are under oracle RAC cluster, recently we have seen lot of multipath path missing on all the 3 hosts and sometimes these hosts reboot automatically.

current scenario is on host 1 all the paths are showing active and ready & rest 2 nodes its showing failed.

what can be the issue here? should i manually rescan the luns? does re-scanning the luns disrupt any service? This is RHEL5 box and some are ASM disks.

Any help would be appreciated. Storage guys said everything is ok at their end.


Host1

mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 4:0:1:94  sdhh 133:112 [active][ready]
 \_ 3:0:1:94  sdhi 133:128 [active][ready]

host2:

mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:1:94  sdhh 133:112 [failed][faulty]
 \_ 4:0:1:94  sdhi 133:128 [active][ready]

Host3

mpath602 (360060e801606e900000106e900000dd3) dm-114 HITACHI,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:1:94  sdhh 133:112 [failed][faulty]
 \_ 4:0:1:94  sdhi 133:128 [active][ready]

F--From the vmcore of host 1 which rebooted on its own

sd 4:0:0:17: Unhandled error code
sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK sd 4:0:0:17: Unhandled error code sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK sd 4:0:0:17: Unhandled error code sd 4:0:0:17: SCSI error: return code = 0x00010000
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK qla2xxx 0000:06:00.0: Mailbox command timeout occured, cmd=0x54 mb[0]=0x54. Issuing ISP abort.
qla2xxx 0000:06:00.0: Performing ISP error recovery - ha= ffff81127ffb44f8.
qla2xxx 0000:06:00.0: LIP reset occured (f7f7).
qla2xxx 0000:06:00.0: LOOP UP detected (4 Gbps).
qla2xxx 0000:06:00.0: scsi(4:1:138): Abort command issued -- 0 13ce28adc 2002.
sd 4:0:1:8: timing out command, waited 60s
device-mapper: multipath: Failing path 133:192.
SysRq : Trigger a crashdump
  • Manually rescanning the LUNS when a LUN is in use will often lead to a crash. You're triggering the Loop Initiation Protocol (LIP) was is sorta like yanking the fibre cable out and sticking it back in. The LIP is probably what caused the crash in the logs above, btw. Do the "Storage guys" also manage your fabrics or just the storage units themselves? Because I'd be looking hard at the switches if I had multiple hosts with connectivity problems. – Brandon Xavier Mar 12 '17 at 13:33
  • Scanning the paths will freeze only the I/O for a few seconds. It will not crash anything, unless you have a buggy multipath daemon, or hit a device mapper bug. – Mircea Vutcovici Aug 23 '17 at 15:29
0

TL;DR If you do not care for the storage to be frozen for a few seconds on your machine, just run:

for i in /sys/class/scsi_host/host*; do echo "- - -" > $i/scan; done; for i in /sys/bus/scsi/devices/*:*:*:*/rescan;do echo 1 > $i;done; for i in /sys/class/fc_host/host*/issue_lip;do echo 1 >$i;done;for i in /sys/block/sd*/device/rescan;do echo 1 > $i;done

If you want to know what is happening, check the kernel log messages:

  • RHEL 6 or older, run: sudo less /var/log/messages and dmesg
  • RHEL 7 or newer, run: sudo journalctl -k

In the kernel messages search for the failed block device, like sdhh in your case. See when and why it failed.

Check if you can read from that sdhh failed path:

sudo dd if=/dev/sdhh bs=4k count=1|hexdump -C

If you have an I/O error, then I would check the physical path, the FC switch, the storage controllers. If you do not have an I/O error, it is safe to scan the SCSI bus, but it will freeze the I/O for a few seconds.

Update multipath package to a recent one. Upgrade to a recent distribution. RHEL5 is not supported and should not be used.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.