1

We are currently utilizing GFS2 to share a SAN LUN between 3 servers. However due to a feature problem with vendor software we are using, we currently have the volume unmounted on two of the boxes, and are instead exporting the GFS2 filesystem via NFS from the first one (the software requires some weird locking mechanics that GFS2 doesn't support).

As of this morning, NFS was no longer able to read/write to the volume from any of the servers, including the NFS server. I then tried checking the normal mount (the directory that is exported on the NFS server) and I received a weird input/output error just trying to CD into it. When I tried running multipath, I got a DM error, however multipath -l worked just fine. I tried unmounting the GFS2 volume, and the CLI hung. I ran init 0 which killed most services, but then the shutdown appeared to have been hung. I logged in via out of band access (hp ILO) and saw that the shutdown was hung trying to unmount GFS2 volumes.

My main priority was getting the box back online so after about 5 minutes of waiting I did a hard reset. I am now trying to figure out what went wrong. What are the correct logs to investigate? I've never run into SAN issues like this before. The SAN is connected via 2 fibre connections. Any help would really be appreciated. Everything appears to be up and functional now.

1
  • Since you are asking, syslog (/var/log/messages or where syslog is directed) probably has nothing on it. Could it be that syslog files are on the same troubled volume? Sep 28, 2012 at 16:23

1 Answer 1

1

There are no special logs on Linux - use dmesg/syslog normally.

For SAN switches, way of accessing logs is vendor-specific (on Brocade -> ssh admin@x -> errdump ).

For SAN disk storage, way of accessing logs is vendor-specific (on LSI -> GUI -> Event log -> turn off show_critical_only -> refresh).

First of all, verify you have ntp (or other time synchronization) for both SAN switches and SAN storage, or you'll never know which error is a cause and which is an effect.

Check the most probable cause, verify how your multipath/GFS2 reacts to accidental FC cable removal.

Check the second probable cause, verify how your multipath/GFS2 reacts to accidental outage of SAN disk array controller.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .