Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What are the best tools for troubleshooting SAN performance bottlenecks?

share|improve this question
    
Have you any particular vendors in mind for the various components? –  Chopper3 Jun 2 '09 at 21:57

9 Answers 9

A lot depends on the hardware you're playing with. Bottlenecks can come from a variety of sources:

  • Host based bottlenecks Sometimes, a server just can't shovel I/O blocks fast enough. For that you'll need to use whatever performance metrics your operating system (or application) have to diagnose.
  • Fabric based bottlenecks Brocade switches have performance metrics in handy charts, as well as actual numbers. Following these can illuminate where you're running into issues, perhaps with saturated ISLs. SMI-S should help you here, if you have the ability to use it.
  • Array bottlenecks These can come in a variety of flavors, like saturated controllers, overworked disk groups, and the like. Like the switches, newer arrays should support SMI-S for tracking things down.
share|improve this answer
    
This looks like a good starting point. I will have to research this further to see if I can find appropriate tools for measuring this which was part of what I was hoping to find by asking this question. So far it appears for windows-centric shops the responses have pointed to PerfMon, PAL, SCOM, and whatever our vendor-specific tools are for our SAN. –  Registered User Jun 16 '09 at 15:20

Sorry this is so windows-centric, but the PAL (Performance Analysis of Logs) tool - http://www.codeplex.com/PAL is useful for identifying problems with SAN setups, though you may have to pull a fairly long time period of .blg performance counter logs. Hope this helps.

share|improve this answer
    
I heard about this on RunAsRadio and used this as a starting point. This led me to focus on monitoring Disk Queue Length and I/O average waits. I haven't found a way to scale PAL beyond a few servers, though, in a meaningful way. SCOM seems to be doing the trick a bit better at the moment. –  Registered User Jun 16 '09 at 15:17

Your choice of tool depends on your hardware platform. In any case bottlenecks will manifest in one of 3 points in your architecture:

  1. Host
  2. Switch Fabric
  3. Storage Array

You will need a tool (or tools) that have the capability to monitor each of these components. You might want to adopt a best of breed strategy and use 3 different tools or you might prefer a Lord of the Rings approach and select a single tool "to rule them all." Whatever works for you. Start by contacting your vendor(s) and see which tools are available for your devices. You should be able to start gathering metrics at each of these points in your I/O chain which will let you identify where you need to be focusing your effort.

share|improve this answer

Monitor disk queue length on servers:

  • perfmon/scom on Windows
  • sar on unix
  • Virtual Center/ esxtop on VMware
share|improve this answer

If you want an all-in-one enterprisey solution, take a look at TPC for Disk/Fabric from IBM. You can monitor any components of your SAN (that support SMI-S as well as other standards) from one interface and be able to view or query historical data.

If this isn't an option, you can query the various SAN devices for their statistics and setup some sort of RRD monitoring to graph the performance and identify the bottlenecks.

Most disk subsystems and switches have some sort of built-in performance monitoring in the form of live graphs - try looking at those as well.

(disclaimer: my company sells TPC)

share|improve this answer

You can use Storage Essentials from HP. They have a performance module, the software can analyze from server to SAN. The analyze on the server side work only with HP server I think.

share|improve this answer

Good Question,

For EMC's Major Storage platforms: Workload Analyzer works for EMC Symmetrix. This tool is very robust and part of the EMC ControlCenter family. For CLARiiON and Celerra arrays, Navi Analyzer can provide similar performance data. This tool is enabled at the array level. For SAN Performance, there are tools from Brocade and Cisco that can help (fabric manager or Connetrix Manager/DFCM). Also, if remote replication is involved, networking tools may be required. Feel free to contact an EMC Sales Specialist or EMC Services Specialist if you need detailed guidance in the SAN Space.

share|improve this answer

Collect data. All of it. All the time :)

To chase bottlenecks, quite often you need to be able to see the individual queuedepth and latency of the exchanges (broken down by Initiator, Target, IT, or ITL), and B2Binfo and link events (LR, C3D) on all switch ports SAN-wide.

I work with different companies each day. I tend to find situations such as an old server with huge queuedepth, large exchanges, a few CRCs to weaken the links a bit, and bursty load so that the SAN seems to work find except those rare cases where it "browns out". By checking san-wide for indications of slowdown, and "zooming into" those affected links, seeing whether demand is excessive, or a correlation between that server's utilization, B2B info, and/or other data, we can see a few most-likely suspects.

We use a specific tool to collect this data and show it, but however you can collect latency, demand, and events from these different sources should work for you. I don't want to jump on a 5-yr-old unanswered question with an advertisement. If you want details and name-dropping, comment back.

The key is that you won't know what data to look at until after it happens. So collect it all. The more you can research after-the-fact, the better chance you have to get an idea of what happened, and how to avoid recurrence.

share|improve this answer

Usually if there is a SAN performance issue, the realistic solution is to solve the political problems that forced you to buy a hard-to-diagnose and maintain SAN in the first place. Replace the SAN and host with an HP DL380 G5 with two controllers with 512MB cache and 4*146GB 15KRPM drives per controller in a RAID 1+0 (or perhaps a RAID 5 per controller). Stripe the two RAID groups. Add four 320GB FusionIO PCI-E SSD cards, RAID 1+0'd in software.

The overall solution is faster than any SAN, and gives you more storage than most systems require.

You don't have controller level redundancy in this system. Buy two identical servers and mirror your databases in software, or mirror between the two controllers in software for a solution similar (well, greater) in reliability to the SAN.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.