What are the best tools for troubleshooting SAN performance bottlenecks?
A lot depends on the hardware you're playing with. Bottlenecks can come from a variety of sources:
- Host based bottlenecks Sometimes, a server just can't shovel I/O blocks fast enough. For that you'll need to use whatever performance metrics your operating system (or application) have to diagnose.
- Fabric based bottlenecks Brocade switches have performance metrics in handy charts, as well as actual numbers. Following these can illuminate where you're running into issues, perhaps with saturated ISLs. SMI-S should help you here, if you have the ability to use it.
- Array bottlenecks These can come in a variety of flavors, like saturated controllers, overworked disk groups, and the like. Like the switches, newer arrays should support SMI-S for tracking things down.
Sorry this is so windows-centric, but the PAL (Performance Analysis of Logs) tool - http://www.codeplex.com/PAL is useful for identifying problems with SAN setups, though you may have to pull a fairly long time period of .blg performance counter logs. Hope this helps.
Your choice of tool depends on your hardware platform. In any case bottlenecks will manifest in one of 3 points in your architecture:
- Switch Fabric
- Storage Array
You will need a tool (or tools) that have the capability to monitor each of these components. You might want to adopt a best of breed strategy and use 3 different tools or you might prefer a Lord of the Rings approach and select a single tool "to rule them all." Whatever works for you. Start by contacting your vendor(s) and see which tools are available for your devices. You should be able to start gathering metrics at each of these points in your I/O chain which will let you identify where you need to be focusing your effort.
If you want an all-in-one enterprisey solution, take a look at TPC for Disk/Fabric from IBM. You can monitor any components of your SAN (that support SMI-S as well as other standards) from one interface and be able to view or query historical data.
If this isn't an option, you can query the various SAN devices for their statistics and setup some sort of RRD monitoring to graph the performance and identify the bottlenecks.
Most disk subsystems and switches have some sort of built-in performance monitoring in the form of live graphs - try looking at those as well.
(disclaimer: my company sells TPC)
For EMC's Major Storage platforms: Workload Analyzer works for EMC Symmetrix. This tool is very robust and part of the EMC ControlCenter family. For CLARiiON and Celerra arrays, Navi Analyzer can provide similar performance data. This tool is enabled at the array level. For SAN Performance, there are tools from Brocade and Cisco that can help (fabric manager or Connetrix Manager/DFCM). Also, if remote replication is involved, networking tools may be required. Feel free to contact an EMC Sales Specialist or EMC Services Specialist if you need detailed guidance in the SAN Space.
Collect data. All of it. All the time :)
To chase bottlenecks, quite often you need to be able to see the individual queuedepth and latency of the exchanges (broken down by Initiator, Target, IT, or ITL), and B2Binfo and link events (LR, C3D) on all switch ports SAN-wide.
I work with different companies each day. I tend to find situations such as an old server with huge queuedepth, large exchanges, a few CRCs to weaken the links a bit, and bursty load so that the SAN seems to work find except those rare cases where it "browns out". By checking san-wide for indications of slowdown, and "zooming into" those affected links, seeing whether demand is excessive, or a correlation between that server's utilization, B2B info, and/or other data, we can see a few most-likely suspects.
We use a specific tool to collect this data and show it, but however you can collect latency, demand, and events from these different sources should work for you. I don't want to jump on a 5-yr-old unanswered question with an advertisement. If you want details and name-dropping, comment back.
The key is that you won't know what data to look at until after it happens. So collect it all. The more you can research after-the-fact, the better chance you have to get an idea of what happened, and how to avoid recurrence.
Usually if there is a SAN performance issue, the realistic solution is to solve the political problems that forced you to buy a hard-to-diagnose and maintain SAN in the first place. Replace the SAN and host with an HP DL380 G5 with two controllers with 512MB cache and 4*146GB 15KRPM drives per controller in a RAID 1+0 (or perhaps a RAID 5 per controller). Stripe the two RAID groups. Add four 320GB FusionIO PCI-E SSD cards, RAID 1+0'd in software.
The overall solution is faster than any SAN, and gives you more storage than most systems require.
You don't have controller level redundancy in this system. Buy two identical servers and mirror your databases in software, or mirror between the two controllers in software for a solution similar (well, greater) in reliability to the SAN.