up vote 4 down vote favorite
3
share [g+] share [fb]

What are the best tools for troubleshooting SAN performance bottlenecks?

link|improve this question
Have you any particular vendors in mind for the various components? – Chopper3 Jun 2 '09 at 21:57
feedback

8 Answers

up vote 5 down vote

A lot depends on the hardware you're playing with. Bottlenecks can come from a variety of sources:

  • Host based bottlenecks Sometimes, a server just can't shovel I/O blocks fast enough. For that you'll need to use whatever performance metrics your operating system (or application) have to diagnose.
  • Fabric based bottlenecks Brocade switches have performance metrics in handy charts, as well as actual numbers. Following these can illuminate where you're running into issues, perhaps with saturated ISLs. SMI-S should help you here, if you have the ability to use it.
  • Array bottlenecks These can come in a variety of flavors, like saturated controllers, overworked disk groups, and the like. Like the switches, newer arrays should support SMI-S for tracking things down.
link|improve this answer
This looks like a good starting point. I will have to research this further to see if I can find appropriate tools for measuring this which was part of what I was hoping to find by asking this question. So far it appears for windows-centric shops the responses have pointed to PerfMon, PAL, SCOM, and whatever our vendor-specific tools are for our SAN. – InnerJoin Jun 16 '09 at 15:20
feedback
up vote 2 down vote

Sorry this is so windows-centric, but the PAL (Performance Analysis of Logs) tool - http://www.codeplex.com/PAL is useful for identifying problems with SAN setups, though you may have to pull a fairly long time period of .blg performance counter logs. Hope this helps.

link|improve this answer
I heard about this on RunAsRadio and used this as a starting point. This led me to focus on monitoring Disk Queue Length and I/O average waits. I haven't found a way to scale PAL beyond a few servers, though, in a meaningful way. SCOM seems to be doing the trick a bit better at the moment. – InnerJoin Jun 16 '09 at 15:17
feedback
up vote 2 down vote

Your choice of tool depends on your hardware platform. In any case bottlenecks will manifest in one of 3 points in your architecture:

  1. Host
  2. Switch Fabric
  3. Storage Array

You will need a tool (or tools) that have the capability to monitor each of these components. You might want to adopt a best of breed strategy and use 3 different tools or you might prefer a Lord of the Rings approach and select a single tool "to rule them all." Whatever works for you. Start by contacting your vendor(s) and see which tools are available for your devices. You should be able to start gathering metrics at each of these points in your I/O chain which will let you identify where you need to be focusing your effort.

link|improve this answer
feedback
up vote 2 down vote

Monitor disk queue length on servers:

  • perfmon/scom on Windows
  • sar on unix
  • Virtual Center/ esxtop on VMware
link|improve this answer
feedback
up vote 1 down vote

If you want an all-in-one enterprisey solution, take a look at TPC for Disk/Fabric from IBM. You can monitor any components of your SAN (that support SMI-S as well as other standards) from one interface and be able to view or query historical data.

If this isn't an option, you can query the various SAN devices for their statistics and setup some sort of RRD monitoring to graph the performance and identify the bottlenecks.

Most disk subsystems and switches have some sort of built-in performance monitoring in the form of live graphs - try looking at those as well.

(disclaimer: my company sells TPC)

link|improve this answer
feedback
up vote 1 down vote

You can use Storage Essentials from HP. They have a performance module, the software can analyze from server to SAN. The analyze on the server side work only with HP server I think.

link|improve this answer
feedback
up vote 1 down vote

Good Question,

For EMC's Major Storage platforms: Workload Analyzer works for EMC Symmetrix. This tool is very robust and part of the EMC ControlCenter family. For CLARiiON and Celerra arrays, Navi Analyzer can provide similar performance data. This tool is enabled at the array level. For SAN Performance, there are tools from Brocade and Cisco that can help (fabric manager or Connetrix Manager/DFCM). Also, if remote replication is involved, networking tools may be required. Feel free to contact an EMC Sales Specialist or EMC Services Specialist if you need detailed guidance in the SAN Space.

link|improve this answer
feedback
up vote -3 down vote

Usually if there is a SAN performance issue, the realistic solution is to solve the political problems that forced you to buy a hard-to-diagnose and maintain SAN in the first place. Replace the SAN and host with an HP DL380 G5 with two controllers with 512MB cache and 4*146GB 15KRPM drives per controller in a RAID 1+0 (or perhaps a RAID 5 per controller). Stripe the two RAID groups. Add four 320GB FusionIO PCI-E SSD cards, RAID 1+0'd in software.

The overall solution is faster than any SAN, and gives you more storage than most systems require.

You don't have controller level redundancy in this system. Buy two identical servers and mirror your databases in software, or mirror between the two controllers in software for a solution similar (well, greater) in reliability to the SAN.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.