SQLIO and SAN performance problems

Question

I've had a feeling our SAN is performing badly, so I ran SQLIO with the help from Mr Ozar's blog post.
My first problem is that I couldn't create a 20GB test file.

This is the result

using 64KB random IOs
enabling multiple I/Os per thread with 8 outstanding
buffering set to use hardware disk cache (but not file cache)
using current size: 2011 MB for file: M:\TestFile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  3873.51
MBs/sec:   242.09
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 16
Max_Latency(ms): 1465
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  3  2  3  3  4  5  5  5  5  5  5  5  5  5  4  4  4  3  3  2  2  2  2  1 13


using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
buffering set to use hardware disk cache (but not file cache)
using current size: 2011 MB for file: M:\TestFile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  5279.99
MBs/sec:   329.99
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 11
Max_Latency(ms): 267
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  1  1  3  5 11  8  6  5  4  4  5  7  7  6  5  4  3  3  2  2  1  1  1  1  4

What can be said about the metrics from the SAN?
Should I consider another provider?
The SAN is said to be "state of art HP SAN".

Answer 1

If your SAN really is new state of the art, you will have automatic tiering between disks and SSD. The performance on any work done to SSDs will be much faster than to the disk. Also, you'll have a lot of cache, which means that writes won't even touch the disks/ssd usually. I assume you're talking about the p9500, however if this assumption is wrong, you should really edit your question to have more information about your configuration.

If you're testing reads (which is more typical), then you will want to try and isolate the space based on the pool composition. Put the LDEV your file lives on in a pool with more spindles or a higher percentage of SSD, and it should go faster.

All that said, the only metric that matters is latency when you're firing on all cylinders. Any machine can go fast when it's underutilized, but take a look once you've got a ton of simultaneous applications running on it. My recommendation is to simulate a lot more IO over a lot more servers, and see how much you can generate before the latency goes over 10ms. When generating load, you want to have the same read to write and sequential to random ratios as your real workload. Failing that, you want 70% reads to 30% writes, and something like 20% sequential to 80% random.

Answer 2

What can be said about the metrics from the SAN?

Nothing.

using current size: 2011 MB

This is no a realistic test. Likely the SAN has a write back cache and your file is busy anough and small enough to be fully cached, block by block. So, you dont really measure anything worthwhile.

This is like measuring top speed of a car by the acceleration in the first 10 meters. Won#t work.

Your test file must be realistic given the hardware and workload. This means nless you typciall have a non busy SAN and a small data set, you msut run that on a prouction size test file - at least one which is SIGNFICANTLY larger than the caches involved. So, on a 32gb SAN cache (seen on HP EVA) alone for you it is good to have a 128gb test file.

IOs/sec: 3873.51

This is either a HIGH END SAN or a decent cache. Given 450 IOPS per disc roughly that would mean you run 7-8 discs independently. Can be - or can be as I said you run from cache.