How do I spec new storage performance requirements based on current data?

Question

I've asked a lot of questions about this lately, and I think I'll just quit beating around the bush...

Let's say I've got a storage system that is being used for a variety of applications with different I/O patterns. I've been collecting performance statistics on the system:

• transfers/sec
• sec/trasnfer
• bytes/transfer
• bytes/sec
• %idle

and I've calculated 95th percentile, average, and median for each. I also know what my average read / write ratio is.

I understand how to calculate potential average IOPS and throughput for a new system given disk, array, and average workload parameters.

I'm struggling to put this all together. Currently observed statistics are limited by the existing system which is struggling to keep up. Thus I might know that I require X IOPS, but this value may be low because of the current disk bottleneck, etc. (I know it's overworked because I'm seeing constant high disk usage and many multi-second periods of very high transfer times)

To be frank, I'm not doing anything hardcore and I can pretty much just buy some faster disks and configure my arrays better and it'll work out. But I'd like to understand how I might take a more formal approach to justifying an expense and to not over-buy.

Answer 1

I don't think you have enough data... what you also need to know is the utilisation of the other parts of the system that might become the bottleneck if the IO speeds up, so you can estimate how far you have to go on the IO before it becomes CPU, bus or network limited.

Some definition of 'fast enough' would help too. But it sounds like you want the long periods of waiting for IO to go away. Depending on exactly what you're doing, you may just not have enough memory to cache it properly.

Answer 2

I agree with Andrew McGregor's answer in principle, but the reality is that you probably don't have the luxury to perform any kind of benchmarking to see where the next bottlenecks are after you unclog the disk bottleneck. In a perfect world you'd either (a) have access to a faster disk subsystem in a "demo" capacity to play around with running your current workload on it, or (b) you could export trace data captured from the current environment and run it thru a "magic" mathematical model of your application software to plot the next bottlenecks.

It's not likely you can get demo hardware, and no mathematical models exist for "playing back" a live trace of your Exchange workloads (I read your other questions). Discovering what your next bottleneck will be in any objective way is going to be very difficult, at best. With that in mind, I'd do what you know will improve performance and plan for higher performance disk. There will be another bottleneck beyond that, but unless you can find a financially feasible or realistically possible way to predict it, you're really just guessing.

(I'm marking this "community wiki" becuse I'd really have rather left it as a comment on Andrew's answer but, obviously, one can't post a comment this long...)

Answer 3

Be sure to do what I call theorectical drive math to get your max IOPS and reduce by 20% or more based on real world usage. This varies by manufacturer, disk speed, etc. Then compare to your new drive configuration and make sure your peak loads fit well under this limit.