I am currently doing research to determine the consolidation ratio my company could expect should we start using a virtualization platform. I find myself continually running into a dead end when researching how to translate observed performance (weeks of perfmon data) to hdd array requirements for a virtualization server. I am familiar with the concept of IOPs, but they seem to be an overly simplistic measurement that fails to take into account cache, write combining, etc. Is there a seminal work on storage array performance analysis that I'm missing? This seems like an area where hearsay and 'black magic' have taken over for cold, hard fact.
Honestly it's a whole lot of good guessing and gut feelings. You can start with basic IOPs, but then you have to take into account applications that are dependent on each-other and will cause concurrent IO, where other applications are independent and less likely to be concurrent. This usually leads to calculations so complex you end up rounding and fudging until it sounds reasonable.
When in doubt, go for higher performance, users never complain about an application responding too fast. Just be sure to stay within your budget, or management will be complaining about that.
Do take a look at the numbers, don't just outright guess. What many of the numbers aren't going to show is what the cache hit rate is; and if you go with a SAN, it'll likely have a larger cache (where the cache hit rate might be drastically different).
I would suggest that it greatly overcomplicates your provisioning if you try to guess cache hit rates and other implementation details. Since you are already so far ahead of the game by actually taking some perf measurements that focus on IOPS ahead of time I'd add up all your write IOPS to get a ballpark of what you need.
If you can you can do some disk array testing to confirm for yourself the kind of IO you're going to be able to get out x number of disk spindles. My rule of thumb is 150, 200, 250 IOPS for 7.2k, 10k, 15k disks respectively and for RAID 10 each mirrored pair added to the stripe set adds that many IOPS to the pool. So striping over two mirrors of 7.2k disks would be about 300 IOPS, or 500 IOPS if you use 15k disks.
Presuming you have a larger environment to consolidate you are probably trying to figure out how many shelves of disk you need. If you are using 15 disk shelves then 2 shelves get you 14 mirror pairs to stripe over and 2 hotspares so up to 3500 IOPS. When you add in caching and IO ordering and scheduling you may get higher performance than that but you shouldn't get lower performance.
YMMV so it's good to do some testing with large test sets of random IO to validate that your equipment performs what the model predicts or you can adjust the model to match observed behavior.