physical vs logical disk transactions

Question

I'm trying to analyze server requirements for a set of virtual machines. We're using LVM on them, so iostat shows numbers for both the logical volume and the physical disk, and they often don't match at the transaction level, and I'm trying to decide which metric matters: tps or total blocks read+written, and for the physical drive or lv? My guess is physical drive total blocks, but tps will take into account seeks that blocks won't, which will be important as the key decision is whether or not we can get away with running these on an array of spinning disks instead of the flash drives they currently are (we're about to double the storage).

"iostat 10" output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       2.38    1.35    1.40    0.69    0.00   94.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb             225.48      2922.30      3511.65 79987371570 96118589648
sda               1.92        11.86        39.47  324647920 1080458180
dm-0              5.00        10.57        38.62  289229114 1057115528
dm-1              0.27         1.29         0.85   35343720   23342584
dm-3            414.45      1501.13      3393.89 41087911850 92895186296

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       7.74    0.06    4.88    0.63    0.00   86.69

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb             259.54       527.47     28098.30       5280     281264
sda               1.20         0.00        20.78          0        208
dm-0              2.60         0.00        20.78          0        208
dm-1              0.00         0.00         0.00          0          0
dm-3           3515.48       527.47     28097.50       5280     281256

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       3.00    0.04    0.56    0.43    0.00   95.98

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb              96.20       893.69      2175.78       8928      21736
sda               0.70         0.80        19.22          8        192
dm-0              2.40         0.00        19.22          0        192
dm-1              0.10         0.80         0.00          8          0
dm-3            267.77       893.69      2175.78       8928      21736

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       2.80    0.01    0.64    0.73    0.00   95.83

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb             110.50      1284.80      2376.00      12848      23760
sda               0.80         0.00        15.20          0        152
dm-0              1.90         0.00        15.20          0        152
dm-1              0.00         0.00         0.00          0          0
dm-3            272.30      1284.80      2376.00      12848      23760

Answer 1

This answer says that linux kernel optimizes transactions before they go to physical volume (PV) level, hence a different value of tps.

Storage performance requirements guidelines (for most systems - not for all!):

• peaks should be identified at the scale of a month or even a year; tools like iostat, sar need some scripting to handle that; Zabbix graphs would work better
• random IOps rate is usually crucial; it's approximately tps, when you look at PVs (sda, not dm-3)
• read throughput (Blk_read/s) is mostly relevant for full backup
• write throughput (Blk_wrtn/s) is mostly relevant for restore from backup; usually invisible in gathering tools

Answer 2

The sum of TPS of the physical volumes approximates the IOPS. Random IOPS, as each VM is doing IO to different parts of the underlying storage. You might get 100 random IOPS out of a spindle, with lower cost per GB. Much more out of solid state storage, with lower cost per IOPS.

What you can get away with depends on your performance and capacity needs are, constrained by things like how many disks fit in your array and how much you want to spend powering it.

If you want, say, sub 1 ms response times, that implies solid state. Even the best spindle arrays have difficulty getting low single digit ms response times for a cache miss.