I've been testing AWS EC2 with EBS optimized instances using EBS provisioned IOPS volumes in the hopes of finding out just how much better they perform in reality. I'm confused by the results and was hoping for some advice from the folks here.
I've been testing on an m1.large (7.5GB RAM, EBS optimized, 8G non-PIO EBS root, 4 ECUs across 2 vcores), doing my EBS piops volume tests in a second volume I've attached to the machine. The machine runs Debian 6.0.6 (official image) with PostgreSQL 9.2.2.
Everything I've tested has had significantly worse performance when I use a single provisioned IOPS volume than when I use a regular EC2 volume.
For example, comparing a normal EBS volume to a 600 iops PIOPS volume:
pg_test_fsyncgets over 1000 fdatasyncs/second and over 450 fsyncs/second on normal ebs, while it gets almost exactly 600 fdatasyncs and 151 fsyncs on a piops volume.
pgbench with the same settings gets 900 tps on the normal EBS volume vs 200 tps on the piops volume.
Database creation takes about 4x as long on the piops volume vs the normal volume
Bonnie++ results on the piops volume are dramatically worse than the ordinary EBS volume.
The way that the PIOPS volume never seems to exceed its piops setting suggests that the iops value for piops volumes is a maximum after which performance is throttled, not just a guaranteed minimum. In other words, piops volumes can be slower than regular EBS volumes.
I haven't found anything in the documentation about this yet, but it's pretty consistent and the way I get exactly 600 fdatasyncs/second or 150 fsyncs/second on a 600 iops volume strongly suggests throttling.
Are EBS PIOPS volumes throttled to a maximum iops, not just guaranteed a minimum?
Update: Also asked on AWS forums.