Recently we [finally] migrated all of our production hardware from the deprecated default networking into VPCs including EC2 and RDS nodes. Everything went pretty well except when we started to run performance tests in production and noticed quite significant degradation compared to the old configuration: at least 30-40% penalty.
After scratching our heads a good bit, comparing software versions and configuration, we decided to try to regenerate our RDS instances. Initially the goal was to migrate our primary databases into a different AZ to compare apples to apples. When the migration finished we found that performance returned to normal however I seriously doubt that AZ was the source of our pain. I just can't believe that a couple milliseconds (at most) of latency caused that many problems, especially since different query client AZs seems to have no effect on performance.
I suspect that we either got some bad m3.large RDS hardware or were on an instance with a particularly noisy neighbor.
- Anyone see these sorts of per-instance performance differences before?
- Any other debugging that we could have done to better diagnose this?
- In reporting this problem to AWS support, the tier 1 guy is saying that they don't know the hardware that I was on internally. Can this really be true? I'm not asking for any identification but I'd like to think that they could flag the hardware for the physical team. Anything I could have done differently?
Thanks for any feedback.