2

Recently we [finally] migrated all of our production hardware from the deprecated default networking into VPCs including EC2 and RDS nodes. Everything went pretty well except when we started to run performance tests in production and noticed quite significant degradation compared to the old configuration: at least 30-40% penalty.

After scratching our heads a good bit, comparing software versions and configuration, we decided to try to regenerate our RDS instances. Initially the goal was to migrate our primary databases into a different AZ to compare apples to apples. When the migration finished we found that performance returned to normal however I seriously doubt that AZ was the source of our pain. I just can't believe that a couple milliseconds (at most) of latency caused that many problems, especially since different query client AZs seems to have no effect on performance.

I suspect that we either got some bad m3.large RDS hardware or were on an instance with a particularly noisy neighbor.

  • Anyone see these sorts of per-instance performance differences before?
  • Any other debugging that we could have done to better diagnose this?
  • In reporting this problem to AWS support, the tier 1 guy is saying that they don't know the hardware that I was on internally. Can this really be true? I'm not asking for any identification but I'd like to think that they could flag the hardware for the physical team. Anything I could have done differently?

Thanks for any feedback.

1

You use the classic IT solution - turn it off then on again. This almost certainly moves you to new hardware. If you're concerned about downtime you may have to set up a mirror or read replica first.

You don't report it, you just move on.

  • Yeah that's what we did although with RDS, the turn it off/on takes longer and is a bit more involved than with an EC2 instance. I was more surprised that it happened and that AWS support (which is usually stellar) was not interested in our experience. – Gray Mar 3 '16 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.