7

After searching and reading as many posts, comments and discussions I could find I did not find one specific to my issue.

I have multiple AWS EC2 deployments with a single RDS all in the same Delivery zone us-west-2(c)

I am testing the load of the instances at a fraction of what I will be expecting very soon. The issue that concerns me is the performance while pushing updates. We will be frequently acquiring updates of 1,000 records at a time and will take and compare our data and update our data as appropriate. Consequently one read and one write per entry. It is not uncommon to have 100,000 updates come to us in an hour.

Currently I have a MySQL Database on an AWS t2.medium class RDS running 5 update processes at 22% CPU and less than 1GB Memory.

Even with these low numbers the read time to search the database of 106.3K records takes 2 to 3 full seconds and the write time is another 2 seconds.

I need some thoughts on how to improve these read/write times.

Additional info: I have a replica instance running as well. The CMS driven sites (100 and growing daily) connect to the replica instance for their content.

Thanks!

Improve this question
4
  • If it takes 3 seconds to search the database I wonder if you've optimised your indexes and queries. A 2 second write time is also super slow, unless you have some insane triggers or indexes. T2 is burstable, it could be that you've run out of CPU credits and you're using only 10% of a core. Try a c4.large, it's twice the price but better suited for what you're doing. Maybe you could describe your application, sounds like a batch processing candidate.
    – Tim
    Apr 16, 2016 at 9:04
  • @Tim, Thank you for the ideas. There are essentially two applications. 1) This issues application maintains up-to-date copy of real estate IDX records. Each time that a listing is updated we must also update our records. We maintain our own copy for numerous reasons. I am not sure exactly how batch processing would help in this instance but I will look into it. 2) The replica is for all the Internet visitors. That application is read only of the replica based upon their dynamic on-demand criteria.
    – Steven K7FAQ
    Apr 16, 2016 at 14:50
  • I did upgrade the instance from a T2 Micro after seeing bottlenecks. I am thinking this is a tuning issue since I had better results on a single cpu 2GB VM node at a different provider. So far the CPU performance of this T2 Medium has not increased beyond 30% overnight.
    – Steven K7FAQ
    Apr 16, 2016 at 14:57
  • 1
    Ok, interesting application. Remember with a t2.medium you only get 40% of a core, and once you run out of credits you're throttled, which is why a different instance type with constant CPU access could help. Strongly suggest you try the c4.large as a benchmark - spot instances are so cheap they essentially make testing free.
    – Tim
    Apr 16, 2016 at 18:02

3 Answers 3

Reset to default
6

I'm going to turn my comments above into an answer.

T2 instances only get a fraction of a CPU - 10% for a t2.micro, 40% for a t2.medium. You get CPU credits that build up, but once you use them you get throttled for CPU. You also get lower network and IO performance as you have to share physical machine resources.

What I suspect is happening is your testing has run out of CPU credits so you're being throttled. You can monitor CPU credits in CloudWatch, this will tell you whether or not my guess is correct.

Whether I'm right or not, t2 instances aren't suitable for systems that get work coming in constantly. A C4 or other general instance would probably be your best bet. You can monitor your database memory and CPU usage in CloudWatch to help work out what database instance size you need.

Improve this answer
2
  • I upgraded to m4.large and still maxing out the CPU. Am going to look hard at optimization or another storage strategy.
    – Steven K7FAQ
    Apr 18, 2016 at 19:21
  • Optimization is often very worthwhile, indexes and such are an easy place to start. m4.large is only two cores, for a busy database you may benefit from 4 or 8 cores.
    – Tim
    Apr 18, 2016 at 20:22
4

Take note of the disk bottleneck : Disk IOPS. RDS will throttle IO request if it is not provisioned. There is 3iops/GB given to SSD. So if you have 100GB allocated for SSD, you can max up 3x100 = 300 iops block.
If you pay for provisioned SSD, you can provision iops more than that.

While for standard(magentic) storage, additional per millions iops charge count, and the burst is 40-200 iops.

Please go check your RDS io log on the activities to determine better disk io strategy. More info reference : https://stackoverflow.com/questions/18777070/aws-rds-provisioned-iops-really-worth-it

Improve this answer
1
  • Issue is with number of transactions and connections to the RDS instance. Thank you for the additional info.
    – Steven K7FAQ
    Apr 18, 2016 at 19:20
1

I ended up creating a c4.xlarge EC2 and having it operating as my MariaDB 10 server. Due to the large amount of traffic currently and a major increase coming with new marketing campaigns I could not depend on RDS without spending several hundreds per month. And I found numerous complaints about AWS RDS with similar issues as mine.

Now the system is able to maintain two update workers simultaneously while visitors are able to get the dynamic content without a decrease in performance.

Thank you everyone for your thoughts and comments!

  • Steven
Improve this answer
1
  • Interesting. Did you work out the bottleneck, CPU or IO? Did you look at IO on your RDS instance? Limited IOPS could be the issue. RDS is meant to be very scalable. Did you need to change your application?
    – Tim
    May 8, 2016 at 21:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .