1

Let me first briefly describe our stack.

  • Single EC2 instance with 4 vCPU and 16 GB RAM running Ubuntu [m4.xlarge]
  • Single RDS instance with 2 vCPU and 8 GB RAM running MySQL 8 [db.m5.large]
  • We are using cloudfront to cache static assets
  • We are using Nginx as web server
  • It is backed by tomcat instances in which our Java web apps run
  • We have 3 primary apps each run in their own tomcat instance (Storefront website, Admin Backend, Seller Platform)
  • Storefront and Seller Platform talks with Admin backend by use of REST APIs for important events, rest they have direct integration with RDS for DB

  • All our apps are built in Java, using Spring & Hibernate as frameworks

  • We are using tomcat database connection pooling in all apps
  • We are using in memory cache for complete catalog and other things which do not frequently change to avoid hits on DB

What Happened:

On usual days, we have around 125 users on website as observed by Google analytics and get around 1 order in every 2 mintue. But one day, while we were running some aggressive campaigns we had around 4000 users on website at a time and about 18 orders were getting placed on website every minute. Because we promise same day delivery, our backend and seller platform was equally working on full capacity to serve those orders.

This high load was for 4 hours on our system and it resulted in as many as 20 downtimes, collectively making it around 50 minutes of downtime. It was primarily due to database issues.

Issue Observed

  • Worst hit was our Backend application. After about every 15 minute it started throwing error that it is unable to get a database connection. Sometimes it resolve automatically in a minute or two else we need to reboot Backend app.
  • Although Storefront is not connected with Backend for all operations. But still while backend was throwing error. Storefront app was also becoming unresponsive with timeouts happening.
  • We have assigned around 200 connections to each app but still RDS was using maximum of 250 connections including all apps.
  • RDS CPU was running in range of above 80% at that time
  • Worst hit happened when RDS went to 100% CPU usage and it stayed there. Everything becomes unresponsive. We need to shutdown all apps, reboot RDS and start again and after that it was all normal for rest of the day even with decent load. It cost us 15 min downtime

Question

As I mentioned we are at a medium scale so we do not have sufficient logging to understand why and what really happened that day. What could possibly be done to avoid such incidents to happen in future? How can we prepare ourself, be it change architecture of app, scaling hardware or anything. All suggestions are welcome

I am attaching few cloudwatch metrics of RDS that day

enter image description here

enter image description here

enter image description here

enter image description here

1 Answer 1

1

Relational databases always come with scaling challenges. Nevertheless you could try following:

  1. Increase the instance size for RDS
  2. Use read replicas in RDS for your database and direct all read operations from the app to read replica.
  3. Also I hope, you have been watching IOPs utilization and not hitting the limits. Having provisioned IOPs if you are hitting the limits would be helpful.
  4. It looked like that you were also having scaling issues at app layer(storefront). You may want to implement horizontal scaling by putting your server behind a load balancer and let it autoscale. You might also want to increase the instances using autoscaling group ahead of running the campaigns and scale it back after the campaign (Make a calculated judgement).
  5. Also you may want to use a circuit-breaker pattern (https://martinfowler.com/bliki/CircuitBreaker.html) between storefront and admin service.
2
  • * I am not sure of our scale but I feel like at this load we don't need to increase instance size or create read replicas. Please advice any relevant article. * We have 300 write IOPS, but I think they get accumulated. On daily basis we don't use more than 50 IOPS * How can I find out if I have a scaling issue at app layer, because on EC2 monitoring it was looking healthy * Thanks for CircuitBreaker pattern, I will take a look in that
    – Abhinav
    Feb 17, 2019 at 15:01
  • Can I check IOPS credit for RDS? I am not able to find any option. And what will happen if I start hitting IOPS limit, does it contribute to high CPU usage?
    – Abhinav
    Feb 17, 2019 at 15:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .