Let me first briefly describe our stack.
- Single EC2 instance with 4 vCPU and 16 GB RAM running Ubuntu [m4.xlarge]
- Single RDS instance with 2 vCPU and 8 GB RAM running MySQL 8 [db.m5.large]
- We are using cloudfront to cache static assets
- We are using Nginx as web server
- It is backed by tomcat instances in which our Java web apps run
- We have 3 primary apps each run in their own tomcat instance (Storefront website, Admin Backend, Seller Platform)
Storefront and Seller Platform talks with Admin backend by use of REST APIs for important events, rest they have direct integration with RDS for DB
All our apps are built in Java, using Spring & Hibernate as frameworks
- We are using tomcat database connection pooling in all apps
- We are using in memory cache for complete catalog and other things which do not frequently change to avoid hits on DB
What Happened:
On usual days, we have around 125 users on website as observed by Google analytics and get around 1 order in every 2 mintue. But one day, while we were running some aggressive campaigns we had around 4000 users on website at a time and about 18 orders were getting placed on website every minute. Because we promise same day delivery, our backend and seller platform was equally working on full capacity to serve those orders.
This high load was for 4 hours on our system and it resulted in as many as 20 downtimes, collectively making it around 50 minutes of downtime. It was primarily due to database issues.
Issue Observed
- Worst hit was our Backend application. After about every 15 minute it started throwing error that it is unable to get a database connection. Sometimes it resolve automatically in a minute or two else we need to reboot Backend app.
- Although Storefront is not connected with Backend for all operations. But still while backend was throwing error. Storefront app was also becoming unresponsive with timeouts happening.
- We have assigned around 200 connections to each app but still RDS was using maximum of 250 connections including all apps.
- RDS CPU was running in range of above 80% at that time
- Worst hit happened when RDS went to 100% CPU usage and it stayed there. Everything becomes unresponsive. We need to shutdown all apps, reboot RDS and start again and after that it was all normal for rest of the day even with decent load. It cost us 15 min downtime
Question
As I mentioned we are at a medium scale so we do not have sufficient logging to understand why and what really happened that day. What could possibly be done to avoid such incidents to happen in future? How can we prepare ourself, be it change architecture of app, scaling hardware or anything. All suggestions are welcome
I am attaching few cloudwatch metrics of RDS that day