1

I'm running an OpsWorks stack with:

  • 10 c3.2xlarge instances running across 4 availability zones.
  • The CPU never surpasses 5% on each instance.
  • Each instance is connecting (through a memcached layer) to a db.r3.8xlarge (32 vCPU, 244 GiB RAM) RDS instance.
  • The DB's CPU is staying nicely between 15-20 percent.
  • Each EC2 instance is set to 250 connections with the RDS (total 2,500 connections).
  • Each instance's memory is gradually increasing but I think this is a separate issue.
  • The load_1, load_5, load_10 metrics are all really high 0.9-1.0.
  • The active concurrent users connecting (based on a 3 minute window) are getting stuck at between 2500-3000 connections.

We tried increasing RDS instance size and while it did stop our RDS from maxing out we're trying to figure out why the conncurrent connections seems to be pegging.

Is there some setting in the ELB that limits connections? The latency for calls is really high. I'm just trying to figure out something which might be throttling this since all the hardware seems to be running okay.

5
  • What is "VPC gateway"? VPC Internet Gateway has no instance type. Do you run some sort or custom router/NAT? Dec 16, 2016 at 6:07
  • It's not clear what your problem is, what the m1.small is doing, or what the resource usage of the m1.small is.
    – Tim
    Dec 16, 2016 at 9:45
  • 1
    Given that the common AWS resources of your connections are the ELB and your RDS instance, you might also look into the possibility of your RDS instance being the connection limiting factor.
    – Castaglia
    Dec 28, 2016 at 3:56
  • @Castaglia You're correct! After a few days it turned out there was a single query involving an order by rand() which was running on the million+ records and consuming the CPU on the RDS. I fixed the query and everything is running smoothly again.
    – Nick Gotch
    Dec 28, 2016 at 15:55
  • I pulled the "VPC gateway" out of the question. It turned out to be an EC2 instance a former admin had set up. Not relevant to the question.
    – Nick Gotch
    Dec 28, 2016 at 15:57

1 Answer 1

2

It turned out after much digging there was a single query being run against the DB which involved order by rand() and which was running on almost the entire 1.3 million records in the table. After fixing this one query the performance dropped to normal.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .