1

I'm designing a highly available WP site on AWS using Elastic Beanstalk, and testing use load with a Locust.

Everything looks alright: my EC2s are t2.mediums, auto scaled over 3-6 availability zones. Load balancer is set to "Cross-zone" load balancing (so traffic should be distributed to 3 servers in 3 different zones), I am using Aurora (db.t2.medium) with a master->read replica setup.

Things are fine when I visit the site in my browser, but as soon as I spin up Locust (with 100-500 users, 90-100 second wait times, 10 user hatch rate) my site will almost instantly lose the connection to the database and eventually throw a 50x error.

My Apache/PHP setup is pretty out of the box from Beanstalk (Amazon Linux AMI, php 5.6), specs listed below. opcache is enabled by default, but phpfpm is not currently installed.

Here is a diagram of my setup, and then the specs:

image

  • EC2
    • 3 t2.Mediums
    • 2 vCPUs
    • 24 CPU credits/hour
    • 4g RAM
  • Apache 2.4
  • PHP 5.6
    • upload_max_filesize => 64M
    • post_max_size => 64M
    • max_execution_time => 120
    • memory_limit => 256M
    • Opcache
    • opcache.enable=1
    • opcache.memory_consumption=128
    • opcache.interned_strings_buffer=8
    • opcache.max_accelerated_files=4000

I am unsure whether or not this is a hardware configuration issue, or if I need to tweak PHP/Apache/MySql

Improve this question
12
  • Define "lose 'the' connection to the database." What's the actual error you're logging that indicates a loss of connection(s?) to the db?
    – Michael - sqlbot
    Commented Aug 10, 2018 at 18:08
  • I don't see any egregious error when tailing the logs, but when I load the page in my browser (while locust is swarming the site) I see the "error establishing a database connection" screen. And then eventually my browser will let me know that Im getting 50x errors.
    – rugbert
    Commented Aug 10, 2018 at 18:31
  • 1
    Well, that's a very different scenario than "losing" the database connection. Connect to your Aurora instance with some kind of admin tool like the mysql cli or workbench. Run the query SHOW FULL PROCESSLIST; and SELECT @@MAX_CONNECTIONS;.
    – Michael - sqlbot
    Commented Aug 10, 2018 at 18:37
  • SELECT @@MAX_CONNECTIONS; shows 45 connections While I am running the swarm, the process list shows a bunch of connections from WordPress (DB_USER's user name), the command is sleep and state is "cleaned up", as well as a bunch of unauthenticated users running the "connect" command.
    – rugbert
    Commented Aug 10, 2018 at 19:08
  • 1
    actually I think you got your answer already. Your max connections is set to 45, which will top out when trying 100 users
    – mzhaase
    Commented Aug 10, 2018 at 20:53

1 Answer 1

Reset to default
0

OK so I think I had several issues:

  1. When I originally created the DBs, I made t2.micros which by default only allow for 40 connections at once. I had later changed the instance, to t2.mediums but the max_connections seemed to stay the same. I have recreated the DBs at t2.mediums and the max_connections are now 90 and I can increate if need be.

  2. I misread the Locust documentation, and I had set the test to hit the site every 90ms, so every .09 seconds which is a lot. I just increased the hit time to 3-10 seconds (actual seconds) and the servers hold up fine now.

Increasing the Locust users to 200 however results in a 75% failure (database disconnect) rate but I think I can either adjust the max_connections even more, or throw a CDN in front of the site (which I'll be doing anyway)

@michael-sqlbot gets the prize here, he led me down the right path.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .