1

I'm designing a highly available WP site on AWS using Elastic Beanstalk, and testing use load with a Locust.

Everything looks alright: my EC2s are t2.mediums, auto scaled over 3-6 availability zones. Load balancer is set to "Cross-zone" load balancing (so traffic should be distributed to 3 servers in 3 different zones), I am using Aurora (db.t2.medium) with a master->read replica setup.

Things are fine when I visit the site in my browser, but as soon as I spin up Locust (with 100-500 users, 90-100 second wait times, 10 user hatch rate) my site will almost instantly lose the connection to the database and eventually throw a 50x error.

My Apache/PHP setup is pretty out of the box from Beanstalk (Amazon Linux AMI, php 5.6), specs listed below. opcache is enabled by default, but phpfpm is not currently installed.

Here is a diagram of my setup, and then the specs:

image

  • EC2
    • 3 t2.Mediums
    • 2 vCPUs
    • 24 CPU credits/hour
    • 4g RAM
  • Apache 2.4
  • PHP 5.6
    • upload_max_filesize => 64M
    • post_max_size => 64M
    • max_execution_time => 120
    • memory_limit => 256M
    • Opcache
    • opcache.enable=1
    • opcache.memory_consumption=128
    • opcache.interned_strings_buffer=8
    • opcache.max_accelerated_files=4000

I am unsure whether or not this is a hardware configuration issue, or if I need to tweak PHP/Apache/MySql

  • Define "lose 'the' connection to the database." What's the actual error you're logging that indicates a loss of connection(s?) to the db? – Michael - sqlbot Aug 10 '18 at 18:08
  • I don't see any egregious error when tailing the logs, but when I load the page in my browser (while locust is swarming the site) I see the "error establishing a database connection" screen. And then eventually my browser will let me know that Im getting 50x errors. – rugbert Aug 10 '18 at 18:31
  • 1
    Well, that's a very different scenario than "losing" the database connection. Connect to your Aurora instance with some kind of admin tool like the mysql cli or workbench. Run the query SHOW FULL PROCESSLIST; and SELECT @@MAX_CONNECTIONS;. – Michael - sqlbot Aug 10 '18 at 18:37
  • SELECT @@MAX_CONNECTIONS; shows 45 connections While I am running the swarm, the process list shows a bunch of connections from WordPress (DB_USER's user name), the command is sleep and state is "cleaned up", as well as a bunch of unauthenticated users running the "connect" command. – rugbert Aug 10 '18 at 19:08
  • 1
    actually I think you got your answer already. Your max connections is set to 45, which will top out when trying 100 users – mzhaase Aug 10 '18 at 20:53
0

OK so I think I had several issues:

  1. When I originally created the DBs, I made t2.micros which by default only allow for 40 connections at once. I had later changed the instance, to t2.mediums but the max_connections seemed to stay the same. I have recreated the DBs at t2.mediums and the max_connections are now 90 and I can increate if need be.

  2. I misread the Locust documentation, and I had set the test to hit the site every 90ms, so every .09 seconds which is a lot. I just increased the hit time to 3-10 seconds (actual seconds) and the servers hold up fine now.

Increasing the Locust users to 200 however results in a 75% failure (database disconnect) rate but I think I can either adjust the max_connections even more, or throw a CDN in front of the site (which I'll be doing anyway)

@michael-sqlbot gets the prize here, he led me down the right path.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.