5

We recently moved our web servers from some 10 year old boxes to AWS EC2.

Usage of the site is currently higher now (it's our busy season) and the site has become much slower, which is unexpected because our instance sizes are much higher than what we had previously.

We run a pretty small site that only gets a few hundred at a time. We're running a c3.large instance on our webserver, and db.m1.large for our RDS MySQL database. We don't have any read replicas and or multiple webservers (load balancing). According to Google Analytics, we only had 18,106 page views for the whole day.

Our users (external and internal) keep seeing the site time out in their browser. It's pretty much across the board instead of any particular page. MySQL PROCESS LIST is also nearly empty without any table locks or whatnot.

If you look at our stats in CloudWatch, everything should be fine. We have very CPU utilization, and what I think is pretty low Network I/O. Likewise on the RDS side, nothing here is screaming "bottleneck".

EC2 Usage (c3.large) UE2 Usage

RDS Usage (db.m1.large) RDS Usage

Any ideas how I should go about troubleshooting this issue?

8
  • RDS? As in Windows Server? Have you tried using PerfMon to see what the OS sees? Also look at the SQL stuff too, it may be stuck running a query.
    – MDMoore313
    Feb 25, 2014 at 18:51
  • 1
    18,106 pageviews a day means one every 5 seconds on average, yet you're averaging 50-100 writes and 200-400 reads per second. I'd troubleshoot that first.
    – ceejayoz
    Feb 25, 2014 at 19:13
  • 2
    @MDMoore313 - no RDS as in AWS's managed db service.
    – EEAA
    Feb 25, 2014 at 19:42
  • @EEAA In fairness, RDS offers Windows SQL Server. OP said MySQL somewhere in the post, though.
    – ceejayoz
    Feb 25, 2014 at 19:43
  • Sorry, forgot to add: EC2 instance is Amazon Linux w/ Apache/PHP/Memcached (although we just started using Memcached and only have it on 2 or 3 places so far)
    – DOOManiac
    Feb 25, 2014 at 19:56

1 Answer 1

6

Finally tracked down the cause of our problems. Apache was misconfigured to have a Keep Alive timeout for 30 seconds. That combined with an overly aggressive AJAX script was causing the site to hang and wait for a connection.

Turning KeepAliveTimeout down to 7, as well as taming the AJAX script, brought everything back to normal.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .