0

I've been hosting a website on an Amazon EC2 instance for years. Recently, users have complained of slowness and connection failures. I've checked memory and CPU usage on both the EC2 LAMP server and on the RDS database server and both seem well within nominal range.

Web Server

  • CPU usage averages about 15%, with rare spikes to around 50-60% about twice a day
  • Memory usage 3.5G total, 3.2G used, 2.7G cached, swap usage zero

DB Server

  • CPU usage typically 2-5%, with daily spikes. These spikes have been gradually getting higher for about a week, but never exceed 10%
  • DB connections under 1 except for infrequent spikes to 2
  • 5GB free RAM

Using netstat, I see at any given time that there are around 1000 connections to the web server:

$ netstat -ant | wc -l 1089 I've seen this number as high as 1480 earlier in the day when the problems occur.

All of this makes me think that the machine is network-found. I.e., there's not enough available network bandwidth to serve all the requested data. I think this lack of bandwidth may be the machine's bottle neck.

Can anyone suggest how to determine if this machine is, in fact, limited by network bandwidth? It would be extremely helpful if I could construct a network usage graph that indicates the problem. I'm not sure what this might look like, but am imagining a graph showing a hard plateau during the times of poor performance.

I've attempted to attach a screenshot here of the AWS monitoring graphs:

Web Server Performance Monitoring

EDIT: I was monitoring the server this morning when the slowness started happening and I have been unable to locate any resource bottleneck. The web server's memory and CPU usage seem fine. The db server's memory and CPU usage seem fine. I don't see any outrageous amount of network bandwidth being used and yet the server responds very slowly to page requests. Then the problem suddenly evaporates.

While the problem persists, it looks from a user perspective (using Firefox) as though there's something slow about the TLS handshake that looks very much like this problem but my apache server has HostnameLookup set to OFF.

The bottleneck, whatever it is, appears to prevent network connections from being made. During the slowness, total network connections were steady around 800:

netstat -n | wc -l

While the connections to the database from the web server were very steady around 200:

netstat -an | grep <db-server-ip-here> | wc -l

As soon as the problem passes (which seems quite erratic) then these numbers jump to about double these values and the server runs lightning fast.

  • I think the AWS network with good instance sizes can give you up to 10Gbps. How much bandwidth are you using? Have you considered latency? Are all your resources inside one AZ? Have you done load testing, watching what happens when you increase load? I have a t2.nano that seems to have a limit somewhere as well that I couldn't locate, with the limited time I spent. CPU / RAM were fine, pages served out of the Nginx cache, but there was a limit somewhere even when everything was on one machine and testing was from a large spot instance in the same AZ / subnet. – Tim Sep 21 '18 at 0:57
  • @Tim, thanks for responding. I don't have access to any reports of bandwidth usage. The hosting account doesn't belong to me. I also have been unable to find any AWS documentation describing what bandwidth allocation corresponds to a m1.medium instance -- or what instance might provide more bandwidth. My resources are both in us-east-1c. The machine is in production, so I don't know how to replicate these conditions for load testing. – S. Imp Sep 21 '18 at 1:03
  • 1
    I'd start by moving to a more modern machine - t3.medium is latest generation and suitable for bursty workloads. Cloudwatch is the best way to monitor bandwidth. I think the 20,000,000 bytes in 7,500,000 out translates to a peak of 2.6Mbps in and 1.3Mbps out, with an average about 1/4 of that, so you're nowhere near the AWS bandwidth limit. I'd look at latency to RDS, I'd look at top / iotop. – Tim Sep 21 '18 at 2:42
  • @Tim Thanks for your suggestion It's baffling to me that there'd be more IN traffic than OUT traffic. That makes no sense at all. Unfortunately, your suggestion is greek to me. I'm familiar with top, but don't understand how it would provide any intel about RDS latency. – S. Imp Sep 21 '18 at 4:09
  • top is to check if any process is using any CPU. iotop is to see if anything is waiting for disk io. You should see if there's anything similar for network I/O. Do users upload files? Are you being attacked? – Tim Sep 21 '18 at 4:57
2

We had a similar issue on one of our higher velocity stats clusters at Speedtest.net - and we discovered that the solution in our case isn't publically documented at AWS; we had to work with the Nitro team directly to solve the issue.

We had a low bandwidth and a low PPS (~10,000 packets per sec) machine that was consistently losing packets. We couldn't figure out why we were losing packets as we were well within the public guidelines for the machine performance. This machine was a statsd aggregator, so thousands of machines were sending UDP datagrams to it. The "stream" count is a key point.

It turns out that if you have any security groups on the listening port which restricts sending IP ranges, AWS imposes a conntrack limit for that given port. In the event the connection count limit is exceeded, AWS will silently drop packets. There are no statistics which expose this, beyond seeing "clipped" peaks on the network graphs. Larger instance sizes have larger conntrack quotas.

The solution is to set the inbound allowed source IP range to 0.0.0.0 for the given service port - this turns off connection tracking on AWS's end and removes the conntrack limit. Ultimately, this does mean that you have to handle the firewall yourself via careful subnetting and machine kernel firewalling.

I can't say if you are hitting the same issue, but it was something we ran into that caused unexplainable network issues at AWS.

  • First, thanks for your help. The EC2 instance in question is an m1.medium instance. The machine has been running quite a long time and is in ClassicLink mode (which may have something to do with things). Looking at the security group associated with it, there are no outbound rules at all. There doesn't even appear to be any option to apply outbound rules. Do you mean to suggest that you remove ALL firewall rules, including inbound ones? That seems pretty dangerous. – S. Imp Oct 3 '18 at 19:34
  • I also want to say that the network graphs don't show any clipping. On the contrary, when the server is running smoothly, the network in and out both show a dramatic increase. – S. Imp Oct 3 '18 at 19:37
  • @S.Imp if you have any security groups on the listening port which restricts sending IP ranges, AWS imposes a conntrack limit for that given port. Yes they are saying that Security Group rules impose a connection limit. – zymhan Oct 3 '18 at 19:52
  • 1
    It definitely is dangerous if your stack isn't well architected, hence the warning in the initial post. You should be using private subnets and iptable rules if this is needed. To your point about the clipped graphs, I'd potentially argue otherwise - seeing the ripples at the peak might be implying that you are exceeding your limit, and then AWS is clamping down, reducing your bandwidth. Then the process repeats after clients are disconnected. – Brennen Smith Oct 3 '18 at 21:34
  • 1
    If that's your current AWS security group config, it sounds like you are hitting a different issue. Best of luck! The only thing I didn't see in your diagnosis is a look at disk IO on the database instance - high IO contention obviously can cause database issues. – Brennen Smith Oct 3 '18 at 22:54

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.