5

When ever I try to load test a site that is load balanced on EC2 I get the following message:

ab -n 20 -c 20 http://www.somesite.com/

Benchmarking www.somesite.com (be patient)...Send request failed!
apr_socket_recv: Connection reset by peer (54)

Any sites that are hosted on EC2 without the load balancer can be load tested successfully, so I figure it's not related to the security group I have configured.

Does AWS EC2 load balancer block this type of request by default? If so how does it differentiate between an AB and normal browser based http request?

2
  • Not as far as I know. Can you confirm that the ab traffic is not hitting the backend servers (via server logs or packet captures)? What is the result with a similar tool such as httperf? Perhaps ELB considers the ab HTTP request header invalid for some reason... Feb 1, 2012 at 12:33
  • what are the results with "ab -n 1 -c 1", that is no concurrency? Does ab succeed in that case? If it does, you're likely seeing some connection-limiting behavior.
    – rmalayter
    Feb 1, 2012 at 17:10

3 Answers 3

6

Does AWS EC2 load balancer block this type of request by default?

Not that I know of - are you by chance running Mac OS X Lion? There seem to be a commonly encountered bug running ab on this particular operating system release - according to Fixing ApacheBench bug on Mac OS X Lion a patch is available (see there for details), though it's probably worthwhile and preferable to check for an official update first, insofar this patch (or an equivalent fix) has supposedly been incorporated in the meantime as of version httpd-2.3.15-beta at least (see Fixing Apachebench on OS X Lion).

Obviously, you could verify this first (and apply it as a workaround as well in case) by running the same test from a respectively different operating system (release).

Good luck!

1
  • 1
    Thanks. Yep this is the exact issue, it's down to the current versions of ApacheBench in Lion. Can't confirm if patch works as haven't applied yet but will post if I get a resolution.
    – Mitul
    Feb 2, 2012 at 13:22
3

Amazon ELB's are scaled up when load increases. When you have 5 requests per second you are on a different load balancer than the people running 1000 requests per second. What happens is that when you're running tests on the loadbalancer it goes from 5 to 1000 requests per second and the loadbalancer cannot handle the load so it will deny requests. If you wait 10-15 minutes for the loadbalancer to scale up while the test is running you should notice that all the requests will complete.

Amazon can manually upgrade your loadbalancer if you expect this is going to cause problems with your production environment. I have a loadbalancer / auto scaling group set up in the cloud and our traffic can go from a few hundred per second to a few thousand in a matter of 1-2 hours on the weekends and we don't actually see this behavior while running in the actual production environment, we only see this when we run tests with jmeter.

2

Load balancers are often configured to rate-limit or block requests that don't look like they came from "real browsers" as a security measure. I suspect that might be the case here; I'm sure Amazon has some docs.

Consider capturing a real request from your browser (using say Fiddler or Firebug) and add the headers you might need with ab's -H parameter. You often need a "Host", "Accept", and "Accept-Encoding" for example, perhaps with a user-agent string that looks realistic.

1
  • Good point regarding load balancing in general, but it is possible to test ELB via ab just fine in principle, and the reported error would be different in case (likely apr_socket_recv: Connection timed out (110)), so I think this doesn't apply here - see e.g. [ELB capacity (simultaneous TCP sessions)?}(forums.aws.amazon.com/message.jspa?messageID=187513) for an illustration. Feb 1, 2012 at 15:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .