We have a system that is currently in Amazon Ec2. I use Amazon ELB right now to load balance across 12 instances in 2 zones. I've been running some load tests with different characteristics:

1) a ramp of simultaneous users from 0 to 10,000 over 60 minutes. 2) a stepped ramp starting at 2,500 for 15 min, to 5,000 for 15 min, to 7,500 for 15 min, to 10,000 for 15 min.

On both, my servers sit around 20-30% cpu, 10-15% network and seem to perform nicely, our app logs everything as expected and is responding very quickly on the machine throughout the entire test. My traffic Requests per second on these machines seem to be pretty spikey and exhibits some weird behavior. One example being servers in zone 1 will drop 200-400 requests per second while servers in zone 2 will simultaneously jump up by 200-400 requests per second. All this while still responding just fine.

However, the results I see from the external load test are much different. I see a lot of timeouts and connection reset errors. Those timeouts and connection errors are somewhat spikey as well, meaning I'll go 5 minutes with no errors then get a little flood of them then back to nothing for awhile.

I'm suspecting the ELB may be rebalancing or something that is causing some of this behavior. I did try to move to just a really beefy hardware running HAPRoxy. That seemed to solve some of the issues on the lower end of the load test ramp, but then seemed to get network starved and couldn't go much beyond 5-6K requests per second.

I'm wondering if anyone has any insight into the Amazon ELB behavior? Also if there are any workarounds to this? That may include completely different solutions...




Answering my own question. I happened to come across this thread (https://forums.aws.amazon.com/thread.jspa?threadID=76834) that made an obscure reference to something called ELB Pre-Warm. I called up Amazon AWS Support and was told that it was indeed a service they offered, you just have to have a support plan (Gold level if you want to call them). They set everything up for me and I was able to run a successful load test with a ramp from 0-10K QPS in 5 minutes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.