We have a system that is currently in Amazon Ec2. I use Amazon ELB right now to load balance across 12 instances in 2 zones. I've been running some load tests with different characteristics:
1) a ramp of simultaneous users from 0 to 10,000 over 60 minutes. 2) a stepped ramp starting at 2,500 for 15 min, to 5,000 for 15 min, to 7,500 for 15 min, to 10,000 for 15 min.
On both, my servers sit around 20-30% cpu, 10-15% network and seem to perform nicely, our app logs everything as expected and is responding very quickly on the machine throughout the entire test. My traffic Requests per second on these machines seem to be pretty spikey and exhibits some weird behavior. One example being servers in zone 1 will drop 200-400 requests per second while servers in zone 2 will simultaneously jump up by 200-400 requests per second. All this while still responding just fine.
However, the results I see from the external load test are much different. I see a lot of timeouts and connection reset errors. Those timeouts and connection errors are somewhat spikey as well, meaning I'll go 5 minutes with no errors then get a little flood of them then back to nothing for awhile.
I'm suspecting the ELB may be rebalancing or something that is causing some of this behavior. I did try to move to just a really beefy hardware running HAPRoxy. That seemed to solve some of the issues on the lower end of the load test ramp, but then seemed to get network starved and couldn't go much beyond 5-6K requests per second.
I'm wondering if anyone has any insight into the Amazon ELB behavior? Also if there are any workarounds to this? That may include completely different solutions...
Thanks,
MikeD
