Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm doing stress testing of our system. Currently we have 5 m1.large instances running behind ELB, sitting in east region. In west region, there are 3 small instances (with JMeter) that I use to hammer the system.

While doing a test that only pushes the app instances to about 80%-90% of their CPU limit (our choke point at the time), I'm seeing an odd behavior, ELB reports that ALL 5 instances are "Out of service - Transient Error - Please check later", all instances stop getting requests, and after about 5-10 seconds everything goes back to normal. This happens every 30 seconds or so. BUT! This doesn't happen every time I run the test. I just ran a half an hour stress test, with the same settings and everything worked perfectly. What is going on?

Btw my health check is

Ping Target:    HTTP:80/index.html
Timeout:    60 seconds
Interval:    300 seconds
Unhealthy Threshold:    10
Healthy Threshold:    2

So there is no way it's failing that. I've never ran into this until yesterday.

share|improve this question
add comment

2 Answers 2

The best way to stress test ELB is to get the ips used behind the cname they provide. Used those to hit the load balancer. Make sure there is at lease one image in every az you selected for the ELB. Amazon dynamically scales the ips behind the ELB, Your load balancer is probably hitting just a single ip. I'm not sure about the sporadic behavior you're experiencing.

share|improve this answer
    
Thank you for the tip! –  Chuck Morris Apr 28 '11 at 1:24
    
Did this help solve your problem? If so please mark this answer as accepted? –  Caleb May 9 '11 at 12:15
    
Good tip, but didn't solve it yet. –  Chuck Morris Jun 3 '11 at 1:12
add comment

We were also having a transient "boxes fail health checks for no good reason" problem and from working with Amazon support it turns out there is an interaction between the ELBs and the Apache KeepaliveTimeout. If the health check interval is larger than the timeout then the healch checker can try to reuse a bad connection and it fails the test and tosses your instance out of the ELB. They called our 60 second interval "unusually long." We're messing with it now but try setting your interval low and matching it with the keepalive setting in Apache.

share|improve this answer
    
Ernest, Thank you. Looking into this. –  Chuck Morris Jun 3 '11 at 1:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.