0

we've been running our web servers at Amazon with load balancer and auto-scaling for over a year with no problem. All of a sudden today the request began to get aborted with the error: 503 ... Backend server is at capacity

The web servers are at 1% CPU and no other alarms trigger.

We use Amazons load balancer and nginx.

Lots of requests like this are showing up in the access_log.

10.246.114.93 - - [05/Jun/2014:20:16:09 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:09 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:09 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:09 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:10 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:10 +0000] "-" 400 0 "-" "-"
10.246.114.93 - - [05/Jun/2014:20:16:10 +0000] "-" 400 0 "-" "-"
10.229.15.214 - - [05/Jun/2014:20:16:10 +0000] "-" 400 0 "-" "-"
10.229.15.214 - - [05/Jun/2014:20:16:10 +0000] "-" 400 0 "-" "-"

Any thoughts?

1
  • 1
    Is the site working intermittently or not at all? Are the 400 errors all you see or just what caught your attention? Do you have any older logs to check whether those are actually common entries even when the site was fine (they may be "normal"-ish, and a distraction from the current issue) Do you have ELB logging enabled? ELB and instances in one AZ or multiple? Jun 5 '14 at 23:38
1

You will get a "Back-end server is at capacity" when the load balancer performs its health checks and receives some simple error due to a mis-configuration.

Try to grep the log files for "ELB-HealthChecker", which is the user agent used by the checks.

This will typically give you a 400 or 500 level error which is easily fixed.

3
  • Unfortunatley this is what i got... /var/log/nginx/access_log:10.246.114.93 - - [05/Jun/2014:20:30:49 +0000] "GET /health.html HTTP/1.1" 200 9 "-" "ELB-HealthChecker/1.0" /var/log/nginx/access_log:10.229.15.214 - - [05/Jun/2014:20:30:49 +0000] "GET /health.html HTTP/1.1" 200 9 "-" "ELB-HealthChecker/1.0"
    – dale
    Jun 5 '14 at 20:37
  • Check this out from today, it looks like there was an issue with OpenSSL: aws.amazon.com/security/security-bulletins/…
    – paranoid
    Jun 5 '14 at 21:05
  • @paranoid I'm struggling to figure out why you think that CVE has anything to do with this.
    – ceejayoz
    Dec 18 '17 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.