1

Background

An HTTP request sent by an end-user to www.stuff.com/things is processed as follows:

  1. Received by an application Load Balancer (the public load balancer) that load balances to an NGINX reverse proxy (Let's call this the security proxy).
  2. The security proxy routes the request to a private application load balancer.
  3. The private load balancer routes traffic based on the URL. if the url is /things/... then the request is routed to another NGINX proxy.
  4. This second NGINX proxy (lets call this the things proxy) routes all requests to a classic load balancer.
  5. The classic load balancer listens on ports 80 and 50000 and balances requests to instances hosting the things application.

So to recap: I have the public load balancer, the security proxy, the private load balancer, the things proxy, and the classic load balancer.

The problem

When a user tries to visit www.stuff.com/things, they wait, and they wait, and they eventually receive an HTTP 504: gateway timeout. Or, the page loads immediately. Or, some elements load quickly while other hang.

Debugging

I've tried issuing requests from several places in the chain:

  1. curl the classic load balancer, response is instantaneous
  2. curl the things proxy, response is instantaneous
  3. curl the private load balancer, response is instantaneous
  4. curl the security proxy, response is instantaneous
  5. curl the public load balancer, response is slooooow... except when it's fast.

Perhaps I need to reconfigure my proxies?

Proxy Configuration

The things proxy is configured as follows:

events {
    worker_connections 1024;
}
http {
    server {
        location ^~ /proxycheck {
            return 200 'available';
            add_header Content-Type text/plain;
        }

        location / {
            proxy_pass ${PROXY_ADDRESS};
        }
    }
}

The security proxy is configured as follows:

events {
    worker_connections 1024;
}
http {
    server {
        location ^~ /proxycheck {
            return 200 'available';
            add_header Content-Type text/plain;
        }

        location / {
            proxy_pass ${PROXY_ADDRESS};
            proxy_read_timeout 90;
            proxy_redirect default;
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

In the future, the security proxy will check certificates. Until then, I've got this other problem.

Any suggestions?

4
  • 1
    What do your error log files show when the request is slow? – Tero Kilkanen Mar 5 '17 at 18:06
  • I count five load balancers or proxies in the path of the request, could you outline why so many are necessary? Can you please edit your post to include a network diagram to explain the network and each node - CloudCraft have a good online tool. – Tim Mar 5 '17 at 18:10
  • @Tim sure! Will do this tonight. The long story short: The security proxy provides corporate security, the things proxy is required because I can't connect a load balancer to a load balancer (AFAIK, AWS doesn't provide a means of doing this). The public load balancer is required to balance load across security proxies, the private load balancer balances for applications (all but things, where it balances those proxies), and the things load balancer is required because an AWS application load balancer doesn't support non-http health checks or more than a single port. – Liam M Mar 5 '17 at 22:56
  • Seems legit. Turn on the logs for the ELBs. You'll find the individual balancer node handling each request (each balancer node logs its own requests, with the node's IP in the filename, at least for Classic) and that may turn out to be useful. Each ELB has a minimum of two nodes, which is something you may or may not be aware of, since it's done transparently. You could have a problem with one but not the other. – Michael - sqlbot Mar 5 '17 at 23:26
0

It looks like this must have been a transient issue with AWS (related to the much-publicised fat-fingers incident), because I haven't been able to replicate it since posting this question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.