AWS Elastic Beanstalk 504 Gateway Time-out - occasionally failing to connect to target

Question

I have a fairly simple AWS Elastic Beanstalk setup (Rails on Puma, with NGINX) and get intermittent but quite regular 504 Gateway Time-out on the client (typically 10 seconds after making the request). The requests that fail are quite simple, so should not be timing out.

NGINX acts as a reverse proxy to the Puma app server running on the same instance, and provides TLS termination. There are two Beanstalk managed EC2 instances with this setup, behind the standard Beanstalk configured AWS application load balancer (ALB). The load balancer also terminates the TLS connection before passing it on to NGINX.

browser (443) ---> (443) ALB (443) ---> | (443) NGINX (socket)-> Puma |

I have followed the AWS troubleshooting: https://aws.amazon.com/premiumsupport/knowledge-center/504-error-alb/

All this really confirms is that the monitoring shows HTTPCode_ELB_5XX counts, which indicates the 504 error originated from the load balancer. No 50x errors are coming back from NGINX. I have checked the ALB logs, and they show that the 504 error occurs, but is unrelated to any specific target. The request never makes it to a target backend. The monitoring does show target connection errors.

I have a specific NGINX server config, with the only real related settings (within the server {...} block) being:

    ssl_session_timeout        5m;
    client_max_body_size       20M;
    proxy_read_timeout         300;
    client_header_timeout      75;
    client_body_timeout        75;
    keepalive_timeout          75;

The ALB configuration has Idle timeout 60 seconds

The EC2s show only about 50% memory utilization and low CPU.

I have also spent some time looking at: https://sigopt.com/blog/the-case-of-the-mysterious-aws-elb-504-errors/ which led me to look into keepalives, etc, and Wireshark. In summary, NGINX shows that a client ALB closed a connection. Soon after, sometimes I'll get a 504 in the browser. This is typically 10 seconds later, and corresponds with the ALB not being able to get a return from a target. But there is no reason why the targets should not respond, and they are not even being touched. It is almost like a security groups issue just very occasionally blocking a request, but these either work or they don't.

The app servers are already prewarmed by the time these errors occur, and there are no CPU spikes when the failures happen.

I know that the application runs fine on a single instance beanstalk without an ALB in front of it, so we can rule out app performance for this discussion.

Any help or suggestions would be much appreciated.

Answer 1

I believe that this is due to Puma completely rejecting connections while it is initially starting up. Prewarming may possibly make this worse, since the time to get all the threads up may slow the initial startup on resource constrained servers.

You can see this in development more clearly if you change the development.rb entries to:

  config.cache_classes = true
  config.eager_load = true

This makes it closer to production mode in terms of loading. Start the rails server and immediately go to http://localhost:3000

If you check the browser developer console for network requests you'll see a net::ERR_CONNECTION_REFUSED status any time up until Puma say it is listening

* Listening on http://127.0.0.1:3001
* Listening on http://[::1]:3001
Use Ctrl-C to stop

For my app, which does a lot of dynamic loading and class creation on the initial startup, this takes more than 10 seconds. So any requests to NGINX during this time will instantly fail, since Puma is just not listening. Future requests that may create new worker threads may well be slow, but won't fail, because Puma is already listening and accepting connections.

This doesn't solve the issue, but does at least help me stop chasing red herrings. Now I need to work out if there is a way to get Puma to start listening faster, or for NGINX to wait in some way for Puma to accept connections before telling clients that there is an issue, then breaking the load balancer.