Nginx proxy large file download fails instantly sometimes

Question

Problem

Downloading a large file (400MB+) with "transfer-encoding: chunked" fails after about 3 seconds. Downloading smaller files works even though possible taking longer than 3 seconds. The connection is just aborted mid-transfer which results in incomplete reads due to expected content missing. A download of the same file or a file of similar size can work a few hours consistently and then fail a few hours or days consistently.

Setup

• Ingress Nginx in proxy mode inside a kubernetes cluster.
• Running on AWS

Notes

The download always works when bypassing nginx by port forwarding to the actual server sending the response. The ingress/nginx pods are not failing and restarting. No error to be found in the log. Configuration options like proxy-buffering seem not to make a difference. It sometimes works, sometimes not.

Any idea what could cause this behavior? Happy to provide more details. Thanks.

Related

Those two issues look very similar:

• https://stackoverflow.com/questions/71428954/downloading-files-throwing-network-failure-from-webapps-hosted-on-kubernetes-usi
• https://stackoverflow.com/questions/67174426/large-files-download-issue-with-eks-nginx-jetty

Answer 1

Do your nginx pods show a high number of restarts? It is possible that the large download causing the liveness probe of nginx to timeout and eventually get terminated by Kubernetes.

If this is the case, you could increase the liveness probe timeout. I saw an issue recently where the liveness probes timeout even when the service is not working particularly hard which might be something to do with how the probes are created in the Go code.

Answer 2

In our case it was the AWS classic load balancer that caused the problem with large downloads. Please see this link for a statement by Amazon. Since switching to a network load balancer the problem is gone.

https://repost.aws/questions/QUXFiHUoT_RLaxa3DuiNvDHA/classic-load-balancer-terminates-large-downloads-mid-request