11

I've got nginx configured as a front-end to a Python application running under gunicorn, but nginx is terminating connections after about 65k of data have been sent.

For example, I've got a view which looks like this:

def debug_big_file(request):
    return HttpResponse("x" * 500000)

But when I access that URL through nginx, I only get 65283 bytes:

$ curl https://example.com/debug/big-file | wc
…
curl: (18) transfer closed with outstanding read data remaining
   0       1   65283

Note that everything works as expected when accessing gunicorn directly:

$ curl http://localhost:1234/debug/big-file | wc
…
   0       1   500000

The relevant nginx config:

location / {
    proxy_pass http://localhost:1234/;
    proxy_redirect off;
    proxy_headers_hash_bucket_size 96;
}

And nginx version 1.7.0

Some other facts:

  • The number of bytes is consistent from request to request, but it varies based on the content (I first noticed it with a large PNG file, which was cut off after 65,372 bytes, not 65,283)
  • 110k bytes are sent correctly (ie, "x" * 110000 returns all 110,000 bytes), but 120k bytes are not
  • tcpdump suggests that nginx is sending a RST packet to gunicorn: nginx sending RST
  • It would be helpful to see (a) how gunicorn is choosing to frame replies from 110k to 120k bytes in size, and (b) how nginx then chooses its framing for that same range of sample payload sizes between 110k and 120k bytes. The three ways that HTTP can frame data: provide content-length; do chunked encoding; or give no framing at all except to promise to close the socket when the body is complete. – Brandon Rhodes May 26 '14 at 18:34
  • A content-length header is being provided. Let me packet dump to see what's going on between the two otherwise… – David Wolever May 26 '14 at 18:35
  • Hrm, very weird. tcpdump suggests that nginx is actively RST-ing the connection (see edit). nginx is also using HTTP/1.0 and Connection: close. The I've also confirmed that the Content-Length header is correct. – David Wolever May 26 '14 at 20:36
10

Okay! After double checking the nginx logs, this turned out to be the problem:

2014/05/26 16:50:56 [crit] 31396#0: *11 open() "…/proxy_temp/2/00/0000000002" failed (13: Permission denied) while reading upstream, client: 1.2.3.4, server: _, request: "GET /debug/big-file HTTP/1.1", upstream: "http://127.0.0.1:1234/debug/big-file", host: "example.com"

Some how the permissions for the proxy_temp directory got messed up which prevented nginx from properly buffering to it.

  • 1
    Yeah, I just solved a problem just like this, looked in nginx logs, had a line containing [crit] 6636#0: *16817 open() "/var/lib/nginx/proxy/7/03/0000000037" failed (13: Permission denied) while reading upstream, did sudo chown -R www-data:www-data /var/lib/nginx/ and it got fixed. – Epigene Nov 3 '15 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.