Nginx reverse proxy repackaging chunked responses

Question

I am running nginx/1.1.19:

# nginx -V
nginx version: nginx/1.1.19
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module --add-module=/build/buildd/nginx-1.1.19/debian/modules/nginx-auth-pam --add-module=/build/buildd/nginx-1.1.19/debian/modules/nginx-echo --add-module=/build/buildd/nginx-1.1.19/debian/modules/nginx-upstream-fair --add-module=/build/buildd/nginx-1.1.19/debian/modules/nginx-dav-ext-module

I am using it to reverse proxy a service that can stream data as a chunked response. Each chunk from the backend service is a valid JSON document, but when accessed through the nginx proxy, the chunks are repackaged into 'round' sizes, like 0x1000, 0x2000, 0x4000, etc. This makes the chunks no longer valid JSON blobs, since each chunk will have the end of one document at the beginning, and the beginning of another document at the end.

My location block is configured like so:

    location / {
            proxy_pass http://backend;
            proxy_read_timeout 600;
            proxy_buffering off;
    }

I thought that the proxy_buffering off; part should avoid the behavior I am seeing. Is there some other setting I need to change?

Answer 1

You should never bind your business data to chunks. Chunk size can be changed by proxies, or event by different web servers. Even some proxies based on some heuristics may aggregate the chunks. Your chunk parser (client SDK if you have any or etc.) should be able to parse your data even if each chunk has part of a JSON or 1.5 JSON or multiple JSONs.