Upload files slow (nginx + http2)

Question

I'm migrating a php webapp from an apache server to an nginx. Everything works as expected except uploading files.

A file of 156 MBs takes ~11s to be uploaded on current server with apache, while the same file on the new server on nginx takes ~38s.

First approach (apache coexisting with nginx on the new server)

First I thought about server broadband issues, so I installed Apache on the new server to access the same webapp as nginx but in a different port.

The upload takes the same ~11s as on the old server. The new server is ok.

Second approach (nginx as reverse proxy)

Reading about apache and nginx, it seems that nginx could be faster for assets and cache, but slower working with php files (you need to pass the request to php fpm).

I tried using nginx as a reverse proxy for apache. The assets served by nginx, and the php requests proxy-passed to apache. It worked great, with the desired ~11s for uploads.

I thought I've got it all solved. While doing all these tests, I had disabled the tls certificates. I tested it in a non secure way through the 80 port. Once I enabled the tls certificate, I went back to the ~38s for uploads.

The new server is secured and using HTTP/2 (~38s).

The current server with apache is also secured, but is using HTTP/1.1 (~11s).

Third approach (back to nginx and php fpm)

I went back to first step, I didn't get advantages using the reverse proxy. The webapp with just nginx and php fpm, and without certificate, works ok: uploads again at ~11s.

So using nginx with a certificate and HTTP/2 is the problem.

Fourth approach (configuring nginx)

I've read three interesting resources about flow control window:

How to use Multiplexing http2 feature when uploading

Delivering HTTP/2 upload speed improvements

HTTP/2 Flow Control

I thought that maybe there were a lot of roundtrips from the client to the server uploading a large file giving the default nginx configuration. I've tried all kind of stuff configuring nginx (buffering and http2) to have large control window, but it turns out that the default configuration works good.

I used the nghttp command to see the interactions between a client and the server when uploading a large file. No roundtrips at all, the client is allowed to send the whole file without interruptions and taking a little bit more than the ~11s (but acceptable). The command's output looks like:

[  0.116] Connected
The negotiated protocol: h2
[  0.176] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.176] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.176] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.177] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.177] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.177] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.177] send HEADERS frame <length=56, flags=0x24, stream_id=13>
          ; END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: POST
          :path: /
          :scheme: https
          :authority: www.xxxxxxx.com
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.40.0
          content-length: 156631084
[  0.177] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.177] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.177] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.177] send DATA frame <length=16383, flags=0x00, stream_id=13>
[  0.222] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[  0.222] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.224] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.224] send DATA frame <length=1, flags=0x00, stream_id=13>
[  0.273] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.273] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
          (window_size_increment=2147418111)
[  0.273] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.274] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.274] send DATA frame <length=16384, flags=0x00, stream_id=13>
[  0.274] send DATA frame <length=16384, flags=0x00, stream_id=13>

(...)

[ 14.587] send DATA frame <length=16384, flags=0x00, stream_id=13>
[ 14.587] send DATA frame <length=16384, flags=0x00, stream_id=13>
[ 14.587] send DATA frame <length=16384, flags=0x00, stream_id=13>
[ 14.587] send DATA frame <length=16384, flags=0x00, stream_id=13>
[ 14.587] send DATA frame <length=16384, flags=0x00, stream_id=13>
[ 14.587] send DATA frame <length=44, flags=0x01, stream_id=13>
          ; END_STREAM
[ 17.404] recv (stream_id=13) :status: 200
[ 17.404] recv (stream_id=13) server: nginx
[ 17.404] recv (stream_id=13) date: Wed, 27 Jan 2021 07:57:30 GMT
[ 17.404] recv (stream_id=13) content-type: text/html; charset=UTF-8
[ 17.404] recv (stream_id=13) content-length: 0
[ 17.404] recv (stream_id=13) x-frame-options: SAMEORIGIN
[ 17.404] recv (stream_id=13) x-xss-protection: 1; mode=block
[ 17.404] recv (stream_id=13) x-content-type-options: nosniff
[ 17.404] recv HEADERS frame <length=126, flags=0x05, stream_id=13>
          ; END_STREAM | END_HEADERS
          (padlen=0)
          ; First response header
[ 17.404] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

But when the client is the browser, I'm stuck again with the upload taking the ~38s.

I have proxy_request_buffering on, nginx handles all the upload, later on sends the request to the fastcgi sock.

I don't understand why the nginx server interacting with nghttp client is taking an acceptable time and the upload goes without interruptions, and when the client is the browser, it takes the ~38s. Does a browser communicates with the server in a different way than nghttp?

Is there a way to debug the communications between a browser and a server? The developer tools doesn't seem to provide information like the log I posted.

How does nginx handle uploads on http2? Anybody can shed some light about it? Or give me a clue about what to look for?

Answer 1

HTTP/2 upload buffer autotuning is the patch proposed by Cloudflare. Detailed problem explanation and fix at

https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/

"Each HTTP/2 stream has its own flow control window and there is connection level flow control for all streams in the connection. If it’s too tight, the sender will be blocked by the flow control"

a quick fix is disabling http/2 or applying the patch.

It may not be a solution, adding more details on what I found.