Following is mostly a "theoretical" kind of answer:
I suppose what you observe is the double-fold issue: the nature of your test (requesting particularly small files), and, Chrome behavior/implementation of HTTP/2.
If you take a look at your HTTP/2 image, you can see that quite many (18) requests end at the same time. This may tell you that even though they don't start at the same time, they run in parallel.
By the nature of HTTP/2 multiplexing:
it’s even possible to intermingle parts of one message with another on the wire
So it appears as if this is what Chrome is doing - sees an unfinished request to server, and attempts to make another request over the same multiplexed connection, in parallel.
For some reason, your server has some sort of "scaling" issue with the parallel requests.
You said that
foo.txt is a small file, but how exactly small it is?
There is http2_chunk_size directive (default value is
8k), which would mean that HTTP/2 response is spit in chunks of that size, and if the requests indeed run in parallel, and
foo.txt is smaller than
8k, then it will result in "waiting" of one request for
foo.txt to complete another request for
foo.txt, until a total of
8k chunk is accumulated for output to browser.
I would suggest playing with that directive and lowering it to be smaller than requested file sizes, and/or repeating the tests with a text file that is larger than
Finally, if the connection is not reliable, you might be hitting the problem of TCP HOL blocking, which is worse on HTTP/2. To quote here:
But are there scenarios where it could worsen the situation? Yes. The reason is that applications using HTTP/2 can send many more requests over a single TCP connection due to the pipelining/multiplexing features. Unexpectedly large variations in latency or a loss affecting the segment at the TCP head of line makes HTTP/2 more likely to hit HOLB, and its impact is larger as well. Basically, the receiver stands idle till the head is recovered, while all the subsequent segments are held by the TCP stack. This means that an image might be downloaded successfully and still not show up due to HOLB.