HTTP 502 response generated by a proxy after it tries to send data upstream to a partially closed connection (reset packet)

Question

I'm getting sporadic 502s returned by a proxy server. When inspecting the packet flow I see nginx sending a POST request to a socket that the origin server already sent a [FIN,ACK]. I want to understand how is this possible and any potential solutions. Is it the issue with the origin (it sends the FIN,ACK only after 5 seconds after sending the response) or on the proxy?

Here is the screenshot of the PCAP that is illustrative of the problem:

My understanding:

• the response from the origin is a [PSH, ACK];
• proxy sends an [ACK] for the data received with that [P.] (wireshark confirms that the next [ACK] is for the [PSH-ACK] received before);
• 7 seconds passed (take note of the timestamp btw/ the [FIN, ACK] and our POST ([PSH, ACK]));
• origin sends a [FIN, ACK]. When the first [FIN, ACK] is sent the origin TCP state machine should be in FIN_WAIT_1 state.
• then we send another POST causing a [RST] in return since the origin was not expecting a [PSH, ACK].

Question:

• What is the possible explanation for this case?
• Why is the proxy (nginx) sending another request if it already received a FIN and is actually acknowledging it! (the ack number in the POST [PSH, ACK] packet is actually SEQ_NUMBER + 1 of the [FIN,ACK] - so it's acknowledging the phantom bit FIN.
• What are the possible reasons for the origin returning a [FIN,ACK] only after 5 seconds and not immediatelly? Read-timeout / idle-timeout?

I do not own the origin - so can not capture there.

Additional details:

The error log on the proxy (nginx error log):

2017/04/17 06:51:07 [error] 123091#0: *225010841 upstream prematurely closed connection while reading response header from upstream, client: X.90.10, server: www.example.com, request: "POST /web/?a=b HTTP/1.1", upstream: "http://X.32.238:80/web/?a=b", host: "www.example.com"

The SEQ and ACK numbers are shown for the last requests in this screenshot:

Answer 1

What is the possible explanation for this case?

A race condition between a ~5 seconds idle counter on the origin and variably client-side activity. The third involved variable is of course the network latency.

There seems to be a ~5 seconds idle timer on the origin, while your client is taking ~5 seconds to make a second request (POST) through the Nginx proxy. If the former is longer than the latter (including network latency) you have no problems. If it takes just a teeny bit more for the client request to be sent then you have a problem.

You can see how both the POST and the FIN,ACK from the Nginx are sent pretty much together: 2.4ms and 2.6ms after the origin's FIN,ACK respectively. This could've put you off track here, because I don't think the POST is a response to the origin's FIN,ACK at all. As it is sent 2.4ms after the origin's FIN,ACK

Why is the proxy (nginx) sending another request if it already received a FIN and is actually acknowledging it! (the ack number in the POST [PSH, ACK] packet is actually SEQ_NUMBER + 1 of the [FIN,ACK] - so it's acknowledging the phantom bit FIN.

The ACK number on the POST packet is for the "200 OK" packet most probably. There's no extra data coming from the server side after that HTTP response, so any ACK from the client will be ACKing the same number.

Update: We now know that the POST packet has the ACK number increased in 1, so Nginx knew about the [FIN,ACK]. Further investigation shows this is fine: A machine might send a request and end with a [FIN,ACK] if it's not planning to continue the connection after it receives the response from the remote side, who was to send the requested data back and end with continuing the [FIN,ACK] process.

This doesn't change the fact that there's a race condition where the origin decided to close the connection after 5 seconds idle, thus ignoring the POST packet that comes shortly after (and even sending a RST back - although it's not clear if this RST would've been sent regardless).

What are the possible reasons for the origin returning a [FIN,ACK] only after 5 seconds and not immediatelly? Read-timeout / idle-timeout?

You don't have to return a FIN,ACK immediately, especially since HTTP 1.1 and the introduction of persistent connections. These ~5 seconds seem to be an idle timer on the origin.

Both things are confirmed here: https://en.wikipedia.org/wiki/HTTP_persistent_connection - including a default 5 seconds idle timeout in Apache 2.2 or newer.

Suggested solutions

I can't really suggest a solution without knowing more about your infrastructure, but in rough terms you have a few options:

• Investigate why the client takes 5 seconds to send a second request. Drawbacks: Time consuming and probably implies application changes.
• Increase origin's (Apache?) timeout to maybe 10 seconds. Drawbacks: Problems to scale as you're keeping more resources idle. Might need application changes to dispose of the connections as soon as possible.
• Don't reuse the TCP connection for a second HTTP request, by issuing the "Connection: Close" header. Drawbacks: Higher cost per request as you have to establish a new TCP session. Might entail application changes to issue the header on all requests or changes on Nginx, thus deviating from your default configuration (increased admin costs).
• Use the "keepalive" option on Nginx inside your upstream configuration to set a keepalive lower than 5 seconds. Drawback: Lots of extra traffic/noise.

Hope this helps :)

Answer 2

I think this caused by the upstream server socket keep alive timeout, and socket will be closed，default socket.setsolinger will not open.

I think we can let nginx upstream servers keepalive timeout。Here another author solve it，please see this