I'm getting sporadic 502s returned by a proxy server. When inspecting the packet flow I see nginx sending a POST request to a socket that the origin server already sent a [FIN,ACK]. I want to understand how is this possible and any potential solutions. Is it the issue with the origin (it sends the FIN,ACK only after 5 seconds after sending the response) or on the proxy?
Here is the screenshot of the PCAP that is illustrative of the problem:
My understanding:
- the response from the origin is a [PSH, ACK];
- proxy sends an [ACK] for the data received with that [P.] (wireshark confirms that the next [ACK] is for the [PSH-ACK] received before);
- 7 seconds passed (take note of the timestamp btw/ the [FIN, ACK] and our POST ([PSH, ACK]));
- origin sends a [FIN, ACK]. When the first [FIN, ACK] is sent the origin TCP state machine should be in FIN_WAIT_1 state.
- then we send another POST causing a [RST] in return since the origin was not expecting a [PSH, ACK].
Question:
- What is the possible explanation for this case?
- Why is the proxy (nginx) sending another request if it already received a FIN and is actually acknowledging it! (the ack number in the POST [PSH, ACK] packet is actually SEQ_NUMBER + 1 of the [FIN,ACK] - so it's acknowledging the phantom bit FIN.
- What are the possible reasons for the origin returning a [FIN,ACK] only after 5 seconds and not immediatelly? Read-timeout / idle-timeout?
I do not own the origin - so can not capture there.
Additional details:
The error log on the proxy (nginx error log):
2017/04/17 06:51:07 [error] 123091#0: *225010841 upstream prematurely closed connection while reading response header from upstream, client: X.90.10, server: www.example.com, request: "POST /web/?a=b HTTP/1.1", upstream: "http://X.32.238:80/web/?a=b", host: "www.example.com"
The SEQ and ACK numbers are shown for the last requests in this screenshot: