We have a strange HA Proxy issue. Although we solved it, we don't really understand why.
The setup is very basic: a load balancer and an Apache backend. Our problem arises for a http request which takes 100 minutes running on Apache before any output is send back.
In the first setup we had
timeout server of 30 minutes. In that case a 504 error was presented by HAProxy, in the log showed
sH--. According to the docs:
The "timeout server" stroke before the server could return its response headers.
So we raised the timeout server to 60 minutes. This time we got (after 100 minutes) a 502 error and in the log
The server aborted before sending its full HTTP response headers, or it crashed while processing the request.
At first we suspected Apache crashing, but then we noticed that after 60 minutes Apache received a second request from HAProxy which didn't show up in any log file. Could it be that what happened was the following:
- Request send to HAProxy, and from there to Apache
- After 60 minutes, HA Proxy decides that the connection is bad and resends a request (I guess with a different source port). Apache starts processing this second request.
- After 40 minutes, Apache finishes the first attempt and sends the results back to HAProxy. HAProxy gets confused as it is a response from the first attempt, so it aborts with a 502 and logs a retry count 0 (as we saw in the log).
BTW Only after we raised timeout server to 2 hours, the request processed without errors (nor retry).
Stripped HAProxy 1.5.9 config:
global maxconn 4000 # Sets the maximum per-process number of concurrent connections. maxsslconn 1000 # Sets the maximum per-process number of concurrent SSL connections. maxcompcpuusage 95 # Sets the maximum CPU usage HAProxy can reach before stopping or decreasing the compression level. defaults HTTP mode http option http-server-close # Preserve client persistent connections while handling every incoming request individually, dispatching them one after another to servers, in HTTP close mode option httplog option forwardfor timeout connect 4s timeout client 20s timeout server 100s timeout http-request 20s # Set the maximum allowed time to wait for a complete HTTP request maxconn 200 default-server inter 2s fall 6 rise 2 port 80 frontend dc2--fe bind 126.96.36.199:80 mss 1422 default_backend dc2--active backend dc2--active timeout server 7200s server app37 10.0.0.97:80 check