1

We have a strange HA Proxy issue. Although we solved it, we don't really understand why.

The setup is very basic: a load balancer and an Apache backend. Our problem arises for a http request which takes 100 minutes running on Apache before any output is send back.

In the first setup we had timeout server of 30 minutes. In that case a 504 error was presented by HAProxy, in the log showed sH--. According to the docs:

The "timeout server" stroke before the server could return its response headers.

So we raised the timeout server to 60 minutes. This time we got (after 100 minutes) a 502 error and in the log SH--:

The server aborted before sending its full HTTP response headers, or it crashed while processing the request.

At first we suspected Apache crashing, but then we noticed that after 60 minutes Apache received a second request from HAProxy which didn't show up in any log file. Could it be that what happened was the following:

  1. Request send to HAProxy, and from there to Apache
  2. After 60 minutes, HA Proxy decides that the connection is bad and resends a request (I guess with a different source port). Apache starts processing this second request.
  3. After 40 minutes, Apache finishes the first attempt and sends the results back to HAProxy. HAProxy gets confused as it is a response from the first attempt, so it aborts with a 502 and logs a retry count 0 (as we saw in the log).

BTW Only after we raised timeout server to 2 hours, the request processed without errors (nor retry).

Stripped HAProxy 1.5.9 config:

global
   maxconn         4000     # Sets the maximum per-process number of concurrent connections.
   maxsslconn      1000     # Sets the maximum per-process number of concurrent SSL connections.
   maxcompcpuusage 95       # Sets the maximum CPU usage HAProxy can reach before stopping or decreasing the compression level.

defaults HTTP
   mode            http
   option          http-server-close        # Preserve client persistent connections while handling every incoming request individually, dispatching them one after another to servers, in HTTP close mode
   option          httplog
   option          forwardfor
   timeout connect 4s       
   timeout client  20s      
   timeout server  100s
   timeout http-request 20s  # Set the maximum allowed time to wait for a complete HTTP request
   maxconn         200
   default-server  inter 2s fall 6 rise 2 port 80

frontend dc2--fe
   bind            8.8.8.8:80 mss 1422
   default_backend dc2--active

backend dc2--active
   timeout server  7200s
   server          app37 10.0.0.97:80 check
  • I'm curious why you would have a webserver synchronously process a request for 100 minutes? – Koen. May 18 '19 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.