HAproxy simple reverse proxy very slow

Question

I'm currently setting up an HAproxy machine to act as a reverse proxy for a couple of websites. As the different machines are not physically connected I'm using OpenVPN to create a VPN between the machines.

Technically, everything is working. However, my problem is that requests that I pipe through the proxy are extremely slow. I get the complete response(s) eventually, but it takes up to 5 minutes to receive something like a Jenkins dashboard.

I tried a simple static HTML page served via NGINX as well as some REST API implemented in go - the result is always the same: I get the data eventually but it takes a ridiculous amount of time.

Here's my HAproxy config:

  global
          log /var/run/log local0 info
          log /var/run/log local0 notice
          daemon
          maxconn 8000
          tune.ssl.default-dh-param 2048
          user nobody
          group nobody

  defaults
          log global
          option httplog
          option dontlognull
          mode http
          timeout connect 5s
          timeout client 1min
          timeout server 1min
          option forwardfor
          option http-server-close
          errorfile 400 /usr/local/etc/haproxy/errorfiles/400.http
          errorfile 403 /usr/local/etc/haproxy/errorfiles/403.http
          errorfile 408 /usr/local/etc/haproxy/errorfiles/408.http
          errorfile 500 /usr/local/etc/haproxy/errorfiles/500.http
          errorfile 502 /usr/local/etc/haproxy/errorfiles/502.http
          errorfile 503 /usr/local/etc/haproxy/errorfiles/503.http
          errorfile 504 /usr/local/etc/haproxy/errorfiles/504.http

  frontend http-in
          bind *:80
          bind *:443 ssl crt /usr/local/etc/haproxy/certs/foo.my.org.pem
          mode http
          use_backend jenkins if { hdr(host) -i foo.my.org }
          use_backend test if { hdr(host) -i bar.my.org }
          default_backend test

  backend jenkins
          server jenkins1 <vpn_ip>:8180
          mode http
          http-request set-header X-Forwarded-Port %[dst_port]
          http-request add-header X-Forwarded-Proto https if { ssl_fc }
          reqrep ^([^\ :]*)\ /(.*)     \1\ /\2
          acl response-is-redirect res.hdr(Location) -m found
          rspirep ^Location:\ (http)://<vpn_ip>:8180/(.*)   Location:\ https://foo.my.org:443/\2  if response-is-redirect

  backend test
          server web01 <vpn_ip>:80

Additional information:

• All involved machines are located in data centers and feature a 1G/1G internet connection
• All machines run FreeBSD 11 64-Bit
• OpenVPN version 2.4.4 on all machines
• HAproxy version 1.7.9
• OpenVPN runs in TCP mode
• The HAproxy backend config for Jenkins is taken as provided from the Jenkins documentation but as mentioned the same problem exists with a bare static HTML content webserver and a REST API web service.

And here's some HAproxy log when accessing the Jenkins site:

Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:60243 [27/Feb/2018:01:32:24.093] http-in~ jenkins/jenkins1 0/0/13/134/161 302 153 - - ---- 5/5/0/0/0 0/0 "GET /jenkins HTTP/1.1"
Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:19404 [27/Feb/2018:01:32:24.255] http-in~ jenkins/jenkins1 0/0/25/174/212 200 4492 - - ---- 5/5/0/0/0 0/0 "GET /jenkins/ HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:16321 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/13/30/54 200 8560 - - ---- 6/6/4/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/datasource/datasource-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:54637 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/27/29/58 200 7585 - - ---- 6/6/3/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/autocomplete/autocomplete-min.js HTTP/1.1"Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:59247 [27/Feb/2018:01:32:25.361] http-in~ jenkins/jenkins1 0/0/25/16/51 200 9602 - - ---- 6/6/2/3/0 0/0 "GET /jenkins/static/aeed77bb/jsbundles/page-init.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:40637 [27/Feb/2018:01:32:25.332] http-in~ jenkins/jenkins1 0/0/38/18/81 200 16212 - - ---- 6/6/1/1/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/menu/menu-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:10976 [27/Feb/2018:01:32:25.333] http-in~ jenkins/jenkins1 0/0/37/30/95 200 29110 - - ---- 6/6/0/0/0 0/0 "GET /jenkins/static/aeed77bb/scripts/hudson-behavior.js HTTP/1.1"

I don't see anything else in the log files. No errors or similar.

Can this be due to the OpenVPN?

Edit 1: In the mean time I tested this without OpenVPN by directly using the web server's public IP address in the HAproxy configuration. The result is exactly the same.

Answer 1

Your log doesn't include enough information to explain the 5 minute wait -- it spans less than 2 seconds -- but here is what jumps out:

option http-server-close

This is not appropriate with a back-end on the other side of a WAN link. If you actually have a need for this feature, it needs to be on an HAProxy with extremely low latency (e.g., local) to the back-end server.

Note the large amount of jitter in your Tc timer values. This is the third value in the log field that looks like this:

Tq/Tw/Tc/Tr/Tt

So, in the first entry, 0/0/13/134/161, the Tc value is 13 ms. Tc is the time required to establish the connection to the back-end, so what we see here is that your round-trip jitter is pretty much out of control, jumping in one case to almost three times this value. Assuming this isn't caused by insufficient resources on the back-end server, this is likely to be a consequence of using OpenVPN in TCP mode -- possibly, but not necessarily, indicating some packet loss.

You're tunneling TCP (HTTP) inside TCP (OpenVPN), which is usually fine for non-interactive applications and applications where jitter is unimportant, but not otherwise. (See, for example, Why TCP over TCP is a Bad Idea for discussion of one of the potential meltdown scenarios, though not sufficient reason to avoid this configuration entirely on a clean network.) Through what I assume to be no fault of OpenVPN, TCP tunnels do not have the same consistent latency as UDP tunnels. Ping through the tunnel and you should see this jitter imparted to the ping response times.

Change the tunnel to UDP and the issue should disappear... but note that the significance and performance impact of this jitter -- assuming I am correct that it is a side effect of using a TCP tunnel -- is probably exaggerated by using option http-server-close, which requires that each request establish a new connection between the HAProxy and the back-end.

If your OpenVPN tunnels are compressed using comp-lzo, then consider disabling that, as well, since poteblntial cost savings in transport may be outweighed by the overhead.

Answer 2

This turned out to be an issue with the firewall/NAT configuration.

However, if anybody comes across this issue, there's some more help over at the HAproxy community: https://discourse.haproxy.org/t/reverse-proxy-very-slow-page-load/2172