0

I'm currently setting up an HAproxy machine to act as a reverse proxy for a couple of websites. As the different machines are not physically connected I'm using OpenVPN to create a VPN between the machines.

Technically, everything is working. However, my problem is that requests that I pipe through the proxy are extremely slow. I get the complete response(s) eventually, but it takes up to 5 minutes to receive something like a Jenkins dashboard.

I tried a simple static HTML page served via NGINX as well as some REST API implemented in go - the result is always the same: I get the data eventually but it takes a ridiculous amount of time.

Here's my HAproxy config:


  global
          log /var/run/log local0 info
          log /var/run/log local0 notice
          daemon
          maxconn 8000
          tune.ssl.default-dh-param 2048
          user nobody
          group nobody

  defaults
          log global
          option httplog
          option dontlognull
          mode http
          timeout connect 5s
          timeout client 1min
          timeout server 1min
          option forwardfor
          option http-server-close
          errorfile 400 /usr/local/etc/haproxy/errorfiles/400.http
          errorfile 403 /usr/local/etc/haproxy/errorfiles/403.http
          errorfile 408 /usr/local/etc/haproxy/errorfiles/408.http
          errorfile 500 /usr/local/etc/haproxy/errorfiles/500.http
          errorfile 502 /usr/local/etc/haproxy/errorfiles/502.http
          errorfile 503 /usr/local/etc/haproxy/errorfiles/503.http
          errorfile 504 /usr/local/etc/haproxy/errorfiles/504.http

  frontend http-in
          bind *:80
          bind *:443 ssl crt /usr/local/etc/haproxy/certs/foo.my.org.pem
          mode http
          use_backend jenkins if { hdr(host) -i foo.my.org }
          use_backend test if { hdr(host) -i bar.my.org }
          default_backend test

  backend jenkins
          server jenkins1 <vpn_ip>:8180
          mode http
          http-request set-header X-Forwarded-Port %[dst_port]
          http-request add-header X-Forwarded-Proto https if { ssl_fc }
          reqrep ^([^\ :]*)\ /(.*)     \1\ /\2
          acl response-is-redirect res.hdr(Location) -m found
          rspirep ^Location:\ (http)://<vpn_ip>:8180/(.*)   Location:\ https://foo.my.org:443/\2  if response-is-redirect

  backend test
          server web01 <vpn_ip>:80

Additional information:

  • All involved machines are located in data centers and feature a 1G/1G internet connection
  • All machines run FreeBSD 11 64-Bit
  • OpenVPN version 2.4.4 on all machines
  • HAproxy version 1.7.9
  • OpenVPN runs in TCP mode
  • The HAproxy backend config for Jenkins is taken as provided from the Jenkins documentation but as mentioned the same problem exists with a bare static HTML content webserver and a REST API web service.

And here's some HAproxy log when accessing the Jenkins site:

Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:60243 [27/Feb/2018:01:32:24.093] http-in~ jenkins/jenkins1 0/0/13/134/161 302 153 - - ---- 5/5/0/0/0 0/0 "GET /jenkins HTTP/1.1"
Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:19404 [27/Feb/2018:01:32:24.255] http-in~ jenkins/jenkins1 0/0/25/174/212 200 4492 - - ---- 5/5/0/0/0 0/0 "GET /jenkins/ HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:16321 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/13/30/54 200 8560 - - ---- 6/6/4/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/datasource/datasource-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:54637 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/27/29/58 200 7585 - - ---- 6/6/3/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/autocomplete/autocomplete-min.js HTTP/1.1"Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:59247 [27/Feb/2018:01:32:25.361] http-in~ jenkins/jenkins1 0/0/25/16/51 200 9602 - - ---- 6/6/2/3/0 0/0 "GET /jenkins/static/aeed77bb/jsbundles/page-init.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:40637 [27/Feb/2018:01:32:25.332] http-in~ jenkins/jenkins1 0/0/38/18/81 200 16212 - - ---- 6/6/1/1/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/menu/menu-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:10976 [27/Feb/2018:01:32:25.333] http-in~ jenkins/jenkins1 0/0/37/30/95 200 29110 - - ---- 6/6/0/0/0 0/0 "GET /jenkins/static/aeed77bb/scripts/hudson-behavior.js HTTP/1.1"
I don't see anything else in the log files. No errors or similar.

Can this be due to the OpenVPN?

Edit 1: In the mean time I tested this without OpenVPN by directly using the web server's public IP address in the HAproxy configuration. The result is exactly the same.

  • Why not using NginX out of the box? – Bert Feb 27 '18 at 14:33
  • Because I need load balancing for some of the back-ends that will follow. – Joel Bodenmann Feb 28 '18 at 7:33
0

Your log doesn't include enough information to explain the 5 minute wait -- it spans less than 2 seconds -- but here is what jumps out:

option http-server-close

This is not appropriate with a back-end on the other side of a WAN link. If you actually have a need for this feature, it needs to be on an HAProxy with extremely low latency (e.g., local) to the back-end server.

Note the large amount of jitter in your Tc timer values. This is the third value in the log field that looks like this:

Tq/Tw/Tc/Tr/Tt

So, in the first entry, 0/0/13/134/161, the Tc value is 13 ms. Tc is the time required to establish the connection to the back-end, so what we see here is that your round-trip jitter is pretty much out of control, jumping in one case to almost three times this value. Assuming this isn't caused by insufficient resources on the back-end server, this is likely to be a consequence of using OpenVPN in TCP mode -- possibly, but not necessarily, indicating some packet loss.

You're tunneling TCP (HTTP) inside TCP (OpenVPN), which is usually fine for non-interactive applications and applications where jitter is unimportant, but not otherwise. (See, for example, Why TCP over TCP is a Bad Idea for discussion of one of the potential meltdown scenarios, though not sufficient reason to avoid this configuration entirely on a clean network.) Through what I assume to be no fault of OpenVPN, TCP tunnels do not have the same consistent latency as UDP tunnels. Ping through the tunnel and you should see this jitter imparted to the ping response times.

Change the tunnel to UDP and the issue should disappear... but note that the significance and performance impact of this jitter -- assuming I am correct that it is a side effect of using a TCP tunnel -- is probably exaggerated by using option http-server-close, which requires that each request establish a new connection between the HAProxy and the back-end.

If your OpenVPN tunnels are compressed using comp-lzo, then consider disabling that, as well, since poteblntial cost savings in transport may be outweighed by the overhead.

  • I followed your advice and remove the http-server-close option from HAproxy and I changed the OpenVPN configuration to use UDP instead of TCP. Unfortunately, I don't see any difference. Here's the full log of loading the jenkins dashboard: paste. It takes 10.9 minutes to finish loading the page (Finish: 10.9 min, DOMContentLoaded: 6.0 min, Load 10.9 min) according to chrome and the HAproxy show 6 minutes. When I ping my back-end from the HAproxy (over VPN) I now have a very consistent ping of 13 ms. The back-end has more than enough resources. – Joel Bodenmann Feb 27 '18 at 8:43
  • In the meantime I tried this without OpenVPN by just using the web server's public IP address in the HAproxy backend config and the result is exactly the same. – Joel Bodenmann Feb 27 '18 at 12:14
  • Your logs suggest that both changes were made and accomplished their intended purpose. Unfortunately, I now see nothing troubling in the log that suggests the problem lies with either the proxy or the back-end... everything the browser requests seems to load quickly, leaving unexplained the reason why the browser waits so long before issuing certain requests. Googling "Jenkins dashboard slow" seems to suggest that while the proxy and tunnel may not have been optimal, initially, they may not be the real issue at all. – Michael - sqlbot Feb 27 '18 at 12:32
  • I'm not trying to point fingers, with that analysis... just suggesting that the logs seem to suggest that the browser simply isn't waiting on this, and must be busy doing... something else. But what? Also, I'm unclear what "the same problem exists with a bare static HTML content webserver and a REST API web service" means in the original question. – Michael - sqlbot Feb 27 '18 at 12:34
  • Well, I tried this with a regular wordpress website now. When I access the wordpress website via the normal public IP of the web server it loads within less than a second. When I do this over HAproxy it takes 6.9 minutes chrome screenshot and this is the corresponding HAproxy log: paste This is the my current HAproxy config: paste I get the same result with various other websites. – Joel Bodenmann Feb 27 '18 at 17:55
0

This turned out to be an issue with the firewall/NAT configuration.

However, if anybody comes across this issue, there's some more help over at the HAproxy community: https://discourse.haproxy.org/t/reverse-proxy-very-slow-page-load/2172

  • If there is anything at that link that would be useful, it should be edited in to your answer. Links have a way of going bad over time. – Michael Hampton Mar 6 '18 at 21:18

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.