0

My haproxy server running on Google Cloud has an odd behavior. Sometimes, there are many client connections, but it does not forward these requests to backends properly, making client wait time longer (sometimes with timeouts). Why this is happening?

Here is the output of the hatop command. Notice the SCUR column has 100 connnections on frontend but 0 connections on backend. And there are 154 tasks.

  HAProxy Version: 1.8.8-1ubuntu0.4  (released: 2019/01/PID: 2016 (proc 1)

         Node: haproxy-b (uptime 0d 0h26m18s)

        Pipes: [                                                       0/0]
  Connections: [|||||||||||||||||||||||||||||||||||||||||||||||||||100/100]

  Procs:   1   Tasks:   154    Queue:     0    Proxies:   1   Services:   52

NAME                    W STATUS      CHECK             ACT      BCK      QCUR      QMAX       SCUR       SMAX       SLIM       STOT

>>> new-fp
FRONTEND                0 OPEN                            0        0         0         0        100        100        100      26936
..t10.240.0.170         1 UP          L7OK                1        0         0         0          0          2          0        519
..t10.240.0.119         1 UP          L7OK                1        0         0         0          0          3          0        548
..t10.240.0.118         1 UP          L7OK                1        0         0         0          0          3          0        572
..t10.240.0.173         1 UP          L7OK                1        0         0         0          0          2          0        537
..t10.240.0.177         1 UP          L7OK                1        0         0         0          0          2          0        525
..t10.240.0.176         1 UP          L7OK                1        0         0         0          0          2          0        546
..t10.240.0.106         1 UP          L7OK                1        0         0         0          0          2          0        541
..t10.240.0.178         1 UP          L7OK                1        0         0         0          0          2          0        540
..t10.240.0.124         1 UP          L7OK                1        0         0         0          0          3          0        542
..nt10.240.0.55         1 UP          L7OK                1        0         0         0          0          3          0        534
..nt10.240.0.93         1 UP          L7OK                1        0         0         0          0          3          0        538
..t10.240.0.148         1 UP          L7OK                1        0         0         0          0          3          0        509
..t10.240.0.179         1 UP          L7OK                1        0         0         0          0          3          0        570
..t10.240.0.122         1 UP          L7OK                1        0         0         0          0          3          0        564
..t10.240.0.108         1 UP          L7OK                1        0         0         0          0          3          0        546
..nt10.240.0.86         1 UP          L7OK                1        0         0         0          0          2          0        491
..t10.240.0.168         1 UP          L7OK                1        0         0         0          0          2          0        493
..nt10.240.0.71         1 UP          L7OK                1        0         0         0          0          2          0        493
..t10.240.0.144         1 UP          L7OK                1        0         0         0          0          3          0        522
..nt10.240.0.80         1 UP          L7OK                1        0         0         0          0          2          0        493
..t10.240.0.143         1 UP          L7OK                1        0         0         0          0          2          0        505
..t10.240.0.172         1 UP          L7OK                1        0         0         0          0          2          0        542
..nt10.240.0.83         1 UP          L7OK                1        0         0         0          0          2          0        514
..t10.240.0.153         1 UP          L7OK                1        0         0         0          0          2          0        548
..t10.240.0.149         1 UP          L7OK                1        0         0         0          0          3          0        516
..t10.240.0.175         1 UP          L7OK                1        0         0         0          0          2          0        544
..t10.240.0.134         1 DOWN        L4TOUT              1        0         0         0          0          2          0        425
..t10.240.0.163         1 UP          L7OK                1        0         0         0          0          2          0        507
..nt10.240.0.75         1 UP          L7OK                1        0         0         0          0          2          0        524
..nt10.240.0.99         1 UP          L7OK                1        0         0         0          0          2          0        517
..int10.240.0.7         1 UP          L7OK                1        0         0         0          0          3          0        509
..t10.240.0.174         1 UP          L7OK                1        0         0         0          0          2          0        546
..nt10.240.0.98         1 UP          L7OK                1        0         0         0          0          3          0        502
..t10.240.0.164         1 UP          L7OK                1        0         0         0          0          2          0        507
..t10.240.0.105         1 UP          L7OK                1        0         0         0          0          2          0        538
..nt10.240.0.17         1 UP          L7OK                1        0         0         0          0          3          0        534
..int10.240.0.3         1 UP          L7OK                1        0         0         0          0          2          0        514
..t10.240.0.166         1 UP          L7OK                1        0         0         0          0          2          0        553
..t10.240.0.180         1 UP          L7OK                1        0         0         0          0          3          0        584
..nt10.240.0.85         1 UP          L7OK                1        0         0         0          0          3          0        558
..t10.240.0.145         1 UP          L7OK                1        0         0         0          0          2          0        516
..nt10.240.0.10         1 UP          L7OK                1        0         0         0          0          3          0        491
..t10.240.0.116         1 UP          L7OK                1        0         0         0          0          2          0        489
..t10.240.0.110         1 UP          L7OK                1        0         0         0          0          2          0        492
..nt10.240.0.62         1 UP          L7OK                1        0         0         0          0          2          0        536
..t10.240.0.171         1 UP          L7OK                1        0         0         0          0          2          0        531
..t10.240.0.167         1 UP          L7OK                1        0         0         0          0          2          0        509
..nt10.240.0.81         1 UP          L7OK                1        0         0         0          0          3          0        524
..nt10.240.0.88         1 UP          L7OK                1        0         0         0          0          2          0        522
..nt10.240.0.67         1 UP          L7OK                1        0         0         0          0          2          0        500
BACKEND                49 UP                             49        0         0         0          0        100         10      26220

here is my haproxy config file:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 100
    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
        timeout connect 10000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


listen new-fp
        bind *:9000
        mode http
        stats enable
        stats uri /haproxy?stats
        stats realm Strictly\ Private
        stats auth *:*
        balance leastconn
        option httpclose
        option forwardfor
        option httpchk GET /index.html
        timeout connect 5000
        maxconn 100
        #server master-fp 10.240.0.2:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.170 10.240.0.170:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.119 10.240.0.119:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.118 10.240.0.118:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.173 10.240.0.173:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.177 10.240.0.177:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.176 10.240.0.176:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.106 10.240.0.106:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.178 10.240.0.178:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.124 10.240.0.124:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.55 10.240.0.55:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.93 10.240.0.93:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.148 10.240.0.148:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.179 10.240.0.179:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.122 10.240.0.122:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.108 10.240.0.108:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.86 10.240.0.86:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.168 10.240.0.168:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.71 10.240.0.71:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.144 10.240.0.144:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.80 10.240.0.80:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.143 10.240.0.143:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.172 10.240.0.172:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.83 10.240.0.83:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.153 10.240.0.153:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.149 10.240.0.149:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.175 10.240.0.175:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.134 10.240.0.134:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.163 10.240.0.163:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.75 10.240.0.75:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.99 10.240.0.99:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.7 10.240.0.7:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.174 10.240.0.174:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.98 10.240.0.98:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.164 10.240.0.164:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.105 10.240.0.105:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.17 10.240.0.17:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.3 10.240.0.3:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.166 10.240.0.166:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.180 10.240.0.180:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.85 10.240.0.85:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.145 10.240.0.145:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.10 10.240.0.10:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.116 10.240.0.116:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.110 10.240.0.110:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.62 10.240.0.62:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.171 10.240.0.171:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.167 10.240.0.167:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.81 10.240.0.81:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.88 10.240.0.88:9000 check inter 5s rise 1 fall 1 weight 1
        server server10.240.0.67 10.240.0.67:9000 check inter 5s rise 1 fall 1 weight 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.