HAProxy: Use saved state to restore backend IPs possible?

Question

We are using HAProxy 1.8.13 on Centos 7 with dynamically assigned backends (backend IPs and ports assigned over stats socket). This works fine.

We need a method to make the dynamically assigned stuff stick over restarts and wanted to use the "load-server-state-from-file" directive. Sadly we are running into an error (or maybe its by design?) that the state file does not restore the configured IP address:

Our test config:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    nbproc      2
    stats socket /run/haproxy/1.sock mode 0744 level admin process 1
    stats socket /run/haproxy/2.sock mode 0744 level admin process 2
    server-state-file /run/haproxy/server_state

defaults
    load-server-state-from-file global
    timeout server 10s
    timeout client 15s
    timeout queue 6s
    timeout connect 10s

frontend  main
    bind *:5000
    default_backend             app

backend app
    balance     roundrobin
    server  app1 127.0.0.1:5001 check
    server  app2 127.0.0.1:5002 check
    server  app3 127.0.0.1:5003 check
    server  app4 127.0.0.1:5004 check

We are configuring the IPs via script and then saving the state with:

echo "show servers state" | socat /run/haproxy/1.sock - > /run/haproxy/server_state

Which produces for example:

1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
7 app 1 app1 127.0.0.1 0 1 1 1 60 8 2 0 14 0 0 0 - 5001
7 app 2 app2 10.10.10.115 2 0 1 1 23 6 3 4 6 0 0 0 - 31501
7 app 3 app3 10.10.10.113 2 0 1 1 22 6 3 4 6 0 0 0 - 31375
7 app 4 app4 10.10.10.114 2 0 1 1 22 6 3 4 6 0 0 0 - 31400

When haproxy restarts it restores up/down state information and port, but resets the IP to 127.0.0.1:

1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
7 app 1 app1 127.0.0.1 0 1 1 1 7 8 2 0 14 0 0 0 - 5001
7 app 2 app2 127.0.0.1 0 0 1 1 2 8 2 0 6 0 0 0 - 31501
7 app 3 app3 127.0.0.1 0 0 1 1 2 8 2 0 6 0 0 0 - 31375
7 app 4 app4 127.0.0.1 0 0 1 1 1 8 2 0 6 0 0 0 - 31400

We toyed around with "init-addr" but this only affects DNS based backend addresses. Do we do something wrong? It this expected behavior? Or is this some kind of bug?

Answer 1

We found out it works when we replace '127.0.0.1' with 'localhost'.

HAProxy seems to ignore backend IPs in the state file, at least with the versions we tested (1.8.7, 1.8.13, 1.9-dev). Backend servers that have DNS names as configured default do get the IPs from the state file applied.

We took a deep dive in the code and opened a bug with a patch on the HAProxy ML here.