NGINX: Failover to backup in ip_hashed upstream

Question

I'm using NGINX as a load balancer to run multiple instances of my site. I'm working on conditional routing to send users to specific servers based on an incoming HTTP header. The end goal is to have a beta upstream with X servers load-balanced using ip_hashing to send to users who have the beta tester flag and a stable upstream with Y servers for those who haven't opted in.

My issue is with fail-over from one group to another -- specifically, beta users should be re-routed to stable servers if no beta servers are available. Unfortunately, neither the backup nor weight directives are working within an ip_hashed upstream despite the current NGINX documentation seeming to indicate that they should work.

As a workaround, I'm trying to have a custom error location for 502's and 504's that does a proxy_pass to my stable upstream. However, when I test my configuration I find that the page is blank with a 504 header and no redirect. Basing my current solution off of this thread:

worker_processes  2;

events {
    worker_connections  1024;
}


http {
    upstream dataserver_stable {
        ip_hash;
        server localhost:8081;
    }

    upstream dataserver_beta {
        ip_hash;
        server localhost:8082 fail_timeout=1m;
        #server localhost:8081 backup; # backup and weight aren't working with the ip_hash directive
    }

    geo $beta_user {
        default stable;
        10.17.12.246 beta;
    }

    map $beta_user $upstream {
        stable dataserver_stable;
        beta dataserver_beta;
    }

    server {
        listen       81;
        server_name  localhost;

        set_real_ip_from 27.0.0.1;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;

        location / {
                error_page 502 504 @fallback;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $realip_remote_addr;
                proxy_pass       http://$upstream;
                proxy_connect_timeout 5s;
                proxy_read_timeout 10s;
        }

        location @fallback {
            proxy_pass  http://dataserver_stable;
        }
    }

}

Any advice would be appreciated.

Answer 1

Although this post is 6 years old, I came across this same need for the 'hash consistent' load-balancing feature of nginx. Based on this nginx example config, I believe what you are missing is the directive proxy_intercept_errors on; so that nginx consumes the error rather than passing it to the client.

In full, your config would look like this:

worker_processes  2;

events {
    worker_connections  1024;
}


http {
    upstream dataserver_stable {
        ip_hash;
        server localhost:8081;
    }

    upstream dataserver_beta {
        ip_hash;
        server localhost:8082 fail_timeout=1m;
        #server localhost:8081 backup; # backup and weight aren't working with the ip_hash directive
    }

    geo $beta_user {
        default stable;
        10.17.12.246 beta;
    }

    map $beta_user $upstream {
        stable dataserver_stable;
        beta dataserver_beta;
    }

    server {
        listen       81;
        server_name  localhost;

        set_real_ip_from 27.0.0.1;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;

        location / {
                error_page 502 504 @fallback;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $realip_remote_addr;
                proxy_pass       http://$upstream;
                proxy_connect_timeout 5s;
                proxy_read_timeout 10s;
        }

        location @fallback {
            proxy_pass_intercept_errors on;
            proxy_pass  http://dataserver_stable;
        }
    }

}