I've got an Nginx reverse proxy sitting in front of a cluster of microservices. When a new client connects I'd ideally like to use whatever upstream has the least number of connections. They'll receive a token in the response body that they'll pass in the header for all subsequent requests.
I would like the same upstream server to handle all future requests related to that token.
Using "ip_hash" won't work because I need to load balance requests from the same origin IP. (But w/ different header tokens) I can't hardcode a "map" because the microservices are transient. Each request already has to pass through an "auth_request" that is capable of exposing a variable with the proper upstream IP to connect to but I'm not sure how to proxy pass to a dynamic variable.
I'm already using Redis and Memcached for a number of other tasks and I could easily load a "key = header token", "value = upstream IP" into either. But once again I'm not sure how to use a cache value as a proxy upstream target. Most of the cache based solutions I've seen can only return the cache value as a client response, not use it within a "location" block for further routing.
I'm open to any ideas on how to pin a request to a specific upstream using a header value that doesn't rely on hardcoding a map or IP/URI hashing.
Thanks in advance, I've been stuck on this for over 5 hours. :(