92

I am using nginx/0.7.68, running on CentOS, with the following configuration:

server {
    listen       80;
    server_name ***;
    index index.html index.htm index.php default.html default.htm default.php;

    location / {
            root   /***;
            proxy_pass   http://***:8888;
            index  index.html index.htm;
    }
    # where *** is my variables

The proxy_pass is to a DNS record whose IP changes frequently. Nginx caches the outdated IP address, resulting in a request to the wrong IP address.

How can I stop nginx from caching the IP address, when it is outdated?

4
9

It's an intriguing question and AFAIK that's not going to work well. You can try to use the upstream module and use the directives for failover to see if it works as a hack.

2018 edit: a lot of things changed. Check the answer by @ohaal to get real information about this.

4
  • 1
    surprisingly when i changed to upstream, everything worked as expected. I'll then mark this as correct answer
    – xiamx
    Feb 27 '11 at 22:20
  • 1
    According to the documentation, there's a special upstream server flag resolve that's only available in the commercial version (see nginx.org/en/docs/http/ngx_http_upstream_module.html#server)
    – omribahumi
    Jul 19 '15 at 13:48
  • 1
    @gansbrest that site seems to be some kind of spammy site? i'd ask that you remove your response.
    – chizou
    Jun 28 '18 at 0:02
  • It's a shame that in 2020 the upstream module still doesn't fix this issue. Resolving the address once and using that forever is useless.
    – Phil
    Sep 3 '20 at 8:23
145

Accepted answer didn't work for me on nginx/1.4.2.

Using a variable in proxy_pass forces re-resolution of the DNS names because NGINX treats variables differently to static configuration. From the NGINX proxy_pass documentation:

Parameter value can contain variables. In this case, if an address is specified as a domain name, the name is searched among the described server groups, and, if not found, is determined using a resolver.

For example:

server {
    ...
    resolver 127.0.0.1;
    set $backend "http://dynamic.example.com:80";
    proxy_pass $backend;
    ...
}

Note: A resolver (i.e. the name server to use) MUST be available and configured for this to work (and entries inside a /etc/hosts file won't be used in a lookup).

By default, version 1.1.9 or later versions of NGINX cache answers using the TTL value of a response and an optional valid parameter allows the cache time to be overridden:

resolver 127.0.0.1 [::1]:5353 valid=30s;

Before version 1.1.9, tuning of caching time was not possible, and nginx always cached answers for the duration of 5 minutes..

16
  • wouldn't this force a dns query on every single request? that sounds like awful performance...
    – lucascaro
    Mar 10 '15 at 3:42
  • 22
    After spending most of my day on this - on Ubuntu 12.04 with nginx 1.1.19, set inside location doesn't work properly. Beware
    – omribahumi
    Jul 19 '15 at 13:38
  • 13
    Note: for docker, it's DNS resolver resides at 127.0.0.11, so for development, I use this: resolver 127.0.0.11 [::1]:5353 valid=15s; Jun 1 '18 at 11:18
  • 7
    Here's a really nasty gotcha that lost me over a day trying to debug: If you use a variable in the proxy_pass directive, the interpretation of the path changes. When you do this, the path of the original request is completely overridden and replaced by the path given in proxy_pass. You must use a proxy_pass without a path in the url in order to avoid this behaviour. Mar 2 '20 at 1:02
  • 1
    I had to change it to set $backend "http://dynamic.example.com:80$request_uri";, otherwise the request path will be empty (as the comment above me also mentions)
    – gilad905
    Dec 9 '20 at 12:41
12

There is valuable information in gansbrest comment and ohaal answer.

But I think it's important to mention this official nginx article, posted in 2016, it clearly explains nginx behaviour on this matter and the possible solutions: https://www.nginx.com/blog/dns-service-discovery-nginx-plus/

We indeed have to "Set the Domain Name in a Variable" and use the resolver directive.

however, using a variable changes the rewrite behaviour. You may have to use the rewrite directive, it depends on your location and proxy_pass setup.

PS: would have post a comment but not enough points yet...

9

ohaal's answer takes most of us there, but there is a case where the DNS resolver does not live at 127.0.0.1 (eg when you're in a special containerized environment)

In that case, you may want to change the nginx conf to resolver ${DNS_SERVER};. Then, before you start nginx, run

export DNS_SERVER=$(cat /etc/resolv.conf |grep -i '^nameserver'|head -n1|cut -d ' ' -f2)
envsubst '${DNS_SERVER}' < your_nginx.conf.template > your_nginx.conf

Note, that you need the gettext package installed, as that provides the envsubst command.

1
  • awk '/nameserver/{a=(a?a" "$2:$2)} END{print a}' /etc/resolv.conf 2> /dev/null
    – samm
    Feb 26 '21 at 6:06
2

I've hacked together a script to watch a conf.d folder upstreams for dns changes and reload nginx upon detection. It's a first pass, and surely can be improved (next pass, I'll use nginx -T to parse upstreams specifically. Same idea could be used for proxy_pass directives):

#!/bin/bash

get_upstreams() {
  local files=$@
  grep -hEo '(server\s+)[^:;]+' $files | cut -d' ' -f 2
}

resolve_hosts() {
  local hosts=$@
  for h in $hosts; do dig +short $h; done | sort -u
}

watch_dir=$1

[ -d $watch_dir ] || exit 2

upstreams=$(get_upstreams $watch_dir/*)
ips=$(resolve_hosts $upstreams)
if [ ! "$ips" ]; then
  echo "Found no resolvable hosts in $watch_dir files."
fi

host_hash=$(echo $ips | /usr/bin/sha512sum)

echo $host_hash
echo $ips

while [ -d $watch_dir ]; do
  sleep 30
  upstreams=$(get_upstreams $watch_dir/*)
  ips=$(resolve_hosts $upstreams)
  new_hash=$(echo $ips | /usr/bin/sha512sum)
  if [ "$host_hash" != "$new_hash" ]; then
    echo Detected an upstream address change.  $ips
    echo Reloading nginx
    echo $new_hash
    echo $ips
    /sbin/service nginx reload
    host_hash=$new_hash
  fi
done
1
  • do you have a solution which works with nginx -T and proxy_pass?
    – nflaig
    Jun 20 '21 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.