108

I am using nginx/0.7.68, running on CentOS, with the following configuration:

server {
    listen       80;
    server_name ***;
    index index.html index.htm index.php default.html default.htm default.php;

    location / {
            root   /***;
            proxy_pass   http://***:8888;
            index  index.html index.htm;
    }
    # where *** is my variables

The proxy_pass is to a DNS record whose IP changes frequently. Nginx caches the outdated IP address, resulting in a request to the wrong IP address.

How can I stop nginx from caching the IP address, when it is outdated?

4

6 Answers 6

10

It's an intriguing question and AFAIK that's not going to work well. You can try to use the upstream module and use the directives for failover to see if it works as a hack.

2018 edit: a lot of things changed. Check the answer by @ohaal to get real information about this.

4
  • 1
    surprisingly when i changed to upstream, everything worked as expected. I'll then mark this as correct answer
    – xiamx
    Feb 27, 2011 at 22:20
  • 1
    According to the documentation, there's a special upstream server flag resolve that's only available in the commercial version (see nginx.org/en/docs/http/ngx_http_upstream_module.html#server)
    – omribahumi
    Jul 19, 2015 at 13:48
  • 1
    @gansbrest that site seems to be some kind of spammy site? i'd ask that you remove your response.
    – chizou
    Jun 28, 2018 at 0:02
  • It's a shame that in 2020 the upstream module still doesn't fix this issue. Resolving the address once and using that forever is useless.
    – Phil
    Sep 3, 2020 at 8:23
161

Accepted answer didn't work for me on nginx/1.4.2.

Using a variable in proxy_pass forces re-resolution of the DNS names because NGINX treats variables differently to static configuration. From the NGINX proxy_pass documentation:

Parameter value can contain variables. In this case, if an address is specified as a domain name, the name is searched among the described server groups, and, if not found, is determined using a resolver.

For example:

server {
    ...
    resolver 127.0.0.1;
    set $backend "http://dynamic.example.com:80";
    proxy_pass $backend;
    ...
}

Note: A resolver (i.e. the name server to use) MUST be available and configured for this to work (and entries inside a /etc/hosts file won't be used in a lookup).

By default, version 1.1.9 or later versions of NGINX cache answers using the TTL value of a response and an optional valid parameter allows the cache time to be overridden:

resolver 127.0.0.1 [::1]:5353 valid=30s;

Before version 1.1.9, tuning of caching time was not possible, and nginx always cached answers for the duration of 5 minutes..

20
  • 24
    After spending most of my day on this - on Ubuntu 12.04 with nginx 1.1.19, set inside location doesn't work properly. Beware
    – omribahumi
    Jul 19, 2015 at 13:38
  • 15
    Note: for docker, it's DNS resolver resides at 127.0.0.11, so for development, I use this: resolver 127.0.0.11 [::1]:5353 valid=15s; Jun 1, 2018 at 11:18
  • 8
    Here's a really nasty gotcha that lost me over a day trying to debug: If you use a variable in the proxy_pass directive, the interpretation of the path changes. When you do this, the path of the original request is completely overridden and replaced by the path given in proxy_pass. You must use a proxy_pass without a path in the url in order to avoid this behaviour. Mar 2, 2020 at 1:02
  • 4
    I had to change it to set $backend "http://dynamic.example.com:80$request_uri";, otherwise the request path will be empty (as the comment above me also mentions)
    – gilad905
    Dec 9, 2020 at 12:41
  • 3
    I found that if the hostname being resolved is just a shortname (no domain part), as would likely be the case for Docker service discovery by service hostname, then your need to add a trailing dot, e.g. set $backend_host "http://backend.:1234/abc";. Otherwise the hostname is resolved once at startup, as though a variable was not being used. Oct 8, 2022 at 1:30
14

There is valuable information in gansbrest comment and ohaal answer.

But I think it's important to mention this official nginx article, posted in 2016, it clearly explains nginx behaviour on this matter and the possible solutions: https://www.nginx.com/blog/dns-service-discovery-nginx-plus/

We indeed have to "Set the Domain Name in a Variable" and use the resolver directive.

however, using a variable changes the rewrite behaviour. You may have to use the rewrite directive, it depends on your location and proxy_pass setup.

PS: would have post a comment but not enough points yet...

11

ohaal's answer takes most of us there, but there is a case where the DNS resolver does not live at 127.0.0.1 (eg when you're in a special containerized environment)

In that case, you may want to change the nginx conf to resolver ${DNS_SERVER};. Then, before you start nginx, run

export DNS_SERVER=$(cat /etc/resolv.conf |grep -i '^nameserver'|head -n1|cut -d ' ' -f2)
envsubst '${DNS_SERVER}' < your_nginx.conf.template > your_nginx.conf

Note, that you need the gettext package installed, as that provides the envsubst command.

1
  • 3
    awk '/nameserver/{a=(a?a" "$2:$2)} END{print a}' /etc/resolv.conf 2> /dev/null
    – samm
    Feb 26, 2021 at 6:06
2

I've hacked together a script to watch a conf.d folder upstreams for dns changes and reload nginx upon detection. It's a first pass, and surely can be improved (next pass, I'll use nginx -T to parse upstreams specifically. Same idea could be used for proxy_pass directives):

#!/bin/bash

get_upstreams() {
  local files=$@
  grep -hEo '(server\s+)[^:;]+' $files | cut -d' ' -f 2
}

resolve_hosts() {
  local hosts=$@
  for h in $hosts; do dig +short $h; done | sort -u
}

watch_dir=$1

[ -d $watch_dir ] || exit 2

upstreams=$(get_upstreams $watch_dir/*)
ips=$(resolve_hosts $upstreams)
if [ ! "$ips" ]; then
  echo "Found no resolvable hosts in $watch_dir files."
fi

host_hash=$(echo $ips | /usr/bin/sha512sum)

echo $host_hash
echo $ips

while [ -d $watch_dir ]; do
  sleep 30
  upstreams=$(get_upstreams $watch_dir/*)
  ips=$(resolve_hosts $upstreams)
  new_hash=$(echo $ips | /usr/bin/sha512sum)
  if [ "$host_hash" != "$new_hash" ]; then
    echo Detected an upstream address change.  $ips
    echo Reloading nginx
    echo $new_hash
    echo $ips
    /sbin/service nginx reload
    host_hash=$new_hash
  fi
done
1
  • do you have a solution which works with nginx -T and proxy_pass?
    – nflaig
    Jun 20, 2021 at 19:34
1

The most voted answer from @ohaal didn't work for me. I had to use a variant of the proposed solution:

resolver 127.0.0.11 ipv6=off valid=10s;

location ~ ^/SomePath/(.*)$ {
    proxy_pass http://dynamic.example.com:80/SomePath/$1$is_args$args;
}

location ~ ^/Other/Path/(.*)$ {
    proxy_pass http://dynamic.example.com:80/Other/Path/$1$is_args$args;
}

Unlike the other solution, this works also of for urls like:

http://proxied:80/SomePath/bla
http://proxied:80/SomePath/bla/bla?whatever=123
http://proxied:80/Other/Path/bla
http://proxied:80/Other/Path/bla/bla

See https://stackoverflow.com/a/8130872/948938

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .