If a DNS server looks up a record and it's missing, it will often "negatively cache" the fact that this record is missing, and not try to look it up again for a while. I don't see anything in the RFC about the TTL on negative caching should be, so I'm guessing it's somewhat arbitrary. In the real world, how long do these negative records stick around for?
The TTL for negative caching is not arbitrary. It is taken from the SOA record at the top of the zone to which the requested record would have belonged, had it existed. For example:
example.org. IN SOA master-ns1.example.org. Hostmaster.example.org. ( 2012091201 43200 1800 1209600 86400 )
The last value in the SOA record ("86400") is the amount of time clients are asked to cache negative results under
If a client requests
doesnotexist.example.org., it will cache the result for 86400 seconds.
This depends on your exact definition of a "negative query", but in either case, this is documented in rfc2308 «Negative Caching of DNS Queries (DNS NCACHE)»:
- If the resolution is successful, and results in
NXDOMAIN, the response will come with a
SOArecord, which would contain the
NXDOMAINTTL (traditionally known as the
If the resolution is not successful, and results in a timeout (
SERVFAIL), then it may as well not be cached at all, and in all circumstances MUST NOT be cached for longer than 5 minutes.
Note that in practice, caching such results for the full allowable 5 minutes is a great way to diminish the experience of a client should their cache server occasionally suffer brief connectivity issues (and effectively make it easily vulnerable to a Denial-of-Service amplification, where a few seconds of downtime would result in the certain parts of the DNS being down for the five full minutes).
Prior to BIND 9.9.6-S1 (released in 2014), apparently,
SERVFAILwas not cached at all.
E.g., at the time of your question and in all versions of BIND released prior to 2014, the BIND recursive resolver DID NOT cache
SERVFAILat all, if the above commit and the documentation about the first introduction in 9.9.6-S1 is to be believed.
In the latest BIND, the default
1s, and the setting is hardcoded to a ceiling of
30s(in place of the RFC-mandated ceiling of
Furthermore, the following are some noteworthy quotes on the matter:
The outcome of caching SERVFAIL responses has included some situations where it was seen to be detrimental to the client experience, particularly when the causes of the SERVFAIL being presented to the client were transient and from a scenario where an immediate retry of the query would be a more appropriate action.
The second tactic is to claim that widespread DNS clients will do something Particularly Evil when they are unable to reach all DNS servers. The problem with this argument is that the claim is false. Any such client is clearly buggy, and will be unable to survive in the marketplace: consider what happens if the client's routers briefly go down, or if the client's network is temporarily flooded.
In summary, an
NXDOMAIN response would be cached as specified in the
SOA of the applicable zone, whereas
SERVFAIL is unlikely to be cached, or, if cached, it'll be at most a double-digit number of seconds.
There is an RFC dedicated to this topic: RFC 2308 - Negative Caching of DNS Queries (DNS NCACHE).
The relevant section to read is 5 - Caching Negative Answers which states:
Like normal answers negative answers have a time to live (TTL). As there is no record in the answer section to which this TTL can be applied, the TTL must be carried by another method. This is done by including the SOA record from the zone in the authority section of the reply. When the authoritative server creates this record its TTL is taken from the minimum of the SOA.MINIMUM field and SOA's TTL. This TTL decrements in a similar manner to a normal cached answer and upon reaching zero (0) indicates the cached negative answer MUST NOT be used again.
Firstly lets identify the
SOA.MINIMUM and SOA TTL described in the RFC. The TTL is the number before the the record type
900 seconds in the example below). While the minimum is last field in the record (
86400 seconds in the example below).
$ dig serverfault.com soa @ns-1135.awsdns-13.org +noall +answer +multiline ; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> serverfault.com soa @ns-1135.awsdns-13.org +noall +answer +multiline ;; global options: +cmd serverfault.com. 900 IN SOA ns-1135.awsdns-13.org. awsdns-hostmaster.amazon.com. ( 1 ; serial 7200 ; refresh (2 hours) 900 ; retry (15 minutes) 1209600 ; expire (2 weeks) 86400 ; minimum (1 day) )
Now lets look at some examples, the
serverfault.com zone is illustrative as it has authoritative servers from two different providers that are configured differently.
Lets find the authoritative nameservers for the
$ host -t ns serverfault.com serverfault.com name server ns-860.awsdns-43.net. serverfault.com name server ns-1135.awsdns-13.org. serverfault.com name server ns-cloud-c1.googledomains.com. serverfault.com name server ns-cloud-c2.googledomains.com.
Then check the SOA record using an aws nameserver:
$ dig serverfault.com soa @ns-1135.awsdns-13.org | grep 'ANSWER SECTION' -A 1 ;; ANSWER SECTION: serverfault.com. 900 IN SOA ns-1135.awsdns-13.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
From this we can see that the TTL of the SOA record is
900 seconds while the negative TTL value is
86400 seconds. The SOA TTL value of
900 is lower so we expect this value to be used.
Now if we query an authoritative server for a non existent domain we should get a response without an answer and with a SOA record in the authority section:
$ dig nxdomain.serverfault.com @ns-1135.awsdns-13.org ; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> nxdomain.serverfault.com @ns-1135.awsdns-13.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 51948 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nxdomain.serverfault.com. IN A ;; AUTHORITY SECTION: serverfault.com. 900 IN SOA ns-1135.awsdns-13.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 125 msec ;; SERVER: 184.108.40.206#53(220.127.116.11) ;; WHEN: Tue Aug 20 15:49:47 NZST 2019 ;; MSG SIZE rcvd: 135
When a recursive (caching) resolver receives this answer it will parse the SOA record in the
AUTHORITY SECTION and use the TTL of this record to determine how long it should cache the negative result (in this case
Now lets follow the same procedure with a google nameserver:
$ dig serverfault.com soa @ns-cloud-c2.googledomains.com | grep 'ANSWER SECTION' -A 1 ;; ANSWER SECTION: serverfault.com. 21600 IN SOA ns-cloud-c1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
You can see that the google nameservers have different values for both the SOA TTL and the Negative TTL values. In this case the negative TTL of
300 is lower than the SOA TTL of
21600. Therefore the google server should use the lower value in the
AUTHORITY SECTION SOA record when returning an
$ dig nxdomain.serverfault.com @ns-cloud-c2.googledomains.com ; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> nxdomain.serverfault.com @ns-cloud-c2.googledomains.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25920 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;nxdomain.serverfault.com. IN A ;; AUTHORITY SECTION: serverfault.com. 300 IN SOA ns-cloud-c1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 ;; Query time: 130 msec ;; SERVER: 18.104.22.168#53(22.214.171.124) ;; WHEN: Tue Aug 20 16:05:24 NZST 2019 ;; MSG SIZE rcvd: 143
As expected the TTL of the SOA record in the
NXDOMAIN response is
The example above also demonstrates how easy it is to get different answers to the same query. The answer that an individual caching resolver ends up using is down to which authoritative namserver was queried.
In my testing I have also observed that some recursive (caching) resolvers do not return an
AUTHORITY SECTION with a SOA record with a decrementing TTL for subsequent requests whereas others do.
For example the cloudflare resolver does (note the decrementing TTL value):
$ dig nxdomain.serverfault.com @126.96.36.199 | grep 'AUTHORITY SECTION' -A 1 ;; AUTHORITY SECTION: serverfault.com. 674 IN SOA ns-1135.awsdns-13.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 $ dig nxdomain.serverfault.com @188.8.131.52 | grep 'AUTHORITY SECTION' -A 1 ;; AUTHORITY SECTION: serverfault.com. 668 IN SOA ns-1135.awsdns-13.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
While the default resolver in an AWS VPC will respond with an authority section only on the first request:
$ dig nxdomain.serverfault.com @169.254.169.253 | grep 'AUTHORITY SECTION' -A 1 ;; AUTHORITY SECTION: serverfault.com. 300 IN SOA ns-cloud-c1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 $ dig nxdomain.serverfault.com @169.254.169.253 | grep 'AUTHORITY SECTION' -A 1 | wc -l 0
Note: This answer addresses the behavior of