3

On an SMTP server running bind 9.11 for DNS, DNS resolution is failing for one domain causing an email to that domain to fail. There are no problems resolving other domains. However, it can resolve on other DNS servers such as google's or if I run dig +trace. From what I can tell, it's failing due to DNSSEC. If I disable dnssec-validation on Bind, it works. DNSSEC validation tools (dnsviz and verisign's dnssec-analyzer) are not indicating any problem. Any ideas?

Output from dig:

dig friendsadventure.com

; <<>> DiG 9.11.5-P1-1ubuntu2.5-Ubuntu <<>> friendsadventure.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4970
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: c146b587dba9a5ee7737a2255db1b01c98ae834561f56f47 (good)
;; QUESTION SECTION:
;friendsadventure.com.          IN      A

;; Query time: 4212 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Oct 24 10:07:24 EDT 2019
;; MSG SIZE  rcvd: 77

dig +all friendsadventure.com

; <<>> DiG 9.11.5-P1-1ubuntu2.5-Ubuntu <<>> +all friendsadventure.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36655
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 8cb7b3c12a7442d2efdac7005db1b04141ec511e65959d57 (good)
;; QUESTION SECTION:
;friendsadventure.com.          IN      A

;; Query time: 2465 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Oct 24 10:08:01 EDT 2019
;; MSG SIZE  rcvd: 77

dig +trace friendsadventure.com

; <<>> DiG 9.11.5-P1-1ubuntu2.5-Ubuntu <<>> +trace friendsadventure.com
;; global options: +cmd
.                       515510  IN      NS      k.root-servers.net.
.                       515510  IN      NS      c.root-servers.net.
.                       515510  IN      NS      e.root-servers.net.
.                       515510  IN      NS      a.root-servers.net.
.                       515510  IN      NS      h.root-servers.net.
.                       515510  IN      NS      l.root-servers.net.
.                       515510  IN      NS      d.root-servers.net.
.                       515510  IN      NS      g.root-servers.net.
.                       515510  IN      NS      m.root-servers.net.
.                       515510  IN      NS      j.root-servers.net.
.                       515510  IN      NS      b.root-servers.net.
.                       515510  IN      NS      i.root-servers.net.
.                       515510  IN      NS      f.root-servers.net.
.                       515510  IN      RRSIG   NS 8 0 518400 20191106050000 20191024040000 22545 . VMJm6mjyJGRlIHIZFqe63o28rV9XrZpMEOjhFIW094xMFd7s2LL49Dfq +gaiZ549QmIfHUNnTAg9ZGeNHgxs+AFobw5/4ag6oieqo6wJdnwLEIcr AdMeHFz6UJ6FA5MKGWTTY/oBfdfCujbCgTxeMKK1sBwrBLrZ70yfH57x 9/tjVsAYagE5sEi+leATrOtBtJf1FfJqa9wD1ps5GAiOODtI7E+FDFsI 6ZvnTqp0d4qnIcNhf1UiUyvhYoFo7OqnJjDo15h/JMMfG1/9Ope1lAba 9Cdg+ufcIpbfIn63ppq6t/gFGsNUO/+E0rTDno2PdKu0w4rmVxN9ouY/ Hs1/Rw==
;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20191106050000 20191024040000 22545 . aYmq05+eT68QCPzVN5SAQSvxLh82HUwI7Nh0ioeWsyXALVUvN5CVl3S+ qQFTBiUOGn2vbhHDPrfIfLHLQU11VLFQsS9ZCwG8yUBu1agfcpD8/MZF 3GCrnyhBUhWpaj2UptJJlLk/cncoqX+womKaSgbK3vAYAjsmqQ806hhF dlhM3sQodBmTYFqHTTdnmfJVAZWckES7t0K/wjma6DrMsYJK9rgeiTd1 RnmAojPN/y0M+7rLKc1IuJDZK4YFatjuzZACRVMOtEU33Q8GbNrMHMOO 0o5JfwO7r99tVSMXQR/oCWdhT0ljGTpV1Qcl5VldyLr5rkzRFRRoYys/ cKtYvA==
;; Received 1208 bytes from 192.112.36.4#53(g.root-servers.net) in 38 ms

friendsadventure.com.   172800  IN      NS      pdns09.domaincontrol.com.
friendsadventure.com.   172800  IN      NS      pdns10.domaincontrol.com.
friendsadventure.com.   86400   IN      DS      28564 8 1 EAD936FCAE141DD53D38613B1DDB19BCCAC934BA
friendsadventure.com.   86400   IN      DS      18226 8 1 97AE273935A90AA409038F67E6D3F9D3E262AE0E
friendsadventure.com.   86400   IN      RRSIG   DS 8 2 86400 20191031053115 20191024042115 12163 com. a5+9EXcT3oFCxHKwk0kua7Y7eV9R9Suyrzj1MKkgbsrT27/5amOQcGQp J2/K8n1dIuQC5wZRRtDkWXxwyagMGEIJf9MQ4mAtZo9SWl9z46SY/Yh7 59bUao4oIJCzslVUUPsgaqsZutGKgDI5a1DIQLWIKMk3N6dVMbDyAx3m VXFlFaKyo1+ffoA283oQpqjNZ6XIuOxzf1RUwNfptTsF2A==
;; Received 460 bytes from 192.42.93.30#53(g.gtld-servers.net) in 16 ms

friendsadventure.com.   10800   IN      A       160.153.128.37
friendsadventure.com.   10800   IN      RRSIG   A 8 2 10800 20191107175037 20191023175037 12486 friendsadventure.com. BW6U7Jn2wmmT4VOdY/Qb8XZJHVyaLp5FqOFdUpDivP9HG0F781V2V+8u bBmrXKsGPpeZYE/g+dTbhhigdGMKoJtiWkFDRZzo1aQd6SpKkho7vgrk sP4QwTpHviTuF/hbU/PlTGeITVN6JNkY5BX420W35B0kFsxGx+eX+r8E zLmTPRtmc9SQe8iR11Vio5ITsZF6m2Wgo/V4brPo0rbCGhfbUPexhNbH TVhEfFKAvk1Cn/6b2nrpg01EU0Mc8TNQ1eB1/Vf8EyyMU5yJfiOXz7nL kTlT0EVMrEE6phAH5iouS3EwJNzgTC7KhcqsPY91cALNC7Vi10gsT+WS f4Vw+Q==
friendsadventure.com.   3600    IN      NS      pdns09.domaincontrol.com.
friendsadventure.com.   3600    IN      NS      pdns10.domaincontrol.com.
friendsadventure.com.   3600    IN      RRSIG   NS 8 2 3600 20191107175037 20191023175037 12486 friendsadventure.com. gmpyOsvAc9v/GnRV4T9EA1RXxGFQ88C8xG2YljPZEwhvnGjT40j3rrrY tnzKAczZzy064jIwDi2FQ3Q09BUKuswnNALxldPaiZRI22xyj9Mal5n6 AxdYhD4k7esmThO2mUbHtb1Cf7hEOpoPYWZZGCQuHUwsAil+PnbdFto1 +9OhY98Xb8koWHWflNGj+v+2XtemqCXsHrvHncKAY8hZg/DjCFfQMJ5N bE5QnTDKw8uhqHLTm83gsT0pBrSQuz1TGCtNVyqlR37PQwkxrXSJBFtb hrSgRusd5SmYq6kgRN/2Z2n3nYbwKjMikk11FxcppwFUcolledJm59Y4 kzoEyg==
;; Received 737 bytes from 173.201.78.54#53(pdns10.domaincontrol.com) in 14 ms

BIND debug output. Kind of looks like it's trying to resolve using IPV6 (which I disabled as part of my troubleshooting).

grep friendsadventure dns.txt
23-Oct-2019 17:37:27.265 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): query (cache) 'friendsadventure.com/A/IN' approved
23-Oct-2019 17:37:27.265 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): replace
23-Oct-2019 17:37:27.266 fetch: friendsadventure.com/A
23-Oct-2019 17:37:27.267 network unreachable resolving 'friendsadventure.com/A/IN': 2001:500:200::b#53
23-Oct-2019 17:37:27.526 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:83eb::30#53
23-Oct-2019 17:37:27.526 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:39c1::30#53
23-Oct-2019 17:37:27.526 network unreachable resolving 'friendsadventure.com/A/IN': 2001:500:856e::30#53
23-Oct-2019 17:37:27.526 network unreachable resolving 'friendsadventure.com/A/IN': 2001:500:d937::30#53
23-Oct-2019 17:37:27.526 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:d414::30#53
23-Oct-2019 17:37:27.527 network unreachable resolving 'friendsadventure.com/A/IN': 2001:502:7094::30#53
23-Oct-2019 17:37:27.527 network unreachable resolving 'friendsadventure.com/A/IN': 2001:501:b1f9::30#53
23-Oct-2019 17:37:27.527 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:231d::2:30#53
23-Oct-2019 17:37:27.527 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:eea3::30#53
23-Oct-2019 17:37:27.527 network unreachable resolving 'friendsadventure.com/A/IN': 2001:502:8cc::30#53
23-Oct-2019 17:37:27.528 network unreachable resolving 'friendsadventure.com/A/IN': 2001:502:1ca1::30#53
23-Oct-2019 17:37:27.528 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:a83e::2:30#53
23-Oct-2019 17:37:27.528 network unreachable resolving 'friendsadventure.com/A/IN': 2001:503:d2d::30#53
23-Oct-2019 17:37:27.556 network unreachable resolving 'friendsadventure.com/A/IN': 2603:5:22e2::36#53
23-Oct-2019 17:37:27.556 network unreachable resolving 'friendsadventure.com/A/IN': 2603:5:21e2::36#53
23-Oct-2019 17:37:27.592 validating friendsadventure.com/A: starting
23-Oct-2019 17:37:27.592 validating friendsadventure.com/A: attempting positive response validation
23-Oct-2019 17:37:27.592 fetch: friendsadventure.com/DNSKEY
23-Oct-2019 17:37:27.592 network unreachable resolving 'friendsadventure.com/DNSKEY/IN': 2603:5:21e2::36#53
23-Oct-2019 17:37:28.425 network unreachable resolving 'friendsadventure.com/DNSKEY/IN': 2603:5:22e2::36#53
23-Oct-2019 17:37:32.287 client @0x7f0bb80019f0 127.0.0.1#56182 (friendsadventure.com): query (cache) 'friendsadventure.com/A/IN' approved
23-Oct-2019 17:37:32.287 client @0x7f0bb80019f0 127.0.0.1#56182 (friendsadventure.com): replace
23-Oct-2019 17:37:32.288 fetch: friendsadventure.com/A
23-Oct-2019 17:37:32.288 client @0x7f0bb80019f0 127.0.0.1#56182 (friendsadventure.com): next
23-Oct-2019 17:37:32.288 client @0x7f0bb80019f0 127.0.0.1#56182 (friendsadventure.com): request failed: duplicate query
23-Oct-2019 17:37:32.288 client @0x7f0bb80019f0 127.0.0.1#56182 (friendsadventure.com): endrequest
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): query failed (SERVFAIL) for friendsadventure.com/IN/A at ../../../bin/         named/query.c:8579
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): error
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): send
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): sendto
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): servfail cache hit friendsadventure.com/A (CD=0)
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): query failed (SERVFAIL) for friendsadventure.com/IN/A at ../../../bin/         named/query.c:7037
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): error
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): send
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): sendto
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): senddone
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): next
23-Oct-2019 17:37:37.277 client @0x7f0bb801e690 127.0.0.1#56182 (friendsadventure.com): endrequest
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): senddone
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): next
23-Oct-2019 17:37:37.277 client @0x7f0bc009ebb0 127.0.0.1#56182 (friendsadventure.com): endrequest
23-Oct-2019 17:37:37.277 fetch completed at ../../../lib/dns/resolver.c:3930 for friendsadventure.com/A in 10.000141: timed out/success [domain:friendsadventu         re.com,referral:2,restart:1,qrysent:4,timeout:0,lame:0,quota:0,neterr:2,badresp:0,adberr:0,findfail:0,valfail:0]
23-Oct-2019 17:37:37.277 validating friendsadventure.com/A: dns_validator_cancel
23-Oct-2019 17:37:37.278 validating friendsadventure.com/A: in fetch_callback_validator

tcpdump output

17:01:19.075437 IP (tos 0x0, ttl 64, id 27795, offset 0, flags [none], proto UDP (17), length 68)
    extsmtp3.mydomain.com.50210 > a.root-servers.net.domain: 41505 [1au] NS? . (40)
17:01:19.076974 IP (tos 0x0, ttl 64, id 27796, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.43500 > a.root-servers.net.domain: 14448 [1au] A? friendsadventure.com. (61)
17:01:19.090998 IP (tos 0x0, ttl 57, id 41258, offset 0, flags [none], proto UDP (17), length 531)
    a.root-servers.net.domain > extsmtp3.mydomain.com.50210: 41505*-| 13/0/13 . NS e.root-servers.net., . NS h.root-servers.net., . NS l.root-servers.net., . NS i.root-servers.net., . NS a.root-servers.net., . NS d.root-servers.net., . NS c.root-servers.net., . NS b.root-servers.net., . NS j.root-servers.net., . NS k.root-servers.net., . NS g.root-servers.net., . NS m.root-servers.net., . NS f.root-servers.net. (503)
17:01:19.092427 IP (tos 0x0, ttl 57, id 17457, offset 0, flags [none], proto UDP (17), length 525)
    a.root-servers.net.domain > extsmtp3.mydomain.com.43500: 14448-| 0/14/9 (497)
17:01:19.113116 IP (tos 0x0, ttl 64, id 43966, offset 0, flags [none], proto UDP (17), length 68)
    extsmtp3.mydomain.com.34191 > i.root-servers.net.domain: 3117 [1au] DNSKEY? . (40)
17:01:19.124800 IP (tos 0x0, ttl 64, id 56207, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.39560 > j.gtld-servers.net.domain: 55503 [1au] A? friendsadventure.com. (61)
17:01:19.140321 IP (tos 0x0, ttl 57, id 3847, offset 0, flags [none], proto UDP (17), length 488)
    j.gtld-servers.net.domain > extsmtp3.mydomain.com.39560: 55503- 0/5/5 (460)
17:01:19.141418 IP (tos 0x0, ttl 64, id 55938, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.52713 > pdns10.domaincontrol.com.domain: 25015 [1au] A? friendsadventure.com. (61)
17:01:19.160193 IP (tos 0x0, ttl 55, id 18703, offset 0, flags [DF], proto UDP (17), length 457)
    pdns10.domaincontrol.com.domain > extsmtp3.mydomain.com.52713: 25015*-| 2/2/1 friendsadventure.com. A 160.153.128.37, friendsadventure.com. RRSIG (429)
17:01:19.187118 IP (tos 0x0, ttl 64, id 57619, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.48726 > pdns09.domaincontrol.com.domain: 13158 [1au] DNSKEY? friendsadventure.com. (61)
17:01:19.195377 IP (tos 0x0, ttl 56, id 11126, offset 0, flags [none], proto UDP (17), length 56)
    i.root-servers.net.domain > extsmtp3.mydomain.com.34191: 3117*-|$ 0/0/1 (28)
17:01:19.195466 IP (tos 0x0, ttl 57, id 46859, offset 0, flags [DF], proto UDP (17), length 353)
    pdns09.domaincontrol.com.domain > extsmtp3.mydomain.com.48726: 13158*-| 1/0/1 friendsadventure.com. DNSKEY (325)
17:01:19.215932 IP (tos 0x0, ttl 64, id 55940, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.37145 > pdns10.domaincontrol.com.domain: 48829 [1au] DNSKEY? friendsadventure.com. (61)
17:01:20.016338 IP (tos 0x0, ttl 64, id 57790, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.36543 > pdns09.domaincontrol.com.domain: 22039 [1au] DNSKEY? friendsadventure.com. (61)
17:01:20.025279 IP (tos 0x0, ttl 55, id 32534, offset 0, flags [DF], proto UDP (17), length 1181)
    pdns09.domaincontrol.com.domain > extsmtp3.mydomain.com.36543: 22039*-| 4/0/1 friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY (1153)
17:01:20.043305 IP (tos 0x0, ttl 64, id 56055, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.48540 > pdns10.domaincontrol.com.domain: 55907 [1au] DNSKEY? friendsadventure.com. (61)
17:01:21.156737 IP (tos 0x0, ttl 64, id 58059, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.48699 > pdns09.domaincontrol.com.domain: 26029 [1au] DNSKEY? friendsadventure.com. (61)
17:01:21.167443 IP (tos 0x0, ttl 55, id 62141, offset 0, flags [DF], proto UDP (17), length 353)
    pdns09.domaincontrol.com.domain > extsmtp3.mydomain.com.48699: 26029*-| 1/0/1 friendsadventure.com. DNSKEY (325)
17:01:21.188141 IP (tos 0x0, ttl 64, id 56250, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.60396 > pdns10.domaincontrol.com.domain: 48516 [1au] DNSKEY? friendsadventure.com. (61)
17:01:22.788241 IP (tos 0x0, ttl 64, id 58358, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.36707 > pdns09.domaincontrol.com.domain: 24787 [1au] DNSKEY? friendsadventure.com. (61)
17:01:22.798638 IP (tos 0x0, ttl 57, id 63148, offset 0, flags [DF], proto UDP (17), length 353)
    pdns09.domaincontrol.com.domain > extsmtp3.mydomain.com.36707: 24787*-| 1/0/1 friendsadventure.com. DNSKEY (325)
17:01:22.822719 IP (tos 0x0, ttl 64, id 56419, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.50584 > pdns10.domaincontrol.com.domain: 47747 [1au] DNSKEY? friendsadventure.com. (61)
17:01:26.022966 IP (tos 0x0, ttl 64, id 58373, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.43097 > pdns09.domaincontrol.com.domain: 54884 [1au] DNSKEY? friendsadventure.com. (61)
17:01:26.041384 IP (tos 0x0, ttl 55, id 50247, offset 0, flags [DF], proto UDP (17), length 353)
    pdns09.domaincontrol.com.domain > extsmtp3.mydomain.com.43097: 54884*-| 1/0/1 friendsadventure.com. DNSKEY (325)
17:01:26.068980 IP (tos 0x0, ttl 64, id 57148, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.48112 > pdns10.domaincontrol.com.domain: 39385 [1au] DNSKEY? friendsadventure.com. (61)
17:01:26.078022 IP (tos 0x0, ttl 55, id 30505, offset 0, flags [DF], proto UDP (17), length 1181)
    pdns10.domaincontrol.com.domain > extsmtp3.mydomain.com.48112: 39385*-| 4/0/1 friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY (1153)
17:01:26.097372 IP (tos 0x0, ttl 64, id 58379, offset 0, flags [none], proto UDP (17), length 89)
    extsmtp3.mydomain.com.57062 > pdns09.domaincontrol.com.domain: 27082 [1au] DNSKEY? friendsadventure.com. (61)

Config is all default for an ubuntu install, except I did disable IPV6 after this problem occured:

named.conf:
include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";

named.conf.options:
    options {
            directory "/var/cache/bind";
            dnssec-validation auto;
            filter-aaaa-on-v4 yes;
            listen-on-v6 { none; };
    };

named.conf.local: empty
named.conf.default-zones:
zone "." {
        type hint;
        file "/usr/share/dns/root.hints";
};

zone "localhost" {
        type master;
        file "/etc/bind/db.local";
};

zone "127.in-addr.arpa" {
        type master;
        file "/etc/bind/db.127";
};

zone "0.in-addr.arpa" {
        type master;
        file "/etc/bind/db.0";
};

zone "255.in-addr.arpa" {
        type master;
        file "/etc/bind/db.255";
};
17
  • Showing your configuration may be needed as the problem is clearly local, the domain resolves fine. Oct 25 '19 at 3:19
  • Config has been added to the original post
    – Jeremy
    Oct 25 '19 at 13:21
  • Can you check this serverfault.com/questions/639061/… if it helps?
    – asktyagi
    Oct 28 '19 at 4:47
  • 1
    I would try removing filter-aaaa-on-v4 and listen-on-v6 and make sure to start bind with named -4 as advised on kb.isc.org/docs/aa-00821 Besides that to debug DNSSEC problems, do your query normally, expect SERVFAIL, do exact same query using +cd and if it is then not SERVFAIL then 99% chance it is related to DNSSEC as only difference in second case is disabling any checks, that is any DNSSEC validation. Oct 29 '19 at 0:15
  • 1
    Do you filter TCP by any chance or play with MTU? dig DNSKEY friendsadventure.com is 1153 bytes long response, dangerously close to some MTUs... Try dig DNSKEY friendsadventure.com +notcp +bufsize=1100 , it sould say ";; Truncated, retrying in TCP mode." while it should work with +bufsize=1200 Oct 29 '19 at 0:23
3

What I find in Your tcpdump is, that it looks like Your TCP connection to the servers is broken. You get one maxsized (1460 for 1500 MTU) bytes TCP package back, then the connections is torn down, acknowledge it and the next package You get is already the FIN:

12:52:10.773476 IP (tos 0x0, ttl 64, id 30033, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.255.11.53639 > 173.201.78.54.53: Flags [S], cksum 0xa74a (incorrect -> 0xfc0e), seq 1207191269, win 64240, options [mss 1460,sackOK,TS val 2622008582 ecr 0,nop,wscale 6], length 0
12:52:10.784310 IP (tos 0x0, ttl 57, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    173.201.78.54.53 > 172.16.255.11.53639: Flags [S.], cksum 0x5de3 (correct), seq 3333869771, ack 1207191270, win 29200, options [mss 1420,nop,nop,sackOK,nop,wscale 7], length 0
12:52:10.784356 IP (tos 0x0, ttl 64, id 30034, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.255.11.53639 > 173.201.78.54.53: Flags [.], cksum 0xa736 (incorrect -> 0x0cb2), ack 1, win 1004, length 0
12:52:10.784493 IP (tos 0x0, ttl 64, id 30035, offset 0, flags [DF], proto TCP (6), length 103)
    172.16.255.11.53639 > 173.201.78.54.53: Flags [P.], cksum 0xa775 (incorrect -> 0x24cf), seq 1:64, ack 1, win 1004, length 6337614 [1au] DNSKEY? friendsadventure.com. (61)
12:52:10.805803 IP (tos 0x0, ttl 255, id 38197, offset 0, flags [none], proto TCP (6), length 1460)
    173.201.78.54.53 > 172.16.255.11.53639: Flags [.], cksum 0xeeaf (correct), seq 1:1421, ack 64, win 227, length 142037614*- 5/3/1 friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. DNSKEY, friendsadventure.com. RRSIG[|domain]
12:52:10.805837 IP (tos 0x0, ttl 64, id 30036, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.255.11.53639 > 173.201.78.54.53: Flags [.], cksum 0xa736 (incorrect -> 0x06e9), ack 1421, win 1002, length 0
12:52:10.805842 IP (tos 0x0, ttl 255, id 38198, offset 0, flags [none], proto TCP (6), length 40)
    173.201.78.54.53 > 172.16.255.11.53639: Flags [FP.], cksum 0x09e7 (correct), seq 1421, ack 64, win 227, length 0
12:52:10.806044 IP (tos 0x0, ttl 64, id 30037, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.255.11.53639 > 173.201.78.54.53: Flags [F.], cksum 0xa736 (incorrect -> 0x06e7), seq 64, ack 1422, win 1002, length 0
12:52:10.806085 IP (tos 0x0, ttl 64, id 44626, offset 0, flags [none], proto UDP (17), length 89)
    172.16.255.11.34136 > 97.74.110.54.53: 26250 [1au] DNSKEY? friendsadventure.com. (61)
12:52:10.806877 IP (tos 0x0, ttl 255, id 38199, offset 0, flags [none], proto TCP (6), length 40)
    173.201.78.54.53 > 172.16.255.11.53639: Flags [.], cksum 0x09ee (correct), ack 65, win 227, length 0

The probability that You get exactly 1460 bytes payload back is not very high. Additionally I can confirm with my server that the response is larger and not exactly on 1460 byte boundary for the same query (1857).

My DNS communication for that part looks as follows:

00:57:17.692876 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 40) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [S], cksum 0x794a (incorrect -> 0x23b5), seq 3083934744, win 64660, options [mss 1220,sackOK,TS val 27470909 ecr 0,nop,wscale 7], length 0
00:57:17.696289 IP6 (hlim 54, next-header TCP (6) payload length: 40) pdns10.domaincontrol.com.domain > myserver.mydomain.net.eu.org.44427: Flags [S.], cksum 0x5462 (correct), seq 1600069237, ack 3083934745, win 28560, options [mss 1440,sackOK,TS val 1680374125 ecr 27470909,nop,wscale 7], length 0
00:57:17.696384 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 32) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [.], cksum 0x7942 (incorrect -> 0xf0ad), seq 1, ack 1, win 506, options [nop,nop,TS val 27470912 ecr 1680374125], length 0
00:57:17.696634 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 95) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [P.], cksum 0x7981 (incorrect -> 0xc999), seq 1:64, ack 1, win 506, options [nop,nop,TS val 27470912 ecr 1680374125], length 63 17914 [1au] DNSKEY? friendsadventure.com. ar: . OPT UDPsize=4096 DO (61)
00:57:17.700005 IP6 (hlim 54, next-header TCP (6) payload length: 32) pdns10.domaincontrol.com.domain > myserver.mydomain.net.eu.org.44427: Flags [.], cksum 0xf185 (correct), seq 1, ack 64, win 224, options [nop,nop,TS val 1680374128 ecr 27470912], length 0
00:57:17.700961 IP6 (hlim 54, next-header TCP (6) payload length: 34) pdns10.domaincontrol.com.domain > myserver.mydomain.net.eu.org.44427: Flags [P.], cksum 0xea59 (correct), seq 1:3, ack 64, win 224, options [nop,nop,TS val 1680374129 ecr 27470912], length 2
00:57:17.700986 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 32) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [.], cksum 0x7942 (incorrect -> 0xf063), seq 64, ack 3, win 506, options [nop,nop,TS val 27470917 ecr 1680374129], length 0
00:57:17.701005 IP6 (hlim 54, next-header TCP (6) payload length: 1857) pdns10.domaincontrol.com.domain > myserver.mydomain.net.eu.org.44427: Flags [FP.], cksum 0x8063 (incorrect -> 0x768f), seq 3:1828, ack 64, win 224, options [nop,nop,TS val 1680374129 ecr 27470912], length 1825 33792 [b2&3=0x1] [3a] [5q] [1n] [4198au][|domain]
00:57:17.701106 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 32) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [.], cksum 0x7942 (incorrect -> 0xe94c), seq 64, ack 1829, win 495, options [nop,nop,TS val 27470917 ecr 1680374129], length 0
00:57:17.701932 IP6 (flowlabel 0x936c6, hlim 64, next-header TCP (6) payload length: 32) myserver.mydomain.net.eu.org.44427 > pdns10.domaincontrol.com.domain: Flags [F.], cksum 0x7942 (incorrect -> 0xe942), seq 64, ack 1829, win 503, options [nop,nop,TS val 27470918 ecr 1680374129], length 0

That UDP does not succeed in some or many cases if You have DNSSEC (see same in my case) is not very uncommon.
So we need to concentrate on why You do not get whole the TCP response back.

I wonder why Your machine sets DF all the outgoing packages. (Maybe it's because of Path MTU discovery?)
What devices are between Your server and the public, well-defined internet?
May Your NIC has some problems? Which type of NIC do You have and which type of hardware/VM?
Which features are enables on Your NIC (ethtool -k ethX)?
Sometimes there are NICs where some features are broken and could/should be disabled if it is not a device between Your NIC and the public Internet.
You see in my case the NIC is sending my kernel oversized packages, thought the MTU is locally 1500 (and normally the same in internet). Such features can also cause troubles sometimes and need to be disabled in such cases.

30
  • I have no idea why df is set on all outgoing packets. A cisco 2951 router is between the server and internet. It is an ubuntu VM on a Hyper-V server running on An HP DL380. I'll definitely look into what you've found, but it does seem really odd that would only affect DNS resolution on one domain while everything else functions fine. Maybe the remote NS is ignoring the df bit?
    – Jeremy
    Oct 31 '19 at 14:57
  • Features for eth0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on
    – Jeremy
    Oct 31 '19 at 15:01
  • udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on [fixed] tx-vlan-offload: on [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-gre-csum-segmentation: off [fixed] tx-ipxip4-segmentation: off [fixed] tx-ipxip6-segmentation: off [fixed]
    – Jeremy
    Oct 31 '19 at 15:02
  • tx-udp_tnl-segmentation: off [fixed] tx-udp_tnl-csum-segmentation: off [fixed] tx-gso-partial: off [fixed] tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] tx-udp-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off [fixed] esp-hw-offload: off [fixed]
    – Jeremy
    Oct 31 '19 at 15:03
  • esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off [fixed] tls-hw-tx-offload: off [fixed] tls-hw-rx-offload: off [fixed] rx-gro-hw: off [fixed] tls-hw-record: off [fixed]
    – Jeremy
    Oct 31 '19 at 15:03
0

The response appears to be very large, and they are going through UDP, but in this case I'd set a maximum UDP packet size using the parameter:

max-udp-size 512;

This will force the transport protocol of the server to switch over to TCP, which has a higher chance of delivering a successful response of larger packet sizes.

You may also try utilizing minimal responses as well:

minimal-responses yes;
1
  • Tried those settings. No luck.
    – Jeremy
    Nov 4 '19 at 21:06
0

It seems that your server's IPv6 connectivity is broken in a way that the domaincontrol.com name servers cannot be reached via IPv6.

The settings you have done do not disable using IPv6 with outbound queries to other server.

listen-on-v6 { none; }; only disables listening on IPv6 ports for incoming requests.

filter-aaaa-on-v4 yes; filters resolving of AAAA records when clients are using IPv4 to connect to name server.

You can debug connectivity to the nameservers above with traceroute6 utility, and see if the packets drop immediately when leaving your box or if they drop later.

If they drop immediately, it means your IPv6 is completely broken and you need to disable IPv6 addresses in your server. The preferred way to disable IPv6 on the server depends on the operating system in use and also on how the IPv6 addresses are provided to the system.

Furthermore, you might want to report the issue to your service provider.

If the packets drop later, you can report the issue to your service provider, which might be able to fix the routing for that destination.

9
  • We don't use IPV6 anywhere on our network. IPV6 has been disabled entirely on the server. Still no luck.
    – Jeremy
    Nov 3 '19 at 23:06
  • Verified ipv6 is disabled: cat /proc/sys/net/ipv6/conf/all/disable_ipv6 1
    – Jeremy
    Nov 3 '19 at 23:06
  • What does ip a show? The line 23-Oct-2019 17:37:27.267 network unreachable resolving 'friendsadventure.com/A/IN': 2001:500:200::b#53 indicates that Bind is trying to connect to the name server using IPv6. Nov 4 '19 at 6:19
  • 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:15:5d:03:29:1a brd ff:ff:ff:ff:ff:ff inet 172.16.255.11/24 brd 172.16.255.255 scope global eth0 valid_lft forever preferred_lft forever
    – Jeremy
    Nov 4 '19 at 14:10
  • 1
    No that's wrong @Tero it is mentioned that the -4 switch disable IPv6 also on recursion and there are user reports around that confirm that (kb.isc.org/docs/aa-00821). The man page tells generally use only IPv4. Thought the kb is from 2018 so probably more recent than the version used here..
    – EOhm
    Nov 5 '19 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.