4

We have been experiencing problems with our company's DNS server when trying to resolve only certain domains, we are running BIND 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 on a CentOS 6.5 server. We are autoritative for some zones and our internal clients and mail system resolve using this server. One of the domains we are having trouble resolving is www.dhl.com, here's what we get when querying using dig:

[root@serverx etc] dig www.dhl.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> www.dhl.com
;; global options: +cmd
;; connection timed out; no servers could be reached

and

[root@serverx etc]# dig +trace www.dhl.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> +trace www.dhl.com
;; global options: +cmd
.           517419  IN  NS  g.root-servers.net.
.           517419  IN  NS  a.root-servers.net.
.           517419  IN  NS  h.root-servers.net.
.           517419  IN  NS  m.root-servers.net.
.           517419  IN  NS  f.root-servers.net.
.           517419  IN  NS  b.root-servers.net.
.           517419  IN  NS  l.root-servers.net.
.           517419  IN  NS  j.root-servers.net.
.           517419  IN  NS  k.root-servers.net.
.           517419  IN  NS  e.root-servers.net.
.           517419  IN  NS  i.root-servers.net.
.           517419  IN  NS  d.root-servers.net.
.           517419  IN  NS  c.root-servers.net.
;; Received 496 bytes from 192.168.X.X#53(192.168.X.X) in 11 ms

com.            172800  IN  NS  a.gtld-servers.net.
com.            172800  IN  NS  d.gtld-servers.net.
com.            172800  IN  NS  k.gtld-servers.net.
com.            172800  IN  NS  b.gtld-servers.net.
com.            172800  IN  NS  j.gtld-servers.net.
com.            172800  IN  NS  c.gtld-servers.net.
com.            172800  IN  NS  h.gtld-servers.net.
com.            172800  IN  NS  l.gtld-servers.net.
com.            172800  IN  NS  e.gtld-servers.net.
com.            172800  IN  NS  g.gtld-servers.net.
com.            172800  IN  NS  m.gtld-servers.net.
com.            172800  IN  NS  f.gtld-servers.net.
com.            172800  IN  NS  i.gtld-servers.net.
;; Received 489 bytes from 202.12.27.33#53(202.12.27.33) in 6128 ms

dhl.com.        172800  IN  NS  ns4.dhl.com.
dhl.com.        172800  IN  NS  ns6.dhl.com.
dig: couldn't get address for 'ns4.dhl.com': no more

And when I do dig using google's dns server:

dig @8.8.4.4 www.dhl.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> @8.8.4.4 www.dhl.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11325
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.dhl.com.           IN  A

;; ANSWER SECTION:
www.dhl.com.        1619    IN  CNAME   ngw.dhl.com.edgesuite.net.
ngw.dhl.com.edgesuite.net. 8520 IN  CNAME   a1085.g.akamai.net.
a1085.g.akamai.net. 19  IN  A   23.74.2.113
a1085.g.akamai.net. 19  IN  A   23.74.2.120

;; Query time: 229 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Dec  4 14:47:56 2014
;; MSG SIZE  rcvd: 129

No problem!, and from the same server...

When you look to "/var/log/messages" nothing gets logged!!!. Again, this only happens with certain domains and this server was working ok a couple of days ago, we also have disabled selinux for testing purposes.

This is our named.conf file (named is runnig in a chroot environment):

options {
//      listen-on port 53 { 127.0.0.1;192.168.xx.x; };
//      listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
//      allow-query     { localhost; };
        allow-query     { any; };
        recursion yes;
        allow-recursion { recursive-clients; };
//      query-source address * port 53;
//      dnssec-enable yes;
        dnssec-enable no;
//      dnssec-validation yes;
        dnssec-validation no;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";
        allow-transfer { xxx.x.x.x; xxx.x.x.x; xxx.x.x.x; 127.0.0.1; };
        allow-update { 192.168.xx.xx; };
//        forwarders { 8.8.4.4; };
};

acl recursive-clients { xxx.x.x.x/24; 127.0.0.1; xxx.xxx.xx.x/24; xx.xxx.xxx.xxx/29; xxx.xxx.xx.x;};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };

};

zone "." IN {
        type hint;
        file "named.ca";
};


zone "domain.com.xx" IN {
        type master;
        file "db.domain.com.xx";
        allow-transfer { xxx.xxx.xx.xx; xxx.xxx.xxx.x; };
};

zone "xx.xxx.xxx.in-addr.arpa" IN {
        type master;
        file "db.xxx.xxx.xx";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

Any idea guys???... I've been doing tests and researching since yesterday and I cannot figure what is happening!!!

Thanks in advance for any help or idea.

Here it the result using dig +trace +additional www.dhl.com:

dig +trace +additional www.dhl.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> +trace +additional www.dhl.com
;; global options: +cmd
.           518340  IN  NS  h.root-servers.net.
.           518340  IN  NS  l.root-servers.net.
.           518340  IN  NS  e.root-servers.net.
.           518340  IN  NS  k.root-servers.net.
.           518340  IN  NS  i.root-servers.net.
.           518340  IN  NS  m.root-servers.net.
.           518340  IN  NS  b.root-servers.net.
.           518340  IN  NS  c.root-servers.net.
.           518340  IN  NS  g.root-servers.net.
.           518340  IN  NS  f.root-servers.net.
.           518340  IN  NS  d.root-servers.net.
.           518340  IN  NS  a.root-servers.net.
.           518340  IN  NS  j.root-servers.net.
k.root-servers.net. 518345  IN  A   193.0.14.129
k.root-servers.net. 518345  IN  AAAA    2001:7fd::1
b.root-servers.net. 518345  IN  A   192.228.79.201
b.root-servers.net. 518345  IN  AAAA    2001:500:84::b
c.root-servers.net. 518345  IN  A   192.33.4.12
c.root-servers.net. 518345  IN  AAAA    2001:500:2::c
i.root-servers.net. 518345  IN  A   192.36.148.17
i.root-servers.net. 518345  IN  AAAA    2001:7fe::53
f.root-servers.net. 518345  IN  A   192.5.5.241
f.root-servers.net. 518345  IN  AAAA    2001:500:2f::f
h.root-servers.net. 518345  IN  A   128.63.2.53
h.root-servers.net. 518345  IN  AAAA    2001:500:1::803f:235
a.root-servers.net. 518345  IN  A   198.41.0.4
;; Received 508 bytes from 192.168.x.x#53(192.168.x.x) in 11363 ms

com.            172800  IN  NS  d.gtld-servers.net.
com.            172800  IN  NS  l.gtld-servers.net.
com.            172800  IN  NS  b.gtld-servers.net.
com.            172800  IN  NS  m.gtld-servers.net.
com.            172800  IN  NS  c.gtld-servers.net.
com.            172800  IN  NS  k.gtld-servers.net.
com.            172800  IN  NS  i.gtld-servers.net.
com.            172800  IN  NS  a.gtld-servers.net.
com.            172800  IN  NS  f.gtld-servers.net.
com.            172800  IN  NS  g.gtld-servers.net.
com.            172800  IN  NS  j.gtld-servers.net.
com.            172800  IN  NS  h.gtld-servers.net.
com.            172800  IN  NS  e.gtld-servers.net.
a.gtld-servers.net. 172800  IN  A   192.5.6.30
b.gtld-servers.net. 172800  IN  A   192.33.14.30
c.gtld-servers.net. 172800  IN  A   192.26.92.30
d.gtld-servers.net. 172800  IN  A   192.31.80.30
e.gtld-servers.net. 172800  IN  A   192.12.94.30
f.gtld-servers.net. 172800  IN  A   192.35.51.30
g.gtld-servers.net. 172800  IN  A   192.42.93.30
h.gtld-servers.net. 172800  IN  A   192.54.112.30
i.gtld-servers.net. 172800  IN  A   192.43.172.30
j.gtld-servers.net. 172800  IN  A   192.48.79.30
k.gtld-servers.net. 172800  IN  A   192.52.178.30
l.gtld-servers.net. 172800  IN  A   192.41.162.30
m.gtld-servers.net. 172800  IN  A   192.55.83.30
a.gtld-servers.net. 172800  IN  AAAA    2001:503:a83e::2:30
;; Received 489 bytes from 202.12.27.33#53(202.12.27.33) in 4335 ms

dhl.com.        172800  IN  NS  ns4.dhl.com.
dhl.com.        172800  IN  NS  ns6.dhl.com.
ns4.dhl.com.        172800  IN  A   165.72.192.16
ns6.dhl.com.        172800  IN  A   199.40.254.166
dig: couldn't get address for 'ns4.dhl.com': no more

Output from dig +tcp www.redhat.com

dig +tcp www.redhat.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <<>> +tcp www.redhat.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62110
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 8, ADDITIONAL: 8

;; QUESTION SECTION:
;www.redhat.com.            IN  A

;; ANSWER SECTION:
www.redhat.com.     11  IN  CNAME   wildcard.redhat.com.edgekey.net.
wildcard.redhat.com.edgekey.net. 20484 IN CNAME wildcard.redhat.com.edgekey.net.globalredir.akadns.net.
wildcard.redhat.com.edgekey.net.globalredir.akadns.net. 2485 IN CNAME e1890.b.akamaiedge.net.
e1890.b.akamaiedge.net. 20  IN  A   172.229.164.152

;; AUTHORITY SECTION:
b.akamaiedge.net.   2874    IN  NS  n2b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n3b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n4b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n1b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n7b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n5b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n6b.akamaiedge.net.
b.akamaiedge.net.   2874    IN  NS  n0b.akamaiedge.net.

;; ADDITIONAL SECTION:
n5b.akamaiedge.net. 6917    IN  A   201.144.215.107
n1b.akamaiedge.net. 4917    IN  A   23.61.206.74
n6b.akamaiedge.net. 2917    IN  A   201.144.215.108
n2b.akamaiedge.net. 6917    IN  A   192.204.11.244
n4b.akamaiedge.net. 4917    IN  A   201.144.215.110
n7b.akamaiedge.net. 4917    IN  A   201.144.215.113
n0b.akamaiedge.net. 2917    IN  A   23.61.206.68
n3b.akamaiedge.net. 2917    IN  A   201.144.215.114

;; Query time: 132 msec
;; SERVER: 192.168.x.x#53(192.168.x.x)
;; WHEN: Thu Dec  4 16:03:03 2014
;; MSG SIZE  rcvd: 463

Other tests:

traceroute -U -p 53 165.72.192.16
traceroute to 165.72.192.16 (165.72.192.16), 30 hops max, 60 byte packets
 1  192.168.17.10 (192.168.17.10)  0.724 ms  0.718 ms  0.681 ms
 2  168.243.205.74 (168.243.205.74)  4.188 ms  4.677 ms  3.945 ms
 3  172.26.64.21 (172.26.64.21)  179.831 ms  180.282 ms  181.633 ms
 4  172.24.0.13 (172.24.0.13)  182.433 ms  182.585 ms  179.870 ms
 5  172.24.0.9 (172.24.0.9)  180.654 ms  183.023 ms  180.876 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *

And:

ping 165.72.192.16
PING 165.72.192.16 (165.72.192.16) 56(84) bytes of data.
64 bytes from 165.72.192.16: icmp_seq=1 ttl=240 time=356 ms
64 bytes from 165.72.192.16: icmp_seq=2 ttl=240 time=325 ms
64 bytes from 165.72.192.16: icmp_seq=3 ttl=240 time=291 ms
64 bytes from 165.72.192.16: icmp_seq=4 ttl=240 time=260 ms

traceroute -U -p 53 199.40.254.166
traceroute to 199.40.254.166 (199.40.254.166), 30 hops max, 60 byte packets
 1  192.168.17.10 (192.168.17.10)  0.710 ms  0.683 ms  0.764 ms
 2  168.243.205.74 (168.243.205.74)  3.136 ms  3.875 ms  4.191 ms
 3  172.26.64.21 (172.26.64.21)  19.367 ms  18.988 ms  19.698 ms
 4  172.24.0.177 (172.24.0.177)  4.657 ms  6.608 ms  7.088 ms
 5  172.24.0.9 (172.24.0.9)  5.126 ms  7.412 ms  5.518 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *

ping 199.40.254.166
PING 199.40.254.166 (199.40.254.166) 56(84) bytes of data.
64 bytes from 199.40.254.166: icmp_seq=1 ttl=237 time=287 ms
64 bytes from 199.40.254.166: icmp_seq=2 ttl=237 time=280 ms
64 bytes from 199.40.254.166: icmp_seq=3 ttl=237 time=286 ms

UPDATE - 5 Dic 2014

Well, I have installed bind in another server on the same subnet, same OS version and same bind release, and it works just fine!!!, the only thing that changes is the IP address and I can't change that to test because the server with the problem is in production... so I think the IDS theory that Andrew sugests is true... I'll talk to our ISP and investigate our external IP if its blacklisted and post how it goes...

UPDATE - 6 Dic 2014

The server's IP address is not listed in any black list that I checked on the Internet...

  • I forgot to mention that we already downloaded and intalled a new named.ca file to have a fresh root servers config file. – dragonov7 Dec 4 '14 at 21:28
  • So long as one of the nameservers in named.ca were still valid, it should have been able to stub the rest of the root nameservers successfully. – Andrew B Dec 4 '14 at 21:44
  • What happens when you try dig @165.72.192.16 dhl.com SOA and dig @199.40.254.166 dhl.com SOA? – Andrew B Dec 4 '14 at 22:21
  • 1
    That's your problem. Run traceroute -U -p 53 as root to both of those IPs. You can replace the output of those dig queries to save space. (if your traceroute command doesn't support -U, just run it without the -U -p 53) – Andrew B Dec 4 '14 at 22:31
  • 1
    Now comes the fun part. You have to figure out what device is losing the packet and why, which we can't assist you with. If traceroute -U -p 53 8.8.4.4 gets further along the route than your last two traceroutes, you'll have the problem isolated to a network segment. – Andrew B Dec 4 '14 at 22:48
5

Rerun your dig +trace with the +additional flag. I expect it to still fail, but I'll explain what's happening.

  • +trace with +additional will display the nameserver glue (the ADDITIONAL section), which will at least tell you that the TLD nameservers are properly returning the IP address of ns4.dhl.com.
  • The couldn't get address for 'ns4.dhl.com': no more failure is coming from your local nameserver. This is an obscure detail, but dig +trace does not use data from the ADDITIONAL section to determine the IP address of the "next hop" nameserver. It is actually querying your nameserver in /etc/resolv.conf to obtain the IP address of ns4.dhl.com, and this query is failing.

Based on the result of the dig commands I had you run in the comments, there appears to be a problem communicating with DHL's nameservers on port 53. A UDP based traceroute on port 53 (traceroute -U -p 53) will give you a better idea of where the packet is being lost. It's either a device on your network or DHL's losing the query/reply. In the latter case, your nameserver's IP address may have somehow gotten itself into a reputation database for an IDS device.

  • Hey Andrew, thanks for your help... I have edited the question to include the output from dig +trace +additiona www.dhl.com – dragonov7 Dec 4 '14 at 21:55
  • Yeah, that confirms my theory. You're able to get the IP address of the nameserver from the glue record, but for some reason your nameserver in /etc/resolv.conf is reporting that it doesn't know the A record for ns4. Make sure TCP 53 is open (as davidgo suggested), and then flush your cache. You can either restart named or use rndc flushname dhl.com. – Andrew B Dec 4 '14 at 22:03
  • I restarted named and flushed its cache... same result. :-( – dragonov7 Dec 4 '14 at 22:11
  • This answer has been updated to reflect the new evidence. – Andrew B Dec 4 '14 at 22:37
  • So you're saying my IP is been bloqued by some IDS on the way to the name server... I'll have to check this with my ISP and myself on the Internet... but Andrew, how about the other domain with problems?, in some cases I am not getting the nameserver's IP address as in the dhl example, also we could deduct that the blocking is coming from 172.24.0 network... but then, how is it that I am able to query other servers that also pass through this network? – dragonov7 Dec 4 '14 at 23:04
3

Have you checked your firewall is not blocking port 53 TCP - It looks to me that the dhl.com zone is signed and is thus quite large - for large requests DNS falls back from UDP to TCP, and that might explain your problem.

  • Good catch, that would do it. – Andrew B Dec 4 '14 at 21:54
  • Hello davidgo, we have checked our firewall and in fact I can do queries to other domains with no problem at all, for example we can query yah0o.com, google.com, www.redhat.com, etc... the problem seems to be with specific domains! :-(, Also if it was a firewall problem we would get no answer at all from dig +trace www.dhl.com. – dragonov7 Dec 4 '14 at 21:58
  • 1
    @dragonov7 DNS normally relies on UDP, except in specific scenarios. Large reply packets are one of those. Try running a query by hand with +tcp to confirm that your firewall is allowing TCP 53 as well as UDP 53. – Andrew B Dec 4 '14 at 22:02
  • No problems Andrew... our firewall allows tcp and udp packets... I'll post the result from: dig +tcp www.redhat.com on my original question for you to check. – dragonov7 Dec 4 '14 at 22:04
  • 1
    I am running the all tests from the name server that has the problem... also, /etc/resolv.conf on this server has itself as its name server. – dragonov7 Dec 4 '14 at 22:08
0

sometimes nothing is logged because the query is not possible. This error appears, in example, when allow-recursion{} or other value needed for the queries is disabled. Check your named.conf file

-1

I commented this line in my conf:

query-source address * port 53;

Then I was just going to configure the logging feature, but I tried once again to nslookup one of the problem domains. It worked this time. I don't know how and why, but it probably have something to do with the query-source address port. I went on the second DNS server where that line was not commented yet, tried to nslookup the same domain and it failed. At the moment I removed the line, everything start working normal again. Do you have any idea why and how this stuff works?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.