2

I have a encountered a DNS resolution issue. Could someone pls give me some tips on this issue, thanks in advance~

uname -a

Linux 152a580f-e3c2-405f-acde-eac4d928af22 4.4.0-111-generic #134~14.04.1-Ubuntu SMP Mon Jan 15 15:39:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/resolv.conf

nameserver 127.0.0.1
nameserver 10.104.64.25
nameserver 10.104.65.25
options timeout:5 attempts:4 rotate

Issue descriptions:

I have 3 name servers used in my resolve file, and the 127.0.0.1 is listened by a local CONSUL dns, which would able to resolve hostnames under domain 'cf.internal.'

And another 2 nameservers are my local dns servers, which would resolve my internal domain: dummysite.com, as well as recursive queries to public dns names.

The issue is: There's an application, wants to resolve 'bbs.service.cf.internal.', but I can see some fails in logs, like:

{"timestamp":"1542522679.406200409","source":"rep","message":"rep.running-bulker.sync.batch-operations.do-request.failed-doing-request","log_level":2,"data":{"error":"Post http://bbs.service.cf.internal:8889/v1/actual_lrp_groups/list: dial tcp: lookup bbs.service.cf.internal: no such host","session":"13.1.1.3"}}

But, after a while, the application would finally able to get to the right DNS entry and application works.

So far what I expected is: As I have 'rotate' in my options, so the dns query would be like this:

first query would try: nameserver 10.104.64.25, and then try second nameserver 10.104.65.25, and then try another nameserver 127.0.0.1, and bingo, find it 'bbs.service.cf.internal'.

But I used tcpdump, the process is not the same as my thoughts. From the log, I find it's process like this:

QUERY1: 10.104.64.25 => QUERY2: 10.104.65.25 => QUERY3: 10.104.64.25 => QUERY4: 10.104.65.25 => QUERY5: 127.0.0.1 (got it)

Why the DNS query in such a sequence?

The tcpdump logs as a reference:

10.104.148.102.48457 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 26743+ A? bbs.service.cf.internal. (41)
10.104.148.102.48457 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 5283+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.48457: [udp sum ok] 26743 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m16s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.48457: [udp sum ok] 5283 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m16s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)

10.104.148.102.54378 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 51897+ A? bbs.service.cf.internal. (41)
10.104.148.102.54378 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 32472+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.54378: [udp sum ok] 32472 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m43s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.54378: [udp sum ok] 51897 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m43s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)

10.104.148.102.47650 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 23809+ A? bbs.service.cf.internal. (41)
10.104.148.102.47650 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 4790+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.47650: [udp sum ok] 23809 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m15s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.47650: [udp sum ok] 4790 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m15s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)

10.104.148.102.42652 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 60294+ A? bbs.service.cf.internal. (41)
10.104.148.102.42652 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 24929+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.42652: [udp sum ok] 60294 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m42s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.42652: [udp sum ok] 24929 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m42s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)

localhost.46454 > localhost.domain: [bad udp cksum 0xfe44 -> 0xde60!] 41944+ A? bbs.service.cf.internal. (41)
localhost.46454 > localhost.domain: [bad udp cksum 0xfe44 -> 0x924b!] 54509+ AAAA? bbs.service.cf.internal. (41)
localhost.domain > localhost.46454: [bad udp cksum 0xfe76 -> 0x5e62!] 54509* q: AAAA? bbs.service.cf.internal. 0/1/0 ns: cf.internal. [0s] SOA ns.cf.internal. postmaster.cf.internal. 1542524177 3600 600 86400 0 (91)
localhost.domain > localhost.46454: [bad udp cksum 0xfe54 -> 0xff5e!] 41944* q: A? bbs.service.cf.internal. 1/0/0 bbs.service.cf.internal. [0s] A 10.104.149.223 (57)
  • I don't understand the issue. You expect to see three queries, trying all three DNS servers, and that is exactly what is happening. The only thing is that you only take IPv4 into account while DNS also queries for an IPv6 address. – Tommiie Nov 18 '18 at 11:15
  • @Tom yes, it's trying all three, but why the sequence is: cn1c6ocvcu01-> cn1c6ocvcu02->cn1c6ocvcu01->cn1c6ocvcu02->127.0.0.1 ? As of the "rotate" option, it should be: cn1c6ocvcu01 -> cn1c6ocvcu02 -> 127.0.0.1 I think... – roamer Nov 18 '18 at 14:50
1

Your preferred setup in this case is to utilize DNSMasq

Add to/modify /etc/consul.d/config.json

"dns-config": {
    "recursors" : [ "10.104.64.25", "10.104.65.25" ]
}
"ports": {
    "dns": 8600
},

Now install DNSMasq and then configure as below

/etc/dnsmasq.d/00-base.conf

# Never forward plain names (without a dot or domain part)
domain-needed

# Never forward addresses in the non-routed address spaces.
bogus-priv

# Disable negative caching.
no-negcache

# Point to our upstream servers
resolv-file=/etc/resolv.dnsmasq.conf

/etc/dnsmasq.d/10-consul.conf

# Forward queries for ".cf.internal" TLD to the Consul Agent.
server=/cf.internal/127.0.0.1#8600

/etc/resolv.conf (keep this entry remove the rest)

nameserver 127.0.0.1

/etc/resolv.dnsmasq.conf

nameserver 10.104.64.25
nameserver 10.104.65.25
  • I understand this method to take a leverage of DNSMASQ, and consul is also take another two dns nameservers as a recursor: "-recursor=10.104.64.25 -recursor=10.104.65.25", it's the same story as you have proposaled here. But I was running a CloudFoundry Cell (container) on this host, so basically, the containers spun up by this host would be use the same DNS nameservers and options (except 127.0.0.1) as the mother host. so i want to pass in the 'rotate' option into the containers actually. but as far as i have the 'rotate' option in this host, resolution becomes weird as above descriptions.. – roamer Nov 18 '18 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.