I have a encountered a DNS resolution issue. Could someone pls give me some tips on this issue, thanks in advance~
uname -a
Linux 152a580f-e3c2-405f-acde-eac4d928af22 4.4.0-111-generic #134~14.04.1-Ubuntu SMP Mon Jan 15 15:39:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 10.104.64.25
nameserver 10.104.65.25
options timeout:5 attempts:4 rotate
Issue descriptions:
I have 3 name servers used in my resolve file, and the 127.0.0.1 is listened by a local CONSUL dns, which would able to resolve hostnames under domain 'cf.internal.'
And another 2 nameservers are my local dns servers, which would resolve my internal domain: dummysite.com, as well as recursive queries to public dns names.
The issue is: There's an application, wants to resolve 'bbs.service.cf.internal.', but I can see some fails in logs, like:
{"timestamp":"1542522679.406200409","source":"rep","message":"rep.running-bulker.sync.batch-operations.do-request.failed-doing-request","log_level":2,"data":{"error":"Post http://bbs.service.cf.internal:8889/v1/actual_lrp_groups/list: dial tcp: lookup bbs.service.cf.internal: no such host","session":"13.1.1.3"}}
But, after a while, the application would finally able to get to the right DNS entry and application works.
So far what I expected is: As I have 'rotate' in my options, so the dns query would be like this:
first query would try: nameserver 10.104.64.25, and then try second nameserver 10.104.65.25, and then try another nameserver 127.0.0.1, and bingo, find it 'bbs.service.cf.internal'.
But I used tcpdump, the process is not the same as my thoughts. From the log, I find it's process like this:
QUERY1: 10.104.64.25 => QUERY2: 10.104.65.25 => QUERY3: 10.104.64.25 => QUERY4: 10.104.65.25 => QUERY5: 127.0.0.1 (got it)
Why the DNS query in such a sequence?
The tcpdump logs as a reference:
10.104.148.102.48457 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 26743+ A? bbs.service.cf.internal. (41)
10.104.148.102.48457 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 5283+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.48457: [udp sum ok] 26743 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m16s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.48457: [udp sum ok] 5283 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m16s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
10.104.148.102.54378 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 51897+ A? bbs.service.cf.internal. (41)
10.104.148.102.54378 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 32472+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.54378: [udp sum ok] 32472 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m43s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.54378: [udp sum ok] 51897 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m43s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
10.104.148.102.47650 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 23809+ A? bbs.service.cf.internal. (41)
10.104.148.102.47650 > cn1c6ocvcu01.dummysite.net.domain: [udp sum ok] 4790+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.47650: [udp sum ok] 23809 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m15s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu01.dummysite.net.domain > 10.104.148.102.47650: [udp sum ok] 4790 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m15s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
10.104.148.102.42652 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 60294+ A? bbs.service.cf.internal. (41)
10.104.148.102.42652 > cn1c6ocvcu02.dummysite.net.domain: [udp sum ok] 24929+ AAAA? bbs.service.cf.internal. (41)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.42652: [udp sum ok] 60294 NXDomain q: A? bbs.service.cf.internal. 0/1/0 ns: . [1h56m42s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
cn1c6ocvcu02.dummysite.net.domain > 10.104.148.102.42652: [udp sum ok] 24929 NXDomain q: AAAA? bbs.service.cf.internal. 0/1/0 ns: . [1h56m42s] SOA a.root-servers.net. nstld.verisign-grs.com. 2018111800 1800 900 604800 86400 (116)
localhost.46454 > localhost.domain: [bad udp cksum 0xfe44 -> 0xde60!] 41944+ A? bbs.service.cf.internal. (41)
localhost.46454 > localhost.domain: [bad udp cksum 0xfe44 -> 0x924b!] 54509+ AAAA? bbs.service.cf.internal. (41)
localhost.domain > localhost.46454: [bad udp cksum 0xfe76 -> 0x5e62!] 54509* q: AAAA? bbs.service.cf.internal. 0/1/0 ns: cf.internal. [0s] SOA ns.cf.internal. postmaster.cf.internal. 1542524177 3600 600 86400 0 (91)
localhost.domain > localhost.46454: [bad udp cksum 0xfe54 -> 0xff5e!] 41944* q: A? bbs.service.cf.internal. 1/0/0 bbs.service.cf.internal. [0s] A 10.104.149.223 (57)