1

A bit of a weird problem here and hope someone can help describe and/or fix with me. We were on a /24 network A.A.A.A and recently switched to B.B.B.B. The nameservers were all updated with the registrar, they show in the root nameserver and everything - so far so good, right? I can dig the root servers and get the new IPs, whois is good and all the "is it plugged in?" groundwork looks great.

I'm running DNS on the new network at B.B.B.254 and the old network at A.A.A.254 as the changes still propagate around the world. The A.A.A.254 DNS server is actually feeding the world the IPs for B.B.B.B to catch stragglers who haven't seen the upstream changes yet, normal business. Each server also runs reverse DNS for it's own IP range.

SO! In the logging of queries on the old A.A.A.254 server I am seeing an odd request for the PTR zone from B.B.B.254 for it's own reverse IP B network! And, it's always for two IPs -- my firewall (.4) and VPN server (.6) - no others than these two.

04-Aug-2009 09:56:09.755 client B.B.B.254#37384: query: 4.B.B.B.in-addr.arpa IN PTR +

04-Aug-2009 10:00:05.380 client B.B.B.254#37385: query: 6.B.B.B.in-addr.arpa IN PTR +

Why is there a query like this, asking for the IP zone that B.B.B.254 is already hosting, going to my A.A.A.254 server?! I shouldn't be seeing any queries from B.B.B.254 to A.A.A.254 at all, yet I'm still getting these odd PTR jobs that I can't quite figure out. A 'dig -x' from B doesn't cause it to go to A, so it's nothing that simple. I've scoured every config file on B (it's a linux/bind9 box) for any reference to the A network, can't find anything.

Anyone have a clue what's going on here and how I can fix it? Why is the new server B asking A for this PTR, when it knows it hosts that range itself?

I've followed various trails such as:

dig +norecurse 4.B.B.B.in-addr.arpa PTR @x.arin.net

...then...

dig +norecurse 4.B.B.B.in-addr.arpa PTR @a.[authority server name]

...and that all looks right. Any ideas welcome where further to look to track down what's causing this request loop, I'm at a bit of a loss what to look at next.

4 Answers 4

2

Are those log excerpts verbatim apart from the IP addresses?

If so, it seems surprising that over a 4 minute long interval there was apparently no other IP traffic, as indicated by the source port of those queries only going up by one.

I suggest that you should attempt to confirm that it really is B.B.B.254 sending those queries (run tcpdump on the sending host).

If you can then try to find out which process is causing them. If it's named on the new server, enable more logging to find out why (rndc trace and rndc querylog).

Note that if you've been renumbering your network and reconfiguring your machines without taking them down then any long-running daemons may still have the old /etc/resolv.conf cached in them.

Straying into programming land, this is because the /etc/resolv.conf is only read by the res_init() function from the resolver library the first time any program calls gethostbyname() or gethostbyaddr(). The resolver library doesn't detect changes to the config file on the fly.

1
  • 1
    Thanks, I'm following a similar trail this morning as I get back at it -- I sync'd the clocks and enabled massive debugging on both servers and think I might see a pattern. Nothing looks wrong per the logs at first digestion. The frequency of the log entries suggests it's some sort of automated process that is involved; so far I've ruled out Nagios and NTOP, Cacti polling is next on the chopping block. Thanks for the tcpdump idea, I'll fire it up on b.b.b.254.
    – user15590
    Aug 5, 2009 at 16:58
2

Is resolv.conf on B.B.B.254 perhaps configured to send requests to the old DNS server? (It shouldn't be, because recursive and authoritative shouldn't mix). Also, is B.B.B.254 perhaps configured as the default gateway for an RFC1918 range with NAT? In that case, something behind that machine might be misconfigured vis a vis recursive resolution.

Also, having actual IP ranges might be helpful, so people can actually examine your delegations.

2
  • A) resolv.conf is A-OK on B.B.B.254. A find/grep of the entire server shows no IP from the old network (I'd be happy if it were something that boneheaded I did :) ) B) nothing that I have configured on the server ends fits your scenario, however it might be possible a remote VPN client has this configured? We support both server-configured Cisco IPSec and Windows/Mac PPTP self-configured connections, perhaps a remote user has somehow manually added the old network to their PPTP configuration? I'd love to give real IPs/networks but I'm flying this under the radar on purpose, sorry.
    – user15590
    Aug 4, 2009 at 23:05
  • Yes, a VPN counts the same was as "private range", most likely.
    – womble
    Aug 4, 2009 at 23:38
2

I'm happy (ecstatic!) to report it's been solved -- the problem child was in fact syslogd on b.b.b.254! Alnitak sent me down the right path, after running a tcpdump capture on the b.b.b.254 machine several times and putting named into debug level 5 I stumbled upon the fact that the DNS query was being made, yet was not coming from the named process.

I'm no syslogd major expert but it had something to do with the fact that this syslogd runs in remote-logging mode, and accepted the logs from the devices in question (.4, .6). Even though it was properly working on the new network something in it's little daemon brain had still cached info off the old network (I don't quite understand how, though) and it was this process doing the PTR request to the old DNS server for the new reverse IP range. A simple '/etc/init.d/syslogd restart' cleared up the errant requests and now all is well.

It's always the simple things in the end, it just takes 8 hours of debugging to get there. :)

0
1

Shot in the dark: are you sure you've gotten all the DNS caches flushed? I've seen strange stuff like this hang around because something had an old out-of-date view of DNS in some cache that just wasn't getting flushed for some reason...

You must log in to answer this question.