This morning, we found that (due to a changeover) one of our DNS records to an important service is wrong. It has been changed on our primary DNS server, but clients at secondary sites do not see the change. (Our network is running almost entirely with OS X 10.5 Servers and OS X 10.5 clients).

Let me name some machines for example purposes:

  • primary = the primary DNS server
  • secondary = a secondary DNS server
  • client = a client at a secondary site
  • service.ourdomain.com = the service whose DNS records have changed

On the client, (which does DNS lookups through secondary), when probing how things are configured, I get:

nslookup service.ourdomain.com
** server can't find service.ourdomain.com: NXDOMAIN

nslookup service.ourdomain.com secondary
** server can't find service.ourdomain.com: NXDOMAIN

nslookup service.ourdomain.com primary
(returns appropriate information about how to contact the service)

When I ssh into

  • secondary, which does its DNS lookups through primary
  • or primary itself, which does DNS lookups from itself

I get:

nslookup service.ourdomain.com
(returns appropriate information about how to contact the service)

nslookup service.ourdomain.com secondary
** server can't find service.ourdomain.com: NXDOMAIN

nslookup service.ourdomain.com primary
(returns appropriate information about how to contact the service)

I'm perplexed. Secondary seems to know where the service is, but does not return the values when queried. (Granted, the DNS entries it can be entirely independent or what it returns when queried for a domain name, but still -- it looks like it should know!)

I have tried flushing the DNS on secondary and on client. (dscacheutil -flushcache). I have also stopped and restarted DNS on secondary. (sudo serveradmin stop dns and sudo serveradmin start dns)

At our primary site, my coworker rebooted primary and a client there to get the name to resolve right. Unfortunately, we have 14 secondary sites, and I'd rather not reboot the servers, which are sharing files, during the day if possible, but will do it if it solves the problem.

Per request:

host -C ourdomain.com   # [with names substituted]:
ourdomain.com SOA record primary.ourdomain.com. admin.ourdomain.com. 2009121410 21600 3600 604800 345600

[I have no idea what admin.ourdomain.com is -- I don't believe we have a box by that name; I sure can't ping it. The primary DNS server shows up right, though.]

Also per request, here is the output of dig service.ourdomain.com @secondary (with name substitutions):

; <<>> DiG 9.4.3-P1 <<>> service.ourdomain.com @secondary
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19207
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;service.ourdomain.com. IN  A

ourdomain.com.      10800   IN  SOA primary.ourdomain.com. admin.ourdomain.com. 2009121409 21600 3600 604800 345600

;; Query time: 3 msec
;; SERVER: [IP of secondary]#53([IP of secondary])
;; WHEN: Mon Dec 14 10:34:11 2009
;; MSG SIZE  rcvd: 88

And the output of dig service.ourdomain.com @primary:

; <<>> DiG 9.4.3-P1 <<>> service.ourdomain.com @primary
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47885
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;service.ourdomain.com. IN  A

service.ourdomain.com. 10800    IN  A   [IP of service]

ourdomain.com.      10800   IN  NS  primary.ourdomain.com.

primary.ourdomain.com.  10800   IN  A   [IP of primary]

;; Query time: 8 msec
;; SERVER: [IP of primary]#53([IP of primary])
;; WHEN: Mon Dec 14 10:34:18 2009
;; MSG SIZE  rcvd: 92

The most striking differences are that secondary did not reply with an answer, and that the primary said, ";; WARNING: recursion requested but not available".

  • Not familiar with MACOSX Serers but I would use dig service.ourdomain.com @NS_Server_IP to confirm it is returning the right result. If your secondary is still reporting incorrectly it has the wrong result cached. Just for the future, make sure to take TTL down to 5 min or so before switching DNS entries, it makes changing IPs easier :) – Dave Drager Dec 14 '09 at 16:27
  • 1
    Good news: now that it has been a couple of hours, the change has propagated by itself. I still want to know how to propagate things more quickly in the future, though. – Clinton Blackmore Dec 14 '09 at 17:21

You can manually force a zone transfer using the rndc utility. Run this command on all of your secondary DNS servers:

rndc -p 54 retransfer mydomain.example.com IN com.apple.ServerAdmin.DNS.public

You can also use this utility to reload your configuration without restarting named.

rndc -p 54 reload
| improve this answer | |

Without knowing your configuration, I would guess this is a caching issue, or a DNS propagation issue.

Without knowing your domain you're using, I can't really test it from here. I personally don't understand why people omit this sort of relevant information, but they often do.

  • Try "host -C yourdomain.com" and tell me what you see. If you see different SOA records with different serial numbers, then you need to fix your DNS propagation. IF the secondary is not listed in the NS records for this zone, add an "also-notify" line if running BIND.

  • Try changing the serial number on the master to ensure it was properly changed, as well as to test propagation.

  • Try setting a better negative cache time that is much smaller, say 600 (10 minutes) or so. This is one of the values in the SOA record.

  • Try a "dig hostname.yourdomain.com @secondaryserver" and see what it returns. Do the same on primary. If they differ, that is the brokenness.

  • If each of these sites that are returning bad data have a huge cache time, you should be able to ssh to them and simply restart the name server, not reboot each site fully. BIND will quickly restart if that is what is in use.

| improve this answer | |
  • I have added the results of host -C mydomain.com to my post. The numbers look the same when I do it on client, secondary, and primary. (Is the first number the serial number?) The secondary service is not listed in the chain. Mac OS X provides a graphical tool for configuring services, such as DNS, which does not make it at all clear how things map onto config files. [I have not configured DNS at a low level, and my co-workers might be upset if I were to muck with it, as then changes made graphically may or may not stomp on changes made in config files.] – Clinton Blackmore Dec 14 '09 at 17:33
  • The dig results differ. – Clinton Blackmore Dec 14 '09 at 17:35
  • [It looks like I'm going to be away from this problem for a bit -- I intend to post something of the dig results in my question.] – Clinton Blackmore Dec 14 '09 at 17:40
  • If the dig results differ, then you might simply need to bump you serial number on the master and tell it to reload the zone. It sounds like DNS data was changed without bumping the serial number, which means changes would never propagate. btw, the "admin.ourdomain.com" really maps into admin@ourdomain.com -- it is the contact field, and this is the conversion performed. – Michael Graff Dec 14 '09 at 17:43
  • I think I've found how to bump it and flush it -- discussions.apple.com/thread.jspa?messageID=10738953 -- I'll add a note to my todo list to try next week when school is out. – Clinton Blackmore Dec 14 '09 at 22:50

Your secondary server is trying to recursively answer (RD - recursion desired, RA - recursion available) but failing (NXDOMAIN) whilst at the same time also serving the SOA record authoritatively (AA - authoritative answer).

You do seem to have a slightly odd mix here... we need to establish how it is that your secondary server knows about the zone (the SOA record) but doesn't know about the record within the zone.

I'd go with Michael's recommendation - bump the serial number on the master, and then restart BIND on the secondary to ensure that its cache is cleared.

| improve this answer | |
  • Thanks. I had no idea what all those acronyms meant. I didn't set up DNS originally, and, to be sure, don't know a lot about it. – Clinton Blackmore Dec 16 '09 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.