2

I spent several months setting up what I thought was an efficient and best-practice configuration for our organization's DNS infrastructure.

DNS1 - master server hosting internal zone files and forwarding external queries to NS1/NS2

DNS2 - slave server to DNS1 for same purposes

NS1 - DNS server that hosts WAN-facing zone files and also performs recursive queries passed from DNS1/2 for internal clients

NS2 - slave to NS1 for same purposes

All four DNS servers are OpenBSD 5.4 virtual machines in VMware environment.

In normal circumstances, this setup has worked very well the last couple months. However, we are seeing that when DNS1 goes down (VM or ESX server crash), our entire organization's ability to resolve IPs goes down.

I implemented this entire solution with the intention that our clients would quickly resolve to DNS2 if DNS1 was ever unavailable. Clients receive both DNS IPs through DHCP. In this particular outage today, DNS1 was pingable, accessible via SSH, listening on port 53 (netstat -an), but telnet to port 53 was timing out from a client command line (likely due an ESX host crash issue). However, regardless of what's wrong with DNS1, I would assume that it wouldn't affect DNS2 from a client query resolution standpoint (DNS2 was on a different ESX host).

Through additional research, I am seeing that Mac OS X (99% of our internal clients) has no set server order for IP resolution - meaning that it will pick whichever provided DNS server it wants. I am also seeing that the timeout for the DNS resolution can take anywhere from 30 seconds to 15 minutes before the DNS server is deemed "ineligible".

Another issue on this site appeared to be facing similar problems from a few years ago, but no resolution was provided.

Questions:

1) Is the "downtime" our clients reporting due to everyone attempting to use DNS1 first and timing out (which I thought didn't happen in Mac now)? Why aren't they attempting to query DNS2?

2) If they are attempting to query DNS2, why isn't it responding to direct queries (even using dig) when my master DNS server is unavailable (both when it's pingable and not pingable)?

2) Is there anything I need to configure on the Mac clients to ensure that they can successfully query the second DNS server in a timely manner when the other is down? (ex: timeout option)

DNS1/2 sample zone file header:

$TTL 10800 @ IN SOA dns1.myorganization.edu. admin.myorganization.edu. ( 2014111301 ;Serial 3600 ;Refresh for secondary servers to check for updates 600 ;Retry wait time if secondary server update fails 604800 ;Expire 86400 ;Negative caching TTL )

I can provide named.conf files if needed, too.

I want to continue thinking that a redundant DNS setup is smart, but if the secondary server refuses queries every time DNS1 goes down and this is expected behavior, it seems kind of pointless and only adds unnecessary complexity. Any and all help is appreciated!

EDIT: Attached is DNS2's named.conf file - TSIG secrets and domains changed to protect the innocent

// $OpenBSD: named-simple.conf,v 1.10 2009/11/02 21:12:56 jakob Exp $
//
// Example file for a simple named configuration, processing both
// recursive and authoritative queries using one cache.


// Update this list to include only the networks for which you want
// to execute recursive queries. The default setting allows all hosts
// on any IPv4 networks for which the system has an interface, and
// the IPv6 localhost address.
//
acl "internal" {
    127/8; 10.0.0.0/8; 192.168.1.0/24;
};

options {
    version "BIND"; // remove this to allow version queries

    #listen-on    { any; };
    listen-on port 53 { 127.0.0.1; 10.10.250.35; };
    listen-on-v6 { none; };

    dump-file       "data/cache_dump.db";
    statistics-file "data/named_stats.txt";
    memstatistics-file "data/named_mem_stats.txt";

    #empty-zones-enable yes;

    # This DNS server is responsible for fully answering the query 
    recursion yes;
    allow-recursion { "internal"; };
    allow-query { "internal"; };

    # Forward all non-authoritative queries to external DNS servers (ns1/2)
    forwarders { 10.10.250.36; 10.10.250.37; };
    # Only forward these queries, don't attempt to find it yourself
    forward only;

    # Don't notify the other NS servers in the zone files of zone updates
    notify no;
    allow-transfer { none; };

    max-cache-size 512M;

    zone-statistics yes;

};

key dns1-dns2.myorganization.edu. {
    algorithm       hmac-md5;
    secret          "xBBxdjaocjbe33js99zsAG0s/+3g==";
};

# Master server IP
server 10.10.250.1 {
    keys            { dns1-dns2.myorganization.edu.; };
};

logging {
    channel my_query_logging {
            #file "log/queries.log";
            syslog local6;
            severity info;
            print-time yes;
            print-severity yes;
            print-category yes;
    };
    channel my_security {
            #file "log/security.log";
            syslog local6;
            severity notice;
            print-time yes;
            print-severity yes;
            print-category yes;
    };
    channel my_default {
            #file "log/messages.log";
            syslog local6;
            severity info;
            print-time yes;
            print-severity yes;
            print-category yes;
    };
    category queries { my_query_logging; };
    category default { my_default; };
    category security { my_security; };
    category lame-servers { null; };
};

key "rndc-key" {
    algorithm       hmac-md5;
    secret          "PId3xd9Swlc7sniOSGntyDxw==";
};

controls {
    inet            127.0.0.1 port 953
                    allow { 127.0.0.1; }
                    keys { "rndc-key"; };
};

// Standard zones
//
zone "." {
    type hint;
    file "etc/root.hint";
};

zone "localhost" {
    type master;
    file "standard/localhost";
    allow-transfer { localhost; };
};

zone "127.in-addr.arpa" {
    type master;
    file "standard/loopback";
    allow-transfer { localhost; };
};

zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" {
    type master;
    file "standard/loopback6.arpa";
    allow-transfer { localhost; };
};


zone    "myorganization.edu" IN {
    type slave;
    file "slave/db.myorganization.edu.zone";
    masters { 10.10.250.1; };
    forwarders { };
};
zone    "250.10.10.in-addr.arpa" IN {
    type slave;
    file "slave/db.250.10.10.in-addr.arpa.zone";
    masters { 10.10.250.1; };
    forwarders { };
};

EDIT2: Including dig output

client-osx$ dig @10.10.250.35 server1.myorganization.edu
; <<>> DiG 9.8.5-P1 <<>> @10.10.250.35 server1.myorganization.edu
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

EDIT3: Including resolv.conf file of DNS2 server

$ cat /etc/resolv.conf 
lookup file bind
nameserver 127.0.0.1
nameserver 10.10.250.36 (NS1)
nameserver 10.10.250.37 (NS2) 

EDIT4: Additional thoughts and testing

I'm thinking that DNS1 was in such a weird "mostly-but-not-quite-fully up" state that it may have affected DNS2 in some way (but how and why is the real question). I am going to test this again by shutting off DNS1 and making sure DNS2 functions as expected. Assuming DNS2 responds to queries normally, this will take me down the path to figuring out how to decrease the time it takes for a Mac OS X client to move to a 2nd DNS server in its configuration. Any suggestions on this? As explained above, would modifying the resolv.conf file be the best approach with a timeout option?

EDIT5: DNS2 Routing Tables

admin@dns2:~ $ route -nv show
Routing tables
Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface Label
default            10.10.250.254      UGS        2   469221     -     8 em0  
10.10.250/24       link#1             UC        12        0     -     4 em0  
10.10.250.1 (DNS1) 00:50:56:a2:01:fe  UHLc       1     1658     -     4 em0  
10.10.250.4        40:6c:8f:39:f6:81  UHLc       0     1617     -     4 em0  
10.10.250.27       00:0c:29:84:9c:3d  UHLc       0       16     -     4 em0  
10.10.250.36 (NS1) 00:50:56:a2:09:06  UHLc       0     1480     -     4 em0  
10.10.250.37 (NS2) 00:50:56:a2:41:7c  UHLc       0     2591     -     4 em0  
10.10.250.41       00:24:36:f4:83:84  UHLc       0      108     -     4 em0  
10.10.250.46       3c:07:54:10:b2:ba  UHLc       0        1     -     4 em0  
10.10.250.48       00:50:56:a2:04:a6  UHLc       1      663     -     4 em0  
10.10.250.49       00:50:56:a2:2d:fb  UHLc       1      577     -     4 em0  
10.10.250.60       00:50:56:a2:73:2e  UHLc       0    45587     -     4 em0  
10.10.250.253      00:17:c5:16:53:80  UHLc       0        0     -     4 em0  
10.10.250.254      02:04:96:35:e1:8a  UHLc       1        0     -     4 em0  
127/8              127.0.0.1          UGRS       0        0 33192     8 lo0  
127.0.0.1          127.0.0.1          UH         2     3599 33192     4 lo0  
224/4              127.0.0.1          URS        0        0 33192     8 lo0
  • BTW, you don't have super short TTLs on the domains, right? – NickW Nov 18 '14 at 16:33
  • Different ESX hosts (CPU, RAM) and disks are stored on different SAN arrays. I will continue looking at logs, but so far, nothing. And TTL for all internal zones is 3 hours (as shown in the sample zone above) – majedian21 Nov 18 '14 at 16:35
  • @majedian21 Would it be possible to include what is happening if you use dig to query dns2 directly (preferably with server IP specified to remove the system resolver from the equation)? Also, does dns2 normally work or is it just not noticeable that it doesn't work until dns1 is down? – Håkan Lindqvist Nov 18 '14 at 18:09
  • @majedian21 That was without specifying which server to query, though. Should I assume that was directed at dns1? – Håkan Lindqvist Nov 18 '14 at 19:12
  • Also including this if it helps: $ cat /etc/resolv.conf lookup file bind nameserver 127.0.0.1 nameserver 10.10.250.36 (NS1) nameserver 10.10.250.37 (NS2) – majedian21 Nov 18 '14 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.