NSLOOKUP and NBLOOKUP give one IP address; PING finds another

Question

In a Windows Server 2008 domain the on-site domain controller serves as both DNS and WINS server. It is also a DHCP server. One workstation is used as a platform for all admin of the various servers in the domain. All servers are shut down at night after a system-wide backup. Every morning there is a period during which no remote desktop sessions can connect to any of the servers which have IP addresses allocated by DHCP. My strategy is to nail down the IP addresses of these boxes, but I want to understand the problem first. This morning the problem is as follows:

Nslookup and Nblookup both agree on the IP address of the target computer, but when I try to ping the target computer, ping somehow resolves the IP address differently and ping fails. My first indication of the problem was when three different remote desktop sessions all failed to connect. If I wait an hour or so after the servers in question have booted up the problem will resolve itself.

On the workstation that cannot connect I have done IPCONFIG /FLUSHDNS. On the DHCP server I set DHCP to use 30 day leases according to the theory that changing IP addresses somehow cause this problem, but the next morning the problem remains. I have confirmed that no computers are mentioned in any HOSTS or LMHOSTS files. I can't figure out where ping is getting its IP address.

Any ideas?

Answer 1

Nslookup and Nblookup both obtain their data directly from the servers, while ping goes through local resources first, hosts file, DNS cache, then DNS lookup (I may have skipped a step) so if it finds an answer there first, that's what sticks.

Answer 2

Is there an entry for that IP address in your local C:\Windows\System32\Drivers\etc\hosts file?

EDIT: I just saw you mention there's nothing in the hosts file. I can't really explain your reported behavior. I would go back to square one and do a series of sanity checks to verify what you think you know is true and correct.

EDIT 2: The DHCP service is configured by default to update DNS as leases are modified. These changes then have to be replicated to all of your AD integrated DNS servers via AD replication. This process can take up to fifteen minutes to fully complete. During this time, some DNS servers may show the correct IP while others might show the old one. You can confirm this by querying the DNS servers directly. Try running nslookup <query> <resolver> where <resolver> is the name is your internal DNS server. Query each one in turn to verify they are returning consistent results.

Answer 3

This problem just went away. The actions I took were first to force DHCP to register IP address changes in DNS (even if not asked to). That did not fix the problem. Next I set leases to 30 days. That also did not fix the problem. After the problem went away I set leases to one hour, but that didn't cause the problem to reccur. Note that none of these actions are logically connected to the actual problem. They fall in the category of grasping at straws. I can not think of any good reason why, after flushing the DNS, NSLOOKUP and NBLOOKUP would give one IP address and ping (and, presumably, the remote desktop application) would find another IP address. (HOSTS and LMHOSTS are default with no IP addresses mentioned.)

Answer 4

This sounds like a DNS problem for sure. Set the TTL to 1 minute and if it has a setting that lets you bump the serial number, bump it.

If that fails, I'd try isolating the possible culprits one by one. Schedule a maintenance window when no one needs a workstation (say, 8PM or however it works for your office). With all the client machines still turned on, portscan the entire netblock to ensure that no one else is running a DHCP, DNS, or wins server. Ensure that every last option on your DNS server is set correctly.

If that doesn't help, I'd start by picking a new IP to be your DNS server. When everyone has gone home, configure the master to hand out that IP to everyone over DHCP and wins. Shut down all the client machines. Migrate the DNS server to a new, clean machine or VM, then shut down the domain master as well.

Then, turn on the domain master and all the client machines. What happens?

Answer 5

Did you ever sort this problem? It sounds like something is/was fundamentally broken with the address assignment on your network.

I would have Wiresharked while pinging and observed the IPs returned by ICMP ECHO and nslookup/nblookup.

If you're on a domain, nslookup by default will be returning the IP as known by the domain's DNS server (and the Domain Controller will often be handing out its own address as the DNS server for DHCP Clients on smaller domains).

Ping relies on ARP - a host joining a network solicits existing hosts on the network to tell it what their IPs are (and each host maintains its own list of known hosts' IPs mapped to their MACs).

It may be that the DC is getting confused about addresses it's assigning, or you may have competing DHCP servers on the LAN (coincidentally assigning IPs in the same range) and some clients are talking to the wrong one.

Either way, I'd be interested to know if you ever found a resolution to this issue.