We recently had to perform a disaster-recovery fail-over from our main site, which involved changing the IPs at which our main office Domain Controllers (which also serve DNS) reside. It went off well enough, considering, but in the process we discovered something very weird about one of our sites. (We have a primarily hub-and-spoke site layout, for what it's worth.)
At this one site, and only this one site, all our client machines retained the old DC IPs for their DNS servers. Doing an
ipconfig /renew would fetch the updated DNS settings, but after an hour or so, or the next reboot, the machines would come up with the old DNS servers again.
They're mostly XP machines, though there's a Windows 7 box that behaved the same way (all HP desktops), and of course, they were set to obtain their DNS servers through DHCP. This site has a split-scope DHCP (all Windows DHCP serves), with one local server and one back at the main office, in addition to having a DC on site, which would also serve up DNS if any of the clients were set to point at it.
I checked for any GPOs that might be assigning the old DHCP addresses, made sure that none of our DHCP servers were offering up the old addresses, and even went so far as to uninstall and reinstall the DHCP roles, just in case something was causing the old settings to be served out by DHCP.
After a lot of not being able to figure out what the hell was going on, I resorted to manually setting the DNS servers on the clients at that site to what they should be, so the users would be able function on the network. And because I hadn't figured out the cause, I isolated a client no one was using that I could use to track down the root cause of this issue with. I set it back to get its DNS servers from DHCP, rebooted, and was about to begin some serious troubleshooting, except that it didn't revert to the old addresses anymore. I did the same thing on a couple other machines, in case it was a fluke, and it wasn't.
So after manually assigning DNS servers, these machines stopped reverting to the old DNS servers when told to obtain DHCP automatically.
So after manually assigning DNS servers, these machines stopped reverting to the old DNS servers when told to obtain DHCP automatically, for a day or two. I just caught one reverting about an hour ago. During the reversion, I was running this rouge DHCP server checker at five minute intervals, which found nothing. It sure seems like a rogue DHCP server, but I can't find one, and the clients in question still list their DHCP server as the correct one, even when their DNS changes back.
Anyone have any idea what would cause this behavior?