Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I know this has to just be a lack of my understanding but here's the problem.

We recently changed DNS servers from 192.168.1.1 to .2, so I went around to all 8 of my linux servers and changed /etc/resolv.conf to reflect the change. Note that they're all static, there's no DHCP involved.

After making the change I can immediately test the results using nslookup and dig, and it all looks good. I did a /etc/init.d/networking restart - to restart the networking subsystem - and restarted apache and postfix on each of the servers, just to be sure.

A few days later I get a report stating on of our websites isn't sending emails anymore. Perusing the logs I found that the mod_php process couldn't resolve dns entries to send mail. After beating my head on it for about 30 mins I rebooted the server and everything returned to normal.

The next day on a different server (using CentOS rather than our normal Ubuntu), I get a report stating that emails aren't going through, and sure enough looking at the logs indicates that Postfix can't resolve names. Rebooted and it almost instantly delivers all the queued mail.

So what am I missing here? What portion of this process did I fail to understand correctly?

share|improve this question

4 Answers 4

up vote 9 down vote accepted

You probably got bitten by nscd: http://linux.die.net/man/8/nscd

Cheers

share|improve this answer
    
Thanks! It seems highly possible this is what was giving me problems. I wasn't even aware local dns caching was part of the common linux system. –  Gray Aug 31 '09 at 20:51
    
Did you actually test? Jason's hypothesis is possible but not certain. –  bortzmeyer Sep 1 '09 at 8:01
    
@bortzmeyer - yes, I agree. Your own answer is the same as I would have given (and indeed have to two related questions recently). It's much more likely to be cached res_init() state than nscd. –  Alnitak Sep 3 '09 at 12:39

Probably some caching going on. We had a similar problem with sendmail and just restarting the service fixed it.

Sometimes it's easier to just reboot the server and clear all those caches anywhere in the system than spend all that time identifying which service is caching too long. On the other hand, it can turn out to be an investment when it happens again and you know which service to restart.

share|improve this answer
    
I agree, rebooting is the easiest way out, but if the server is critical finding time to reboot can be difficult. Thanks for your help! –  Gray Aug 31 '09 at 20:52

resolv.conf directs resolvers on where to look for names. In most cases, this is going to be the libc resolver, but there may be other cases such as vPostMaster which uses the Python DNS resolver library for SPF lookups.

So, it COULD be that the resolver is caching the resolv.conf information for long-running processes, but it sounded like you restarted postfix, which should have caused it to start using a fresh resolv.conf file.

Check your /etc/nsswitch.conf to see if it specifies anything special happening for "hosts". For example, the default Fedora 11 line on my laptop is:

hosts: files mdns4_minimal [NOTFOUND=return] dns

So in this case it uses mdns as well as /etc/hosts and DNS. In this case, if DNS changes weren't being picked up, I'd wonder if it were the mdns that were causing it.

Sean

share|improve this answer

Most applications intialize the resolver once, at startup (with res_init), and never do it again afterwards. It is not a problem for short-life applications like ping but more serious for long-running daemons.

The Apache process (which runs mod_php) was probably in that case. Restarting Apache would have suffice.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.