I know this has to just be a lack of my understanding but here's the problem.
We recently changed DNS servers from 192.168.1.1 to .2, so I went around to all 8 of my linux servers and changed /etc/resolv.conf to reflect the change. Note that they're all static, there's no DHCP involved.
After making the change I can immediately test the results using nslookup and dig, and it all looks good. I did a /etc/init.d/networking restart - to restart the networking subsystem - and restarted apache and postfix on each of the servers, just to be sure.
A few days later I get a report stating on of our websites isn't sending emails anymore. Perusing the logs I found that the mod_php process couldn't resolve dns entries to send mail. After beating my head on it for about 30 mins I rebooted the server and everything returned to normal.
The next day on a different server (using CentOS rather than our normal Ubuntu), I get a report stating that emails aren't going through, and sure enough looking at the logs indicates that Postfix can't resolve names. Rebooted and it almost instantly delivers all the queued mail.
So what am I missing here? What portion of this process did I fail to understand correctly?