You want DNS to be reliable. Adding a huge amount of complexity to the setup will cause an absolute nightmare when something breaks.
Some of the proposed solutions only work when the redundant DNS servers are at the same site.
The fundamental issue is that the DNS client is broken as designed. It doesn't remember when a server was unreachable, and keeps trying to connect to the same nonresponsive server.
NIS handled this issue by having ypbind keep state. A clumsy solution, but it usually works.
The solution here is to lean on vendors to implement a reasonable solution to this problem. It's getting worse with IPV6, as the AAAA requests are adding to the length of time spent wasted on timeouts. I have seen protocols fail (e.g. an sshd connection) because they spent so much time waiting on DNS timeouts due to a single unreachable DNS server.
In the interim, as has been previously suggested, write a script that replaces resolv.conf with one that contains only valid nameservers. Share this script with vendors to demonstrate the unclean solution that you were forced to implement.
This hasn't been seriously tested, and it assumes an nslookup that parses like mine, and a grep that supports "-q".
Run this out of cron every 5 minutes or so.
I'm not seriously suggesting that anyone actually use cron and a shell script for critical failover management, the error-handling surprises are just too great. This is a proof of concept only.
To test this for real, change the "nameservers=" line at the top, change resolv_conf at the top to /etc/resolv.conf not /tmp/resolv.conf, and the default header for resolv.conf that contains example.com.
You may need to restart nscd if you replace resolv.conf.
#!/bin/bash
# full list of nameservers
nameservers="127.0.0.1 192.168.0.1 192.168.1.1"
# resolv.conf filename, change to /etc/resolv.conf for production use
resolv_conf="/tmp/resolv.conf"
# for tracking during the test
failed_nameservers=""
good_nameservers=""
# test loop
for nameserver in $nameservers; do
if nslookup localhost $nameserver | grep -q 'Address.*127\.0\.0\.1'; then
good_nameservers="$good_nameservers $nameserver"
else
failed_nameservers="$failed_nameservers $nameserver"
fi
done
# if none succeded, include them all
if [ -z "$good_nameservers" ]; then
good_nameservers="$nameservers"
fi
# error reporting, consider writing to syslog
if [ -n "$failed_nameservers" ]; then
echo warning: failed nameservers $failed_nameservers
fi
# create the temporary replacement resolv.conf
new_rc="$resolv_conf.new.$$"
echo domain example.com > $new_rc
echo search example.com >> $new_rc
for nameserver in $good_nameservers; do
echo nameserver $nameserver >> $new_rc
done
# don't deploy a corrupt resolv.conf
if ! grep -q nameserver $new_rc; then
echo warning: sanity check on $new_rc failed, giving up
exit 1
fi
# keep a backup
if [ -f $resolv_conf ]; then
rm -f $resolv_conf.previous
ln $resolv_conf $resolv_conf.previous
fi
# deploy the new one
mv $new_rc $resolv_conf