I have a group of offices that are all connected to the main office via DSL links on the far end to save costs. (We're a non-profit, don't ask)
We've historically had noticeable problems with the link-up between the ISP that handles our remote sites and the ISP that handles the T1 lines our OpenVPN runs on, so those links frequently go down.
Our mail server's public interface is on the 1st provider's network so it worked just fine, but it's much slower because it's also DSL.
To address the upstream network unreliability issues, I'd written a script that simply modifies the DNS records on the remote sites to point to the internal IP if the tunnel is up or the public IP if the VPN tunnel to the main site is down.
How can I do this in a more elegant fashion that will be instantaneous (instead of my cron-driven scripts) and transparent to the users?
Edit: Remote Offices: Ubuntu 9.10 LTSP servers running various vendor-provided Actiontecs & Motorola and a few with Netgears and Linksys firewall. Main office: Almost 100% Linux (CentOS, in this case) with multiple Netgear FVS318/338 series firewalls with individual firewalls for each IP on our /27. (another don't ask, it was before I got here)