The problem with this solution is that DNS will continue to hand out both IP's even if one server/ISP is down which means it won't really accomplish your goal. If you really need a true backup site, the best way to do this is to setup BGP.
It's a process, but you work with your ISP to setup a BGP router at each site advertising your public network. When one router or ISP goes down, it's a matter of seconds for the backup router to begin advertising your network from the backup site. No DNS changes, no waiting for anything to timeout or clear - it just works.
EDIT** Add secondary solution.
If setting up BGP is simply too far out of reach due to budget or some other constraint, I would recommend you setting up a DNS server at each site with a very short TTL (like 10 minutes assuming you don't have a HUGE amount of traffic). On each DNS server, create a single A record with the IP of the server in that site. So, in site A, configure a host record with the IP in site A and in site B create one with the IP in site B. This way, when the ISP goes down, you will no longer be serving out IP's that are unavailable.
Keep in mind with this solution, you will have clients that have cached the "bad" IP and they will have to wait for that to timeout before they will put the "good" IP from the other DNS server.