Here's a step-by-step rundown of my end-of-week nightmare:
-Nov 8 @ 2am: One of our two internet providers began experiencing problems. Packets were being dropped and connectivity for our clients (accessing our web services) was being impacted. For insight I host our primary DNS here locally but outside of our network.
-Nov 9 @ 12:30pm: since we currently had only 2 DNS Resolvers set up (us as the primary and another external as a slave) I added 126.96.36.199 (Google public DNS) in as a name server and configured zone transfers to them. I was under the impression that there would be a delay before Google became a slave.
-Nov 9 @ 1:30pm: because of the packet drops I pulled the A records resolving to the IP address belonging to the bad provider. This left us with A records for only one IP (our good provider). The TTL at the time for all was set to 1 hour. After I pulled the records I set the TTL to 5 minutes so that I would be able to quickly add the removed A records back in when the provider resolved their issues.
-Shortly after this our website started disappearing in remote areas of the world.
-Nov 10 @ 7:30am: the bad provider fixes their equipment and I add back the A records that I had previously deleted.
My question is why is my website still inaccessible from certain parts of the world? Doing NSLOOKUP's I'm fine in Italy but not in Australia. Good in most of the USA but not all.
I also found (to my horror) that some DNS servers were caching my A records beyond their stated TTL. I did not even know this was possible. For instance; I created an A record with a TTL of 5 minutes, waited 10 and then checked with Google public DNS (188.8.131.52). It was there. Then I deleted the record, waited 20 minutes and checked again. It was still there. It stayed for some time but eventually disappeared.
Any help with this issue is greatly appreciated.