As Calle says, it's implementation specific. I'm going to get into a little more detail though.
The most common approach that nameservers use is a memory based approach. A memory ceiling is defined (either as a default or in the configuration by the user), and under normal operation data is cached until it expires. When the memory ceiling is reached, the software begins to discard less used records in order to free up memory for newer records, usually starting with the oldest data in the cache and possibly considering the least requested data as well.
Many nameserver implementations do not implement any sort of IP based query blacklisting. This is mostly due to how easy it is to spoof an IP addresses in UDP, which would make it far too easy to perform a DNS denial of service against a legitimate IP that needs the data. I could pretend to be your IP, hammer Google's nameservers until they ignore your IP, and then you can't get to Google.com. This would be bad, as without the Google Runbook most of the people at your company would lose the ability to pretend that they know how to do their job. Once you consider the fact that the attacker can simply change the IP address that they're spoofing if they're blocked, IP blacklisting becomes a very unattractive option for most server operators.
Due to the increased popularity of label/hostname cycling attacks directed towards recursive DNS servers (which I won't go into here), more and more software packages are starting to enable options for rate limiting incoming queries based on several criteria. The idea here is not to outright block IPs (because the IP is probably spoofed and it's easy to spoof a different one), but instead make it less effective and therefore unattractive to use the DNS server in these sorts of attacks.
Anyway, the short version is that you usually don't need to worry about someone getting your domain or even your IP "banned" from a recursive DNS server. It's certainly possible if the parties in question are not being leveraged in attacks very often and therefore don't know why it's usually a bad idea to ban domains or IPs, but uncommon. The experienced ones will only ban your domain (temporarily) if their platform is being leveraged to attack the nameservers for your domain, in which case they believe that they're doing you a favor by not sending you queries.