First, this kind of attack is not (mainly) targeting DNS itself as your title suggests. It will of course create some additional load on DNS servers but the main purpose is to DDoS someone else. Bad server configuration might make it worse but in the end this issue is inherent in the design of DNS and UDP and, in fact, any stateless communication protocol.
It basically works like this: An attacker sends ordinary (DNS) queries to a (DNS) server. Those queries are forged to appear as if they were originating from the targets system. The DNS server now answers the query, sending the answer back to its alleged origin - the victim. This is why it's called reflection attack.
This is possible because you can verify the source of stateless communication (as DNS over UDP) as good as you can trust the sender address on a postcard. The server has just no way to decide if a query is legitimate or part of such an attack. DNS is just the most popular protocol here because there are lots and lots of servers for it around and you don't need much technical insight or special equipment to (mis)use it.
To make things worse (and at all attack-efficient), look at the amplification part. It would be not much of harm if the traffic of the attacker was equal in size to the resulting traffic. The only benefit for the attacker would be that his address gets hidden behind the DNS server. He could fake the sender address directly, there would totally be no need to re-route over DNS. But DNS answers, and that's another point why DNS is so popular here, can be a lot bigger than the question. You can find varying numbers on this depending on the exact queries used, but it can go up to 1:60 if the server is friendly enough to perform recursive queries for you. So the attacker does not need many machines under his control to produce lots of malicious traffic.
As you can easily find hundreds and thousands of "open" DNS server on the public internet, you can do the quick math how little work an attacker has to do if each open DNS server he knows will reflect his queries amplified sixtyfold to the target. As I said in the beginning, there is no really good way to countermeasure this. Naturally many DNS servers are open to everyone while they should not be, due to misconfiguration. But there are as many open server that have to be open, because exactly that's their purpose.
While you can't tell if a request is part of an attack or not your only option is to not run the server anymore. You can fiddle with rate limiting and other toys but you cannot get completely rid of this. If you are providing DNS for fun you can blacklist the source IP of the requests. But if you are on a larger scale this would damage the victim even more. Remember, all you can see on the DNS server is the address of the victim. Imagine your company is under attack through the DNS of your provider and your provider decides to cut DNS service for your company. The attacker could score this as a bazillion bonus points concerning denial-of-service.
Anyhow, those attacks happen all day and night and they are considered as "background noise" of the internet. If you set up a public (recursive) DNS server it won't take long before you are participating in random attacks. Of course sometimes things get real bad when large infrastructures (like even the dns root servers) are misused to amplify but in those cases proactive countermeasures are taken by personell until the attack goes down to "normal" levels.
So far on the teaching. To answer your question, at last:
You know your server is vulnerable if it answers queries without restriction. Period. If you are serving recursive queries, your server can generate the mentioned 1:60 ratio for the attacker. If it's serving only un-recursive it's not as bad, but still...
- make sure that you really need to run a public DNS server
- if you have to, take a look at BIND's
- if your DNS server will be authoritative for your own zone, there is no need for recursion at all, set
allow-recursion to "none;"
- if you want to run a resolver for other domains, restrict the allowed users for queries and recursive queries. You can define IP addresses, networks or access lists in the mentioned directives
- think about rate limiting DNS traffic not only in BIND but also on system level. As a very simple example, these iptables rules will not allow more than 10 queries per minute from each IP address:
iptables -A INPUT -p udp --dport 53 --set --name dnslimit
iptables -A INPUT -p udp --dport 53 -m recent --update --seconds 60 --hitcount 11 --name dnslimit -j DROP
Now, with these points in mind you should be good to go. There might still be malicious traffic on your server now and then but not in amounts that take your good night's sleep.