Are there any tools for comparing DNSBL effectiveness?

Question

I just added a new DNSBL to the list of ones that I use on my small mailserver. I set it up to be first on the list because I wanted to see what it missed but other lists caught.

I'm watching mail logs out of the corner of my eye while I do other things, just to make sure that things looked sane, and I started wondering about how to measure the relative effectiveness of various lists. I suppose I could give each list a week at the top of the heap and then measure how much the other lists catch that week, but that seems like a real hack.

Are there any tools or techniques for measuring the relative effectiveness of different blocklists on a specific server? I can measure false positives by user complaint volume, so my concern is addresses that one blocklist misses but another catches.

Answer 1

If all you want to do is determine "which is the minimal set of blacklists I should subscribe to in order to catch the maximum amount of spam", then what you really want to do is:

• Tally all of your blocked spam by IP address (so analyse your mail logs to get a table of blocked IPs, with the count of how many spams came from each, perhaps)
• Run all those IPs through all of your blacklists (trivial to do, just needs a quick DNS lookup for each IP)
• Score each blacklist against how many of the IPs (and how many spams came from each IP) each blacklist caught.

This is complicated by the time-varying nature of the contents of each of the blacklists, so perhaps having a logcheck-type process (running regularly, analysing new log entries) is the way to go, which can then get reported back to you as required.

All of this isn't all that tricky to rig up with a few lines of shell or $SCRIPTING_LANGUAGE_OF_CHOICE.

Of course, unless you're running so many blacklists that the time required to go through them all is excessive, the more interesting question is "how many spams didn't get caught by my blacklists, and what additional blacklists would catch those without contributing excessively to my false-positive rate." Again, something that looks up "passed" IP addresses in a set of blacklists you're considering and records the results would be instructive.

Answer 2

In addition to DNSBL you may wish to consider is GeoIP blocking. This is when you take a look at the IP address the connection is coming from, determine the originating country, and then block the connection.

We find that most of our spam comes from countries that we would NEVER expect to receive an email from (China, Brazil, Singapore, etc.)

By blocking these countries completely we have reduced our spam load a HUGE amount before any other sort of more CPU intensive checking is made.

I would estimate that at least 90% (probably more) of all of our spam is caught by GeoIP and less than 10% is actually processed against DNSBL, spamassassin, whitelist, etc.

We used MaxMind's free GeoIP database but I know there are others out there.

To answer your question directly though about DNSBL, we use the following:

sbl-xbl.spamhaus.org bl.spamcop.net dnsbl.cyberlogic.net dnsbl.ahbl.org

We haven't had to update/change from the above in quite some time. I would say that (for us) spamhaus.org tends to catch most stuff that GeoIP doesn't.

Your mileage may vary.

Answer 3

If you have spamassassin apply the blocklists, which it (effectivly) does in paralell you can use its logs to do those hit rate calculations.

Once you know what blocklists to trust you can reconfigure your mailservers to block on the most trusted RBL's.