Testing promotions against spamassassin, scores lower than expected

Question

I recently setup a local installation of spam assassin for testing promotions as we want to catch any spam false positives before we mail our customers (This email is opt-in). We even added several (~30) additional rulesets that auto-update daily with a cron because we would rather be much stricter than average.

Here is my problem: Our SA install consistently ranks things much lower than other spam test we send mails to, even passing mail that our email provider marked as spam (that was embarrassing). It seems as if even the default rules are not working! We do recieve a score, and the worst spam I could find in my spam folder is marked as spam, but on average the score for real spam is only 2-4 (5 is the threshold).

I'm looking for advice from veteran SA users, more diagnostics I can run, and/or recommended rule sets. If this should be split into multiple questions I will happily do so.

System info: Freebsd 7.2 SpamAssassin Server version 3.2.5 running on Perl 5.8.9 Fresh SA install on a fresh OS install.

EDIT I verified that our basic rules are working, I'm looking mainly for things that will help me be stricter on mass marketing emails.

Thanks for any help, Chance

Answer 1

I am not sure that you will get a concise answer to this question... SpamAssassin can be configured by each mail admin differently, including adding new rules, changing the score for existing rules, changing the threshold for mail being marked as spam, etc. For example, we use a default score of 3.5 instead of the 5 that you are using. We also jack up the points for any emails that hit Pyzor, SORBS or any other blacklist database.
Concerning diagnostics you can run:
spamassassin --lint -D < email_message_to_test.txt
will give you more information than you ever cared to know about how spamassassin is grading your email message.
Regarding rulesets:
Here you can get instructions on making your own.
These guys write some extra rules that work well.

Since you are sending out SPAM (remember that SPAM is unsolicited email and promotions are no exception), it is very likely that after you start sending out emails that many of the SPAM databases are going to start flagging your emails as spammy. In fact, DCC and other auto-learning tools do exactly that. Your in-house test will score lower because your email is not in those databases yet, but after your email is in the wild, the scores will start to creep up as the databases catch up.

Answer 2

"We are consistently scoring things lower than we should. Is it standard practice to tweak the rule weights?"

Yes, we've tweaked ours daily at the beginning of SA usage. I kept header information to get statistics for the default scoring which was too low in my opinion.

Finding averages of most often used header scores and raising them slightly continuously has proven to work well in our implementation of SA.

I also agree with Scott about lowering the score and including Pyzor and SPF functionality.

Answer 3

In X-Spam-Report header it's clearly stated how much each check earned.

X-Spam-Status: Yes, score=7.5 required=5.0 tests=BAYES_99,HTML_MESSAGE,
        RCVD_IN_PBL,RCVD_IN_XBL,RDNS_NONE autolearn=no version=3.2.4
X-Spam-Report: 
        *  3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
        *      [score: 1.0000]
        *  0.0 HTML_MESSAGE BODY: HTML included in message
        *  3.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
        *      [119.152.74.233 listed in zen.spamhaus.org]
        *  0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL
        *  0.1 RDNS_NONE Delivered to trusted network by a host with no rDNS