I have a question. Whether the use of sa-learn to teach spamassassin about email spam and ham it matter? How if I do not use it because I do not have a sample of spam or ham in my mail server?


sa-learn is generally used for mail stored on the machine (in mbox or maildir format), and only works when you have filed spam and ham separately. If you are going to use it, it is best used with a decent amount of examples of both to prevent filter bias.

there is a nice doc here which goes through the process and details, but it does require locally stored mail (in either format).

having said this, I use SA on a few boxes and never use sa-learn, and it still does a pretty good job. I do use a number of other anti-spam techniques that aren't SA based, though...

  • 1
    Would be great if you mention which other technologies you use! – 3molo Apr 27 '11 at 8:27
  • indeed - RBL's, header syntax checks (e.g. invalid or incomplete server name or IP in HELO's) eliminate a large portion of bot generated spam, and the dreaded sender verify (the efficiency of which has come close to generating fist fights at the office) reduces accepted spam hugely. It also can effect valid mail (from badly configured servers), so you need to have some good whitelisting processes available to counter it - but the inconvenience is worth it. At most 4 or 5 addresses a month are whitelisted for close to 1500 users. – Mark Regensberg Apr 27 '11 at 9:00

I agree with Mark Regensberg's answer, but to be more specific: as I understand it, sa-learn and the other Bayesian elements of SpamAssassin only affect the Bayesian tests (you can see the current complete list of SA tests for clarification).

That is to say, all the rule-based tests function at full effectiveness regardless of whether you use sa-learn or not. Only the matching of the BAYES_nn rules are affected by your personalised training of the Bayesian engine with sa-learn --ham and --spam.

Having said that, some of those tests score quite highly - the rating of a message can be affected by an amount between -1.9 and +3.8, according to how "spammy" the Bayesian engine thinks it is - so I find quite a lot of value in giving my engine some training. As Mark notes, you will need to file your ham and undetected spam separately in order to do this.

In answer to your note to Mark, the "other" technique that has decreased my spam more than any other is greylisting, which by eliminating "fire-and-forget" mail reduced my incoming spam by well over 90%. Introducing SPF filtering on incoming email was the second most effective, cutting out about 5% of it.

  • I have had colleagues using greylisting with great success - I guess it's whatever recipe you get used to (and that works) first :) – Mark Regensberg Apr 27 '11 at 9:03

SpamAssassin has quite a few techniques for finding spam. One of them is its regular expressions (as noted by MadHatter's answer), but that isn't terribly potent these days. Another (also noted by MadHatter) is SPF, though I'd call that about negligible in its ability to catch otherwise-uncaught spam.

The most potent techniques in SpamAssassin are Bayesian detection and online lookups (DNSBLs (a.k.a. "RBLs") and URI DNSBLs, as well as hashing systems like Razor and Pyzor, see also the SA wiki pages for installing Razor and installing Pyzor).

Online lookups are by far the simplest; configure them right and you're good to go. They'll keep up to date with spam that hits the various spam trap networks (honeypots) out there, but they won't protect you from snowshoe (which is too fast) or targeted attacks like spearphishing (which is too small).

Bayesian detection requires constant maintenance; it is a machine learning system and therefore must be regularly trained on what it has missed (and on what it inappropriately caught). The more attention it gets, the better it gets.

SpamAssassin has an auto-learning system that will assume all very high scoring spam should be learned as such and all very low scoring ham should similarly be learned as ham. The problem is that it is only learning from what is easy, and (especially for ham) may learn from misclassified mail, which will reinforce SpamAssassin's mistakes.

No SpamAssassin deployment should ever trust auto-learning without additional manual training. It is meant to supplement manual training, not replace it. You must use sa-learn (or spamassassin --report, which also reports to Pyzor, Razor, and the SpamCop DNSBL).

You can learn more about Bayes on the SpamAssassin wiki.

  • An excellent post (+1 from me!), but I disagree about SPF. It is designed to prevent joe-jobbing, not to prevent spam, and those that try to use it for spam prevention, or think it should do that, are doing it and themselves no favours. If you run a mail server that insists that senders use real addresses, then SPF's ability to reject faked mail from fully-SPF-enabled domains will have the side-effect of spam reduction - but it's a (handy, beneficial) side effect, not the raison d'etre. I'm sorry I wasn't clearer about that in my answer. – MadHatter Jul 7 '15 at 7:03
  • SPF does not combat joe job attacks, which are targeted backscatter spam. They are sent to random relays with the intent to send bounces to the true victim. If those random relays are properly configured, they would not send bounce messages. Bounce messages are supposed to be sent only by the original outbound SMTP server, not the receiving relay. A misconfigured relay that does send bounces externally could lower the risk of participating in a joe job attack by using SPF, but is still misconfigured. – Adam Katz Jul 7 '15 at 16:27
  • Disclaimer: I'm a developer of SpamAssassin and am professionally involved in the anti-spam world. – Adam Katz Jul 7 '15 at 16:28
  • Then I respect your considerable knowledge, but the Wikipedia link you post doesn't seem to me to back you up. It says that a "joe job is a spamming technique that sends out unsolicited e-mails using spoofed sender data ... typically used by commercial spammers to conceal the true origin of their messages". It notes that backscatter is a related issue, but does not suggest to me that targetted backscatter is the primary purpose of joe-jobbing. – MadHatter Jul 7 '15 at 16:31
  • The sender data is spoofed so that it can be bounced (... or otherwise autoreplied to). I agree that the article isn't as well written as it could be. There are plenty of resources online that demonstrate that joe job attacks are, by definition, targeted backscatter spams. Here's the first hit in a search for backscatter spam "joe job". – Adam Katz Jul 7 '15 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.