0

I want to train my SpamAssasin filter, and I've downloaded all SPAM tars from January 2015 to November 2015 from that site: untroubled.org/spam/

Also, I've downloaded my ham and SPAM emails from my personal gmail account. But my whole ham mails from gmail has about 2500 emails, while these 11 tars from untroubled.org contains about 410000 mails. So, the ham/spam relationship is about 1:160 and thus the SpamAssassin filter will be too much SPAM biased.

On the other hand, that kind of SPAM data set sites are thought mainly for SPAM researchers, not for system administrators.

So, my question is, what is what system administration usually do, or, what are the recommended practice? Do they use these kind of data sets? Is it that dangerous?

1

Considering that the untroubled.org SPAM collection method casts a wide net I don't see any issue with training Spamassassin with that data.

You should consider the scope of your email server and decide if you have a need to manually scan large volumes of known SPAM. SA does a pretty good job of filtering SPAM based on it's internal rules but if you have the time and the inclination to manually scan those messages it won't present any issue for your server.

  • and what about that huge ham/spam ratio? Of course I could get more ham mail from different places to reduce that ratio but I don't think I will be capable of reducing it to less than 1:80 or so. – Peregring-lk Dec 15 '15 at 16:08
  • Reading through the site you linked to I don't see anything about the ratio written there. Specifically it says that it's "all the spam that I have received since early 1998." – red_shift Dec 15 '15 at 22:21
  • The ratio between the amount of spam and ham mail that I have got in total. What I have done anyway is to take, randomly, a 5% of files downloaded from that site, to accomodate it to the amount of ham mail I'm going to use to train Ppamassasin, to reduce that ratio. If I get more ham mails, I'll take then another 5% and so on. – Peregring-lk Dec 15 '15 at 22:29
  • 1
    Understood, in my case I use aggressive HELO connection rules and also run sa-update on a regular basis and I haven't had any major issues. Your plan also makes sense to me but it will depend on your volume. – red_shift Dec 15 '15 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.