Fighting Spam - What can I do as an: Email Administrator, Domain Owner, or User?

Question

This is a Canonical Question about Fighting Spam.

Also related: How to stop people from using my domain to send spam?

There are so many techniques and so much to know about fighting SPAM. What widely used techniques and technologies are available to Administrator, Domain Owners, and End Users to help keep the junk out of our inboxes?

We're looking for an answer that covers different tech from various angles. The accepted answer should include a variety of technologies (eg SPF/SenderID, DomainKeys/DKIM, Graylisting, DNS RBLs, Reputation Services, Filtering Software [SpamAssassin, etc]); best practices (eg mail on Port 25 should never be allowed to relay, Port 587 should be used; etc), terminology (eg, Open Relay, Backscatter, MSA/MTA/MUA, Spam/Ham), and possibly other techniques.

Answer 1

To defeat your enemy, you must know your enemy.

What is spam?

For our purposes, spam is any unsolicited commercial electronic message. Spam these days is intended to lure unsuspecting users into visiting a (usually shady) web site where they will be asked to buy products, or have malware delivered to their computers, or both. Some spam will deliver malware directly.

It may surprise you to learn that the first spam was sent in 1864. It was an advertisement for dental services, sent via Western Union telegram. The word itself is a reference to a scene in Monty Python's Flying Circus.

Spam, in this case, does not refer to mailing list traffic a user subscribed to, even if they changed their minds later (or forgot about it) but have not actually unsubscribed yet.

Why is spam a problem?

Spam is a problem because it works for the spammers. Spam typically generates more than enough sales (or malware delivery, or both) to cover the costs -- to the spammer -- of sending it. The spammer does not consider the costs to the recipient, you and your users. Even when a tiny minority of users receiving spam respond to it, it's enough.

So you get to pay the bills for bandwidth, servers, and administrator time to deal with incoming spam.

How does spam work?

Spam typically is delivered in different ways from normal, legitimate email.

Spammers almost always want to obscure the origin of the email, so a typical spam will contain fake header information. The From: address is usually fake. Some spam includes fake Received: lines in an attempt to disguise the trail. A lot of spam is delivered via open SMTP relays, open proxy servers and botnets. All of these methods make it more difficult to determine who originated the spam.

Once in the user's inbox, the purpose of the spam is to entice the user to visit the advertised web site. There, the user will be enticed to make a purchase, or the site will attempt to install malware on the user's computer, or both. Or, the spam will ask the user to open an attachment which contains malware.

How do I stop spam?

As a system administrator of a mail server, you will configure your mail server and domain to make it more difficult for spammers to deliver their spam to your users.

I will be covering issues specifically focused on spam and may skip over things not directly related to spam (such as encryption).

Don't run an open relay

The big mail server sin is to run an open relay, a SMTP server which will accept mail for any destination and deliver it onward. Spammers love open relays because they virtually guarantee delivery. They take on the load of delivering messages (and retrying!) while the spammer does something else. They make spamming cheap.

Open relays also contribute to the problem of backscatter. These are messages which were accepted by the relay but then found to be undeliverable. The open relay will then send a bounce message to the From: address which contains a copy of the spam.

• Configure your mail server to accept incoming mail on port 25 only for your own domain(s).
• Test your system by sending your SMTP server a mail from outside your network where both the From: and To: addresses are not within your domain. The message should be rejected. (Or, use an online service like MX Toolbox to perform the test.)

Reject anything that looks too suspicious

Various misconfigurations and errors can be a tip-off that an incoming message is likely to be spam or otherwise illegitimate.

• Mark as spam or reject messages for which the IP address has no reverse DNS (PTR record). Treat the lack of a PTR record more harshly for IPv4 connections than for IPv6 connections, as many IPv6 addresses do not yet have reverse DNS.
• Reject messages for which the domain name in the sender or recipient addresses does not exist.
• Reject messages which do not use fully qualified domain names for the sender or recipient domains, unless they originate within your domain and are meant to be delivered within your domain (e.g. monitoring services).
• Reject connections where the other end does not send a HELO/EHLO.
• Reject connections where the HELO/EHLO is:
  • not a fully qualified domain name and not an IP address
  • blatantly wrong (e.g. your own IP address space)
• Reject connections which use pipelining without being authorized to do so.

Authenticate your users

Accept outgoing mail from your users only on port 587 (the submission port; RFC 2476) and never on port 25, then block port 25 in your egress filtering (except from your own mail server, of course). This does two things for you: It separates mail originating from your domain from internet mail, and requires users to authenticate to the mail server to originate a message. This also helps to prevent compromised computers within your domain from sending spam.

• Configure your outgoing SMTP server to accept authenticated connections on port 587.
• Configure your outgoing SMTP server to reject connections on port 25. (Skip this if your incoming and outgoing SMTP servers are the same server.)
• Configure your edge firewall to reject outgoing connections to destination port 25 except when they originate from your outgoing SMTP server.
• Configure your SMTP server to correct invalid From: addresses in your users' outgoing mail. (Your business needs may prevent this, though.)
• Test sending authenticated email via port 587 and via port 25. Only the first one should reach its destination.

Consider using SpamAssassin

SpamAssassin is a mail filter which can be used to identify spam based on the message headers and content. It uses a rules-based scoring system to determine the likelihood that a message is spam. The higher the score, the more likely the message is spam.

SpamAssassin also has a Bayesian engine which can analyze spam and ham (legitimate email) samples fed back into it.

Best practice for SpamAssassin is not to reject the mail, but to put it in a Junk or Spam folder. MUAs (mail user agents) such as Outlook and Thunderbird can be set up to recognize the headers that SpamAssassin adds to email messages and to file them appropriately. False positives can and do happen, and while they're rare, when it happens to the CEO, you will hear about it. That conversation will go much better if the message was simply delivered to the Junk folder rather than rejected outright.

SpamAssassin is almost one-of-a-kind, though a few alternatives exist.

• Install SpamAssassin and configure automatic update for its rules using sa-update.
• Consider using custom rules where appropriate.
• Consider setting up Bayesian filtering.

Consider using DNS-based blackhole lists and reputation services

DNSBLs (formerly known as RBLs, or realtime blackhole lists) provide lists of IP addresses associated with spam or other malicious activity. These are run by independent third parties based on their own criteria, so research carefully whether the listing and delisting criteria used by a DNSBL is compatible with your organization's need to receive email. For instance, a few DNSBLs have draconian delisting policies which make it very difficult for someone who was accidentally listed to be removed. Others automatically delist after the IP address has not sent spam for a period of time, which is safer. Most DNSBLs are free to use.

Reputation services are similar, but claim to provide better results by analyzing more data relevant to any given IP address. Most reputation services require a subscription payment or hardware purchase or both.

There are dozens of DNSBLs and reputation services available, though some of the better known and useful ones I use and recommend are:

Conservative lists:

• Spamhaus
• Barracuda Reputation Database (no purchase necessary)
• Composite Blocking List
• SpamCop

Aggressive lists:

• UCEPROTECT
• Backscatterer
• RFC Ignorant

As mentioned before, many dozens of others are available and may suit your needs. One of my favorite tricks is to look up the IP address which delivered a spam that got through against multiple DNSBLs to see which of them would have rejected it.

• For each DNSBL and reputation service, examine its policies for listing and delisting of IP addresses and determine whether these are compatible with your organization's needs.
• Add the DNSBL to your SMTP server when you have decided it is appropriate to use that service.
• Consider assigning each DNSBL a score and configuring it into SpamAssassin rather than your SMTP server. This reduces the impact of a false positive; such a message would be delivered (possibly to Junk/Spam) instead of bounced. The tradeoff is that you will deliver a lot of spam.
• Or, reject outright when the IP address is on one of the more conservative lists, and configure the more aggressive lists in SpamAssassin.

Use SPF

SPF (Sender Policy Framework; RFC 4408 and RFC 6652) is a means to prevent email address spoofing by declaring which Internet hosts are authorized to deliver mail for a given domain name.

• Configure your DNS to declare an SPF record with your authorized outgoing mail servers and -all to reject all others.
• Configure your mail server to check the SPF records of incoming mail, if they exist.

Investigate DKIM

DKIM (DomainKeys Identified Mail; RFC 6376) is a method of embedding digital signatures in mail messages which can be verified using public keys published in the DNS. It is patent-encumbered in the US, which has slowed its adoption. DKIM signatures can also break if a message is modified in transit (e.g. SMTP servers occasionally may repack MIME messages).

• Consider signing your outgoing mail with DKIM signatures, but be aware that the signatures may not always verify correctly even on legitimate mail.

Consider using greylisting

Greylisting is a technique where the SMTP server issues a temporary rejection for an incoming message, rather than a permanent rejection. When the delivery is retried in a few minutes or hours, the SMTP server will then accept the message.

Greylisting can stop some spam software which is not robust enough to differentiate between temporary and permanent rejections, but does not help with spam that was sent to an open relay or with more robust spam software. It also introduces delivery delays which users may not always tolerate.

• Consider using greylisting only in extreme cases, since it is highly disruptive to legitimate email traffic.

Consider using nolisting

Nolisting is a method of configuring your MX records such that the highest priority (lowest preference number) record does not have a running SMTP server. This relies on the fact that a lot of spam software will only try the first MX record, while legitimate SMTP servers try all MX records in ascending order of preference. Some spam software also attempts to send directly to the lowest priority (highest preference number) MX record in violation of RFC 5321, so that could also be set to an IP address without an SMTP server. This is reported to be safe, though as with anything, you should test carefully first.

• Consider setting your highest-priority MX record to point to a host which does not answer on port 25.
• Consider setting your lowest-priority MX record to point to a host which does not answer on port 25.

Consider a spam filtering appliance

Place a spam filtering appliance such as Cisco IronPort or Barracuda Spam & Virus Firewall (or other similar appliances) in front of your existing SMTP server to take much of the work out of reducing the spam you receive. These appliances are pre-configured with DNSBLs, reputation services, Bayesian filters and the other features I've covered, and are updated regularly by their manufacturers.

• Research spam filtering appliance hardware and subscription costs.

Consider hosted email services

If it's all too much for you (or your overworked IT staff) you can always have a third party service provider handle your email for you. Services such as Google's Postini, Symantec MessageLabs Email Security (or others) will filter messages for you. Some of these services can also handle regulatory and legal requirements.

• Research hosted email service subscription costs.

I'm a regular user, not a sysadmin. What can I do to fight spam?

As a user, the absolute #1 thing you should do to fight spam is:

• DO NOT RESPOND TO THE SPAM.

  If it looks funny, don't click the website link and don't open the attachment. No matter how attractive the offer seems. That viagra isn't that cheap, you aren't really going to get naked pictures of anybody, and there is no $15 million dollars in Nigeria or elsewhere except for the money taken from people who did respond to the spam.

• If you see a spam message, mark it as Junk or Spam depending on your mail client.

• DO NOT mark a message as Junk/Spam if you actually signed up to receive the messages and just want to stop receiving them. Instead, unsubscribe from the mailing list using the unsubscribe method provided.

• Check your Junk/Spam folder regularly to see if any legitimate messages got through. Mark these as Not Junk/Not Spam and add the sender to your contacts to prevent their messages from being marked as spam in the future.

Answer 2

I've managed over 100 separate mail environments over the years and have used numerous processes to reduce or help eliminate spam.

Technology has evolved over time, so this answer will walk through some of the things I've tried in the past and detail the current state of affairs.

A few thoughts about protection...

• You want to protect port 25 of your incoming mail server from being an open relay, where anyone can send mail through your infrastructure. This is independent of the particular mail server technology you may be using. Remote users should use an alternate submission port and some form of required authentication for relaying mail. Port 587 or port 465 are the common alternatives to 25.
• Encryption is also a plus. A lot of mail traffic is sent in cleartext. We're at the point now where most mail systems can support some form of encryption; some event expect it.
• These are more proactive approaches to preventing your mail site from being classified as a spam source...

With regard to incoming spam...

• Greylisting was an interesting approach for a short period of time. Force a temporary reject/delay in hopes that a spammer would disconnect and avoid exposure or the time and resources needed to requeue messages. This had the effect of unpredictable delays in mail delivery, didn't work well with mail from large server farms and spammers eventually developed workarounds. The worst impact was breaking the user expectation of speedy mail delivery.
• Multiple MX relays still need protection. Some spammers would try sending to a backup or lower-priority MX in hopes that it had less robust filtering.
• Realtime Black(hole) Lists (RBL/DNSBL) - These reference centrally-maintained databases to verify whether a sending server is listed. Heavy reliance on RBL's comes with caveats. Some were not as reputable as others. The offerings from Spamhaus have always been good for me. Others, like SORBS, have a poor approach to listing IP's, and often block legitimate email. It's been likened to an extortion plot in some cases, because delisting often involves $$$.
• Sender Policy Framework (SPF) - Basically a means of ensuring that a given host is authorized to send mail for a particular domain, as defined by a DNS TXT record. It's good practice to create SPF records for your outgoing mail, but bad practice to require it from the servers sending to you.
• Domain Keys - Not in widespread use... yet.
• Bounce suppression - Prevent invalid mail from being returned to its source. Some spammers would try to see which addresses were live/valid by analyzing backscatter to create a map of usable addresses.
• Reverse DNS/PTR checks - Check that a sending server has a valid reverse PTR record. This doesn't not need to match the originating domain, as it's possible to have a many-to-one mapping of domains to a host. But it's good to determine ownership of an IP space and to determine whether the originating server is part of a dynamic IP block (e.g. home broadband - read: compromised spambots).
• Content filtering - (unreliable) - Trying to counter permutations of "(Viagra, v\|agra, viagra, vilgra.)" is time-consuming for the administrator and doesn't scale in a larger environment.
• Bayesian filtering - More advance spam solutions allow global or per-user training of mail. Read the linked article on the heuristics, but the main point is that mail can be manually classified as good (Ham) or bad (Spam), and the resulting messages populate a Bayesian database that can be referenced to determine the categorization of future messages. Typically, this is associated with a spam score or weighting, and can be one in a handful of techniques used to determine whether a message should be delivered.
• Rate controlling/throttling - Simple approach. Limit how many messages a given server can attempt to deliver within a certain period of time. Defer all messages over that threshold. This is usually configured on the mail server side.
• Hosted and cloud filtering. Postini comes to mind, as that was a cloud solution before cloud was a buzzword. Now owned by Google, the strength of a hosted solution is that there are economies of scale inherent to processing the volume of mail that they encounter. Data analysis and simple geographic reach can help a hosted spam filtering solution adapt to trends. The execution is simple, though. 1). Point your MX record to the hosted solution, 2). provide a post-filtering server delivery address. 3). Profit.

---

My current approach:

I'm a strong advocate of appliance-based spam solutions. I want to reject at the perimeter of the network and save the CPU cycles at the mail server level. Using an appliance also provides some independence from the actual mail server (mail delivery agent) solution.

I recommend Barracuda Spam Filter appliances for a number of reasons. I've deployed several dozen units, and the web-driven interface, industry mindshare and set-and-forget appliance nature make it a winner. The backend technology incorporates many of the techniques listed above.

• I block port 25 on my mail server's IP address and instead set the MX record for the domain to the Barracuda appliance's public-facing address - e.g. spam.domain.com. Port 25 will be open for mail delivery.
• The core is SpamAssassin-derived with a simple interface to a message log (and Bayesian database) that can be used to classify good mail from bad during an initial training period.
• Barracuda leverages several RBL's by default, including those by Spamhaus.org, and their own BRBL reputation database. Note - the BRBL is usable for free as a standard RBL for other mail systems.
• The Barracuda reputation database is compiled from live data, honeypots, large-scale analysis and any number of proprietary techniques. It has a registered whitelist and blocklist. High-volume and high-visibility mail senders often register with Barracuda for automatic whitelisting. Examples include Blackberry, Constant Contact, etc.
• SPF checks can be enabled (I don't enable them, though).
• There's an interface to review mail and redeliver from the appliance's mail cache as necessary. This is helpful in instances where a user was expecting a message that may not have passed all of the spam checks.
• LDAP/Active Directory user verification helps accelerate the detection of invalid mail recipients. This saves bandwidth and prevents backscatter.
• IP/sender address/domain/country-of-origin can all be configured. If I want to deny all mail from Italian domain suffixes, it's possible. If I want to prevent mail from a particular domain, it's easily configured. If I want to block a user's stalker from sending email to the user, it's doable (true story).
• Barracuda provides a number of canned-reports and a good visual display of the appliance status and spam metrics.
• I like having an appliance onsite to keep this processing in-house and possibly have a post-filter email journaling connection (in environments where mail retention is necessary).
• Plus The appliance can reside in a virtualized infrastructure.

Barracuda Spam & Virus Firewall 300 status console

---

Newer approach:

I've been experimenting with Barracuda's Cloud-based Email Security Service over the past month. This is similar to other hosted solutions, but is well-suited to smaller sites, where an expensive appliance is cost-prohibitive. For a nominal yearly fee, this service provides about 85% of what the hardware appliance does. The service can also be run in tandem with an onsite appliance to reduce incoming bandwidth and provide another layer of security. It's also a nice buffer that can spool mail in the event of a server outage. The analytics are still useful, although, not as detailed as a physical unit's.

Barracuda Cloud Email Security console

All in all, I've tried many solutions, but given the scale of certain environments, and the increasing demands of the user base, I want the most elegant solution(s) available. Taking the multi-pronged approach and "rolling your own" is certainly possible, but I've done well with some basic security and good use monitoring of the Barracuda device. Users are very happy with the result.

Note: Cisco Ironport is great as well... Just costlier.

Answer 3

Partly, I endorse what others have said; partly, I don't.

Spamassassin

This works very well for me, but you need to spend some time training the Bayesian filter with both ham and spam.

Greylisting

ewwhite may feel its day has come and gone, but I can't agree. One of my clients asked how effective my various filters were, so here are approximate stats for July 2012 for my personal mailserver:

• 46000 messages attempted delivery
• 1750 got through greylisting
• 250 got through greylisting + trained spamassassin

So about 44000 never made it through the greylisting; if I'd not had greylisting, and had accepted all those, they'd have all needed spam filtering, all using CPU and memory, and indeed bandwidth.

SPF

This isn't really an anti-spam technique, but it can reduce the amount of backscatter you have to deal with, if you're joe-jobbed. You should use it both in and out, that is: You should check the SPF record of the sender for incoming email, and accept/reject accordingly. You should also publish your own SPF records, listing fully all machines that are approved to send mail as you, and lock out all others with -all. SPF records that don't end in -all are completely useless.

Blackhole lists

RBLs are problematic, since one can get onto them through no fault of one's own, and they can be hard to get off. Nevertheless, they have a legitimate use in spam-fighting but I would strongly suggest that no RBL should ever be used as a bright-line test for mail acceptance. The way spamassassin handles RBLs - by using many, each of which contributes towards a total score, and it's this score that makes the accept/reject decision - is much better.

Dropbox

I don't mean the commercial service, I mean that my mail server has one address which cuts through all my greylisting and spam-filtering, but which instead of delivering to anyone's INBOX, it goes to a world-writeable folder in /var, which is automatically pruned nightly of any emails over 14 days old.

I encourage all users to take advantage of it when eg filling out email forms that require a validatable email address, where you're going to receive one email that you need to keep, but from whom you never wish to hear again, or when buying from online vendors who will likely sell and/or spam their address (particularly those outside the reach of European privacy laws). Instead of giving her real address, a user can give the dropbox address, and look in the dropbox only when she expects something from a correspondent (usually a machine). When it arrives, she can pick it out and save it in her proper mail collection. No user need look in the dropbox at any other time.

Answer 4

I am using a number of techniques which reduce spam to acceptable levels.

Delay accepting connections from incorrectly configured servers. A majority of the Spam I receive is from Spambots running on malware infected system. Almost all of these do not pass rDNS validation. Delaying for 30 seconds or so before each response causes most Spambots to give up before they have delivered their message. Applying this only to servers which fail rDNS avoids penalizing properly configured servers. Some incorrectly configured legitimate bulk or automated senders get penalized, but do deliver with minimal delay.

Configuring SPF for all your domains protects your domains. Most sub-domains should not be used to send email. The main exception is MX domains which must be able to send mail on their own. A number of legitimate senders delegate bulk and automated mail to servers that are not permitted by their policy. Deferring rather than rejecting based on SPF allow them to fix their SPF configuration, or you to whitelist them.

Requiring a FQDN (Fully Qualified Domain Name) in the HELO/EHLO command. Spam often uses an unqualified hostname, address literals, ip addresses, or invalid TLD (Top Level Domain). Unfortunately some legitimate senders use invalid TLDs so it may be more appropriate to defer in this case. This can require monitoring and whitelisting to enable the mail through.

DKIM helps with non-repudiation, but is otherwise not highly useful. My experience is that Spam is not likely to be signed. Ham is more likely to be signed so it has some value in Spam scoring. A number of legitimate senders don't publish their public keys, or otherwise improperly configure their system.

Greylisting is helpful for servers which show some signs of misconfiguration. Servers that are properly configured will get through eventually, so I tend to exclude them from greylisting. It is useful to greylist freemailers as they do tend to be used occasionally for Spam. The delay gives some of the Spam filter inputs time to catch the Spammer. It also tends to deflect Spambots as they usually don't retry.

Blacklists and Whitelists can help as well.

• I have found Spamhaus to be a reliable blacklist.
• Auto WhiteListing in the Spam filter helps smooth out the rating of frequent senders that are occasionally Spamish, or Spammers who are occasionally Hamish.
• I find dnsl.org's whitelist useful as well.

Spam filtering software is reasonably good at finding Spam although some will get through. It can be tricky getting the false negative to a reasonable level without increasing the false positive too much. I find Spamassassin catches most of the Spam that reaches it. I've added a few custom rules, that fit my needs.

Postmasters should configure the required abuse and postmaster addresses. Acknowledge the feedback you get to these addresses and act on it. This allows other to help yous ensure your server is properly configured and not originating Spam.

If you are a developer, use the existing email services rather than setting up your own server. It is my experience that servers setup for automated mail senders are likely to be incorrectly configured. Review the RFCs and send properly formatted email from a legitimate address in your domain.

End users can do a number of things to help reduce Spam:

• Don't open it. Flag it as Spam or Delete it.
• Ensure your system is secure and malware free.
• Monitor your network usage, especially when you aren't using your system. If it generates a lot of network traffic when you aren't using it, it may be sending spam.
• Turn off your computer when you aren't using it. (It won't be able to generate Spam if its turned off.)

Domain owners / ISPs can help by limiting Internet access on port 25 (SMTP) to official e-mail servers. This will limit the ability of Spambots to send to the Internet. It also helps when dynamic addresses return names which do not pass rDNS validation. Even better is to verify the PTR record for mail servers do pass rDNS valiation. (Verify for typographical errors when configuring PTR records for your clients.)

I have started classifying email in three categories:

• Ham (almost always from properly configured servers, properly formatted, and commonly personal e-mail.)
• Spam (Mostly from Spambots, but a certain percentage is from freemailers or other senders with properly configured servers.)
• [Mystery] Meat (Ham or Spam; Includes a lot of mail from mailing lists and automated systems. Ham usually end up here because of DNS and/or server misconfiguration.)

Answer 5

The SINGLE most effective solution I have seen is to use one of the external mail filtering services.

I have experience with the following services at current clients. I am sure there are others. Each of these has done an excellent job in my experience. The cost is reasonable for all three.

• Postini from Google
• MXLogic from McAfee
• SecureTide from AppRiver

The services have several huge advantages over local solutions.

1. They stop most (>99%) of the spam BEFORE it hits your internet connection and your email server. Given the volume of spam, this is a lot of data not on your bandwidth and not on your server. I have implemented one of these services a dozen times and every one resulted in a noticeable performance improvement to the email server.

2. They also do anti-virus filtering, typically both directions. This mitigates the need to have a "mail anti-virus" solution on your server, and also keeps the virii completely

They also do a great job at blocking spam. In 2 years working at a company using MXLogic, I have never has a false positive, and can count the legit spam messages that got through on one hand.