40

I operate a small mail server for my private emails, some friends who have websites and two NGOs. In total my server sends between 60 and 400 messages a day. Now a lot of these emails are personal mails, between two or more people who know each other. Occasionally (usually once or twice a week) there will be a mailing that goes out to "members" of one NGO, informing them what's new etc.

Now I have already moved off the "mass mailings" (about 100 recipients, all personally known and manually subscribed through a paper form) to mailgun.org.

I still get (and increasingly so), rejected messages. Especially big email providers like Gmail, Yahoo or Microsoft (hotmail, live.com, ...) just decide to reject with a 550 or send personal messages to the Spam folder of the recipients. Sometimes this happens:

  • gmail user sends email to user on my system
  • user on my system replies
  • the reply is being rejected or sent to spam

Things I have done:

  • set up DKIM (per-domain signing of all outgoing email)
  • set up SPF, domains usually have ~all, some -all
  • I have a correct PTR for my mail server IP
  • obviously no open relay, users can only send from their own email address after authentication
  • I have DMARC policies for most of the domains
  • I rate limit outgoing messages, for some mail servers down to 1 per minute
  • mail test services report "perfect" scores (all pass) for all of the above
  • I regularily check my IP for blacklisting using http://www.dnsbl.info - it's always all green

Now the paradox comes here: for most of the big mail providers, there is a way to register to monitor rejection rates and IP reputation:

but I do not classify as bulk sender, because of the low volume. So I did register to monitor my reputation and rejection rates, but because I do not send bulk email, there are no reports.

Is there anything else I can do to improve mail delivery rates? Or should I give in and stop trying to operate my own mail server?

In case it is relevant: I use postfix and have very strict rules about incoming mail (i.e. no unknown domains/host names or invalid SPF records, I use spamassassin etc.)

Update

Here is an example, sent from me to my in-laws and it arrived in their SPAM folder: http://pastebin.com/BC6YgjpQ (I replaced the sending address domain with example.com and the receivers address with example@gmail.com)

Since the question came up: Connections to Gmail are Untrusted TLS connection established to gmail-smtp-in.l.google.com[2a00:1450:400c:c0b::1b]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits) encrypted.

  • 3
    I gave up quite a while back. Now I send even my personal email through SendGrid. – Michael Hampton Apr 21 '16 at 19:25
  • 2
    I had similar problems, but I have even fewer mail traffic. In my case I added an SPF DNS record, and google allowed my mails again. For me this topic is very important. Running an own mail server is for me a fundamental human right. Maybe you can talk to the EFF they handle big topics like this. – guettli Apr 21 '16 at 19:45
  • 1
    Possible duplicate of How to send emails and avoid them being classified as spam? – MadHatter Apr 22 '16 at 7:09
  • I noticed you added an authentication header to the outgoing message. Google may treat this as an attempt to spoof authentication. My reading of the authentication header is that it is intended to be added by MX (border servers) to incoming messages. It is permitted/recommended that the MX remove existing Authentication headers. – BillThor Sep 30 '16 at 17:01
20

There should be no issues becoming a small mail provider. You seem to be doing the right things. Many large providers don't get things right, and hopefully get most of their mail delivered.

If mail is being sent to the SPAM folder, it is likely you have missed something. There should be a record of why you have delivery issues:

  • For bounced messages read the response. It should specify why the mail was bounced. If you can, make sure bounce messages are logged.
  • For messages that are sent to the Spam folder, examine the message headers on the delivered message. This should (will for GMail or Yahoo) contain details of at some of the checks that were done. This help you determine what the issue is.

A few things you did not specify although some should be caught by the validation report:

  • rDNS validation of your mail servers address succeeds. (Your PTR record should return only one address.)
  • Your server used the name on the PTR record in its EHLO or HELO message.
  • Setup an SPF record for your mail server's domain ("v=spf1 a -all").
  • You have registered with dnswl.org.
  • You have had the DKIM public key(s) published in the correct location. You can use the same key for multiple domains. It may help to have other organizations use CNAME records to DNS records you control.
  • You have used a large DKIM key 1024 or larger.
  • Process outgoing mail through a spam filter (at least log issues).

If you have DMARC you can configure delivery status reports and bounce reports. This will allow you to receive delivery reports. I receive reports from Google, Microsoft and Yahoo. Please note disposition "none" indicates the mail was delivered.

  • 2
    Here's a 550: host aspmx.l.google.com[2a00:1450:4013:c01::1a] said: 550-5.7.1 [2a02:c200:0:10:3:0:4018:cafe 18] Our system has detected that this message is likely suspicious due to the very low reputation of the sending IP address. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 for more information. n123si21866589wmb.41 - gsmtp (in reply to end of DATA command) That doesn't tell me anything. – Stefan Seidel Apr 22 '16 at 7:08
  • Who is the owner of the ip? Is it a residential or a business provider? Would it be an option to relay your outgoing smtp traffic through your internet provider's smtp servers? So you still would have control of the incoming traffic (I know, not exactly what you want, but still, having to explain every time to your in laws why your messages get flagged as spam or rejected could get awkward soon-ish ;-) ) – natxo asenjo Apr 22 '16 at 7:42
  • @StefanSeidel They have provided you with a link to a site where you can get trusted. You will need to as a CNAME or TXT record to each signing domain. It is a fairly quick and painless process. – BillThor Apr 22 '16 at 11:46
  • 4
    @BillThor are you talking about the registration for postmaster.google.com? That has nothing to do with "getting trusted". It allows you to view the spam reporting rates for verified domains. I have done that and it leads exactly to the situation I describe above: since the sending volume is so low, there is no data. I have verified my 4 top domains and none show any data. – Stefan Seidel Apr 23 '16 at 17:07
  • @StefanSeidel have you also registered the domain name of your mailhost? – BillThor Apr 24 '16 at 2:33
10

One thing missing in the above (excellent) replies is to set up outbound TLS. Gmail has started to punish senders not using TLS, and other providers aren't saying anything but I'm sure they will follow suit.

  • Thanks. Postfix prefers TLS by default, and I can see that is being used. – Stefan Seidel Apr 22 '16 at 6:57
  • 1
    Are you using a self-signed certificate? I don't know this for sure, but I rather suspect that using a third-party-verified certificate improves matters further, and they're getting awfully cheap. – MadHatter Apr 22 '16 at 7:08
  • Nope, StartCom Level 2 validated. Also, this isn't relevant for outgoing mail, right? – Stefan Seidel Apr 22 '16 at 7:14
  • @StefanSeidel why would it not apply for outgoing email? TLS certificate validation applies to all TLS transactions, and outgoing email from a modern MTA, to a server which advertises STARTTLS, will be sent under TLS. – MadHatter May 10 '16 at 5:19
  • 1
    @StefanSeidel You are wrong. Here is a very typical line from my mail server's log in TLS server mode (ie, receiving an incoming connection from another server) showing the validation of the remote server's certificate (verify=OK): May 10 08:01:34 lory sendmail[5884]: STARTTLS=server, relay=mail-bn1bhn0245.outbound.protection.outlook.com [157.56.111.245], version=TLSv1/SSLv3, verify=OK, cipher=AES256-SHA256, bits=256/256 – MadHatter May 10 '16 at 7:08
4

Nowadays, the spam activities are a real headache. The Big guys like Gmail, Microsoft, Yahoo etc. trying to secure their users from the spams. Hence, they must improve their techniques to filter the spams. And due to the security reason, they never disclose their Spam policies as well. Hence, we could not find a guideline to configure a mail server in such way that we can send mails to the big service providers.

There are no specific rules to be not listed in their bad book, but you should keep your server updated with the new guidelines. Here are some of them.

1) Check the root cause of the bounced back mail. Does it relate to the server IP reputation OR domain's incorrect DNS records.

2) Do not use an SPF record with a default value like ~all. Create a specific SPF record like a MX -all

3) Avoid mail forwarding from your server to Gmail/Yahoo/Microsoft/Comcast. If they detect any spam mail in your forwarded mails, they will not bother to check from where the mail is originated. They simply consider your mail server as a spam origin and you might be added to the blacklist.

4) Install an SSL on your server and use Outbound with TLS connection.

5) Keep Double Opt In list in all the newsletters. And many more...

2
Untrusted TLS connection established to gmail-smtp-in.l.google.com

I think you have a certificate chain error here. Make sure you are sending the intermediate certificate along with your certificate.

  • 1
    That just means that my server doesn't trust the gmail SMTP. I have added smtp_tls_CApath to my postfix conf, so that is a Trusted TLS connection now. But since that verification is only local to my mail server, I think it can not be relevant to any scoring. – Stefan Seidel Apr 27 '16 at 15:09
1

@Stefan

You seem very knowledgeable which is awesome, I looked over your headers, without your domain name it makes it very hard to help troubleshoot. The one thing I did notice in your headers is that you're using "Simple/Simple" to sign your DKIM, you should really switch that over to "Relaxed/Relaxed". A lot of mail servers have trouble with simple.

You also left the mail server name in your pastebin, which does show up on a blacklist. I doubt this is causing your issue, but this tool does scan a lot more then the one you were using.

Send an email to mailtest@unlocktheinbox.com to see how many criticals you have. You might get a few clues.

  • That site is a bit difficult to read for my eyes. I only see green rectangles. I did send this email, and allegedly there are 5 warnings and 3 criticals, but all I can see in between the Email Authentication Pro Feature - Learn More links is "pass" and "success". – Stefan Seidel Apr 27 '16 at 15:11
  • Yeah, you must of rolled off that one blacklist. Most public blacklists remove you if they don't detect spam for a certain period of time, usually a day or two. – Henry Apr 27 '16 at 15:22
1

Answer is no.

I state this mostly because I'm aware that there are hard to imagine number of people with more experience in all aspects of being ESP admin than me and yet I run dozen of production email systems that qualifies as "small" without presented problems.

From what you have posted it looks like you have taught of everything and, presumed you have properly implemented what is listed, there is only one option left - that your IP address has done bad things in it's past life.

I mean (really) bad. Your IP address may be pulled from public blacklists at the ISP's initiative (which is vastly more efficient) but before that served virus and phishing traffic repeatedly for a prolonged period of time.

If that is the case, unfortunately there is little that can be done, to my best knowledge. One option is to remain in attempt to maintain legitimate email server for a long time, paying the price of many emails being rejected before sender reputation - completely different thing to being present or not on public blacklists - is slowly established.

Please keep us informed about your case.

  • Thanks, and I think, yes the big deal after all the other stuff may be IP reputation. I recently added a second mail server, and I got some rejections to Microsoft addresses. Turns out my hosting provider put me in a "bad neighbourhood". I was able to open a ticket with Microsoft though and they took my IP out of the range of bad IPs - subject to continuing good behaviour (which I plan to show, obviously). – Stefan Seidel Nov 5 '17 at 20:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.