10

What is the best way to parse the headers of bounced (undeliverable) email that is sent back to my server and determine whether it is a soft or hard bounce?

I only send opt-in emails to my users, but occasionally some email addresses go stale. When an email bounces back to my server, I would like to find why it bounced (soft/hard). Then I can deal with it appropriately in my database and/or flag the user to update their email when the log in next.

I'm using Ubuntu and Postfix. I have successfully implemented VERP with aliases and virtual aliases. So bounced emails have a return-path of [email protected], and I can pipe them to a script.

Now that I have VERP setup, I know who the original email was sent to, but I need to parse the returned mail headers to figure out if it is a soft bounce or hard bounce.

What is the best way to handle this? As I understand it, not all mail servers play by the same rules, and headers can have a variety of formats. Is there some open source project that keeps track of these types of things? Something simple I can implement that will categorize the majority of bounces properly?

I'm trying to protect the reputation of my mail server, so any help is much appreciated!

2 Answers 2

12

As RFC3463 explains, status codes beginning with 5 are used for permanent failures and 4 for persistent transient failures. Instead of trying to parse several messages with different formats you could rely on server logs and try something like this:

grep " dsn=5." /var/log/mail.log | grep -o -P " to=<(.+?)>" | sort | uniq -c

This will find permanent errors from mail.log (Postfix format) and give the addresses and amount of bounces on every address. You can also use " dsn=4." to get addresses with temporary errors.

2
  • Thanks Esa! I didn't realize that postfix had that info in the mail log. Is this the solution that you use? Do you find that postfix categorizes the hard bounces dsn=5 correctly? I have read that some mail servers don't comply with RFC. So I thought a more complicated solution might be required. What has your experience been? This seems like a good solution if we can test postfix to get it right :-)
    – Richard
    Mar 6, 2015 at 16:27
  • 1
    Really useful script - thank you! Recipe here for an alternative to grep's -P flag (for Mac users etc): unix.stackexchange.com/a/437694/275762 grep " dsn=5." /var/log/mail.log | pcregrep -o1 " to=<(.+?)>" | sort | uniq -c
    – Peter M.
    Apr 16, 2020 at 21:30
9

Generally there are two types of bounces

  1. The bounces caused by directrejection of remote mail server when your postfix deliver the email.
  2. The bounces caused by remote server (next-hop server after your postfix) fails to deliver the message to final recipients.

The first case was already covered by excellent answer by Esa Jokinen above. Your best bet is parsing maillog.

The second case was special case of bounces. The example scenario:

  • You send email with recipient [email protected] to mail.example.com server.
  • In mail.example.com, [email protected] was aliased to [email protected] and must be forwarded to mail.example.net.
  • Someday mail.example.net reject the your message so mail.example.com must send bounces to your server.
  • Unfortunately maillog in your server will have "dsn=2" because mail.example.com already accepted the message but failed to forward it to mail.example.net.

Here the example of second type bounces email. There is forwarding rule Yahoo mail server [email protected] -> [email protected]. Unfortunately mail server of example.net reject the message :(

From MAILER-DAEMON  Thu Mar  5 05:07:26 2015
Return-Path: <>
X-Original-To: [email protected]
Delivered-To: [email protected]
Received: from nm21-vm7.bullet.mail.gq1.yahoo.com (nm21-vm7.bullet.mail.gq1.yahoo.com [98.136.217.54])
        (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
        (No client certificate requested)
        by mx.example.org (Postfix) with ESMTPS id D6365565FC
        for <[email protected]>; Thu,  5 Mar 2015 05:07:25 +0700 (WIT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=bounce; t=1425506842; bh=zk/tWZNl6c36dmlPDmakM9ekK8cHVJANXMmSdsbkcWc=; h=From:To:Date:Subject:From:Subject; b=Im95h1qTg6qN3yUI7vF1fXtJ0SbUnzv8rUPwLbpNwxGPN2p8wfosXJzQgJ3nzr4L4ZQ50P2d9E9U4jEUNtnyi7nlFd5kKbtiVuda4H56h1PFnt+7wSpgHcd5Irs/lLODumb6ZZSEpCOWttcB9+JLaDfEUUPjGcbR+xww4XeH5Eo=
From: [email protected]
To: [email protected]
Date: Wed, 04 Mar 2015 22:07:22 -0000
Subject: Failure Notice
X-Yahoo-Newman-Property: bmbounce

Sorry, we were unable to deliver your message to the following address.

<[email protected]>:
Remote host said:
550 5.1.1 User unknown
 [RCPT_TO]

For this case, your only method is parsing the bounces message. Unfortunately there are no standard bounces format, so you must parsing the body and determine the rejection caused.

The feature checklist of your postfix bounce parsing:

  1. Check if VERP address was valid. You don't want to parse invalid message.
  2. Parse the body, determine if they are soft or hard rejection.

For the second feature, you can google some common rejection message. The example is this bounce-regex-list.xml by Jakub Liska.


Esa Jokinen made a good point in the comment below about these two bounce types. If your goal is keep the server reputation, then dealing the first bounce type should be enough. The second bounce was about cleaning your lists. So dead email should be erased thus freeing some resources in your server.

Some mailing list managers such as PHPlist and Mailman also deal with this bounce problem with parsing the email body as they have no resources to parsing the maillog.

2
  • 2
    This solution is useful and more thorough if it is necessary to handle also mails automatically forwarded to another address. However, if the goal is to protect the reputation of the mail server, handling direct rejections should be sufficient. Administrators of the forwarding MTA should remove outdated forwards and mail lists (to protect their reputation and to avoid unnecessary traffic). After that we are back in the case one. OP should use this solution if the amount of secondary bounces is significant. Which ever takes less effort. Mar 5, 2015 at 12:16
  • @masegaloeh, thanks for the info! I hadn't even considered that forwarding situation as a possibility! For now I am mainly concerned with protecting the rep of my mail server, but if bounces increase this may be very useful.
    – Richard
    Mar 6, 2015 at 16:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .