22

I have aliases set up in postfix, such as the following:

[email protected]:    [email protected], [email protected] ...

When an email is sent to [email protected], and any of the recipients in that alias is cc:ed which is quite common (ie: "Reply all"), the e-mail is delivered in duplicates. For instance, if an e-mail is sent to [email protected] and [email protected] is cc:ed, it'll get delivered twice. According to the Postfix FAQ, this is by design as Postfix sends e-mail in parallel without expanding the groups, which makes it faster than sendmail. Now that's all fine and dandy, but is it possible to configure Postfix to actually remove duplicate recipients before sending the e-mail?

I've found a lot of posts from people all over the net that has the same problem, but I have yet to find an answer. If this is not possible to do in Postfix, is it possible to do it somewhere on the way? I've tried educating my users, but it's rather futile I'm afraid...

I'm running postfix on Mac OS X Server 10.6, amavis is set as content_filter and dovecot is set as mailbox_command. I've tried setting up procmail as a content_filter for smtp delivery (as per the suggestion below), but I can't seem to get it right. For various reasons, I can't replace the standard OS X configuration, meaning postfix, amavis and dovecot stay put. I can however add to it if I wish.

4
  • I think it's really just not possible... Feb 15, 2010 at 12:40
  • I'm truly starting to believe that you are right Antoine. Feb 24, 2010 at 10:07
  • Well that was a waste of a bounty... Feb 25, 2010 at 15:26
  • 2
    did you ever find a solution for this? I'm having the same issue. Sep 29, 2014 at 17:12

7 Answers 7

6

There is a way how to do it, but not using Postfix itself.

If you are using a reasonably recent Dovecot with Pigeonhole implementation of Sieve, there is a special extension that can be used for on-delivery deduplication. In Dovecot older than 2.2.18, the name of this extension is "vnd.dovecot.duplicate" and it's disabled by default, in newer releases the extension is called "duplicate" and should be already enabled, but enabling it explicitly won't harm.

/etc/dovecot/conf.d/90-sieve.conf:

plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve

  sieve_before = /mnt/mail/users/global_sieve/deduplicate.sieve
  # sieve_before2 = /mnt/mail/users/global_sieve/antispam.sieve

  sieve_extensions = +vnd.dovecot.duplicate  # for dovecot < 2.2.18
  #sieve_extensions = +duplicate             # for dovecot >= 2.2.18
}

Then create the sieve script to handle the deduplication itself (you can adjust the filename as you see fit).

/mnt/mail/users/global_sieve/deduplicate.sieve:

require "vnd.dovecot.duplicate"; # for dovecot < 2.2.18
# require "duplicate";           # for dovecot >= 2.2.18

if duplicate {
    discard;
    stop;
}

Compile the script using sievec and make sure the script is readable by dovecot user. More in Dovecot docs.

If you are using Cyrus, there is a a duplicate message delivery suppression feature that can be enabled using suppress_duplicates = yes.

4
  • Thanks a ton for pointing this mechanism out. That "finding a needle in a haystack" feeling.
    – lkraav
    Oct 26, 2016 at 21:28
  • Discoveries during implementation: duplicate extension works at user level. I was trying to stop duplicates over multiple users To/Cc-d in a central fileinto mailbox. This doesn't work. Had to also set a postfix virtual_alias_maps catch-all entry, such as @domain.com [email protected], then the sieve duplicate check will hit catchall user's duplicate list.
    – lkraav
    Oct 26, 2016 at 22:54
  • Yeah, that's very similar to "finding a needle in a haystack" feeling
    – Net Runner
    Aug 2, 2017 at 13:01
  • Thank you! This answer works for me. I setup 2 aliases, each one of them, has the same members. And then, I sent from [email protected] to the 2 aliases at the same time and I only received 1 message at each of the alias member inbox. Before I always got 2 copies at each alias member inbox.
    – busythomas
    Aug 5, 2020 at 8:02
5

Postfix has no idea about duplicate emails due to the way it's structured. It is possible to do what you're suggesting by using procmail as your delivery agent.

Essentially, each message coming from a client should be delivered with a unique Message-Id. In the case that it's delivered to multiple people, the Message-Id should be the same, so we save any Message-Id headers we've seen and discard and future ones that match that list.

From http://novosial.org/procmail/

:0 Wh: msgid.lock
| formail -D 8192 ~/.procmail/msgid.cache
6
  • If I understand the problem correctly, it isn't possible to solve in Postfix because Postfix sends the e-mails in parallel, ie it sends to [email protected] at the same time as it expands [email protected] and then again sends to [email protected]. Wouldn't the Procmail solution then possibly introduce a race condition where another unrelated e-mail (with a different message id) is sent in between the two e-mails in question and thus overwriting the cache making it so that duplicate e-mails are delivered anyhow? Feb 15, 2010 at 16:14
  • I'm trying to get your suggestion to work, but I'm not really sure how to combine this with dovecot delivery, which is used by OS X 10.5+ by default. Feb 15, 2010 at 22:46
  • If you're using dovecot as a delivery agent, you can use procmail as a content_filter which will allow you to perform the same task on a global level Feb 16, 2010 at 9:23
  • I have to confess that I probably am in way over my head here. I've been staring myself blind on the configuration parameters page for postfix and googling till my fingers bleed but I can't seem to figure out how to configure this. Any resources you might know of or nudges in the right direction would really be appreciated! Feb 16, 2010 at 18:14
  • I think I sort of understand how it works now, but there's already a content_filter set to amavis. This seems to be some sort of anti virus checker. I've read that it's possible to chain content_filters, but it seems rather unintuitive and the documentation is less than stellar. It's really quite ridiculous how difficult it is to just weed out duplicate e-mails. Design desicions regardless, this seems to me like it should be in there out of the box or at least not require these monstruos hacks. Feb 16, 2010 at 21:25
1

A solution that work for me is add -o receive_override_options=no_address_mappings on master.cf

Here the doc: http://www.postfix.org/postconf.5.html#receive_override_options

1
  • Hi @Pioz, thanks for your answer, I visited the official doc and I have not yet fully understand, because after I set this option on my master.cf, postfix starts ignoring aliases set in Goto column in virtual alias table. Is this settings was supposed to turn off alias / forwarding feature of postfix?
    – busythomas
    Aug 5, 2020 at 4:07
0

This is from some old postfix faq:

Postfix sends duplicate mail Some people will complain that Postfix sends duplicate messages. This happens whenever one message is mailed to multiple addresses that reach the same user. Examples of such scenarios are:

  • One message is sent to the user, and to an alias that lists the user. The user receives one copy of the mail directly, and one copy via the alias.

    • One message is sent to multiple aliases that list the user. The user receives one copy of the mail via each alias.

Some people will even argue that this is the "right" behavior. It is probably more a matter of expectation and of what one is used to.

This can be "fixed" only by making Postfix slower. In the above examples, Postfix would first have to completely expand all distribution lists before starting any delivery. By design, Postfix delivers mail to different destinations in parallel, and local delivery is no exception. This is why Postfix can be faster than sendmail.

So by design you are seeing that behavior. Perhaps if you find a content filter that can strip duplicate message IDs, you can eliminate this after the delivery event.

Sendmail does not have this problem because it expands everything first and strips out duplicates.

5
  • Yes, I read that, perhaps I should have updated my question. By design or not, it's still an issue. I understand why it's behaving the way it does and I understand that trying to find a solution in Postfix is futile. However, I've tried coming up with some content filter solution using Procmail and I just can't seem to get it right. It's highly likely that I'm not understanding content filters or procmail correctly, or both. In either case I still would like to fix this. Unfortunately, the setup as it is cannot be modified, only added to. I can't use sendmail instead of postfix, unfortunately. Feb 19, 2010 at 1:03
  • I updated the question with some more information but honestly, the link to the FAQ was in there to start with. I do appreciate any helpful suggestions though! Feb 19, 2010 at 1:09
  • Did not realize that was a link to the same content. Working in IT security, you become rather paranoid about clicking random links. Can you confirm that the duplicate messages have the same message IDs? Feb 19, 2010 at 1:21
  • Been a long day so I need to review your original post but what about this: postfix.org/postconf.5.html#duplicate_filter_limit Feb 19, 2010 at 1:28
  • I can indeed confirm that the duplicate messages have the same message IDs. Setting the duplicate_filter_limit does nothing to help unfortunately. Feb 19, 2010 at 12:52
0

The oficial solution is here.. http://osdir.com/ml/mail.postfix.devel/2007-05/msg00010.html

duplicate_filter_limit (10000) The maximal number of addresses remembered by the recipient duplicate filters for aliases(5) or virtual(5) alias expansion, or for showq(8) queue displays (with earlier Postfix releases the default limit was 1000).

duplicate_filter_style (strict) The duplicate recipient filter policy: strict or pragmatic.

2
0
enable_original_recipient=no

I put that in my main.cf and it worked fine but only for mail sent from my domain. I still have duplicates if I send email from outside my domain (yahoo to my domain for eg.)

source : http://article.gmane.org/gmane.mail.postfix.user/119783

0

I was facing the same issue. I tried using the pigeonhole (sieve) approach but it resulted more complicated for such a minor feature (prevent duplicates when storing and forwarding at the same time). I also tried most of the suggestions here without success (is not that they are mistaken, they just didn't work for me).

One way to solve this is to clearly identify aliases vs mailboxes. If you are using emails as usernames, I guess you need to adjust this method. In my case, user accounts are like: user1000 for the email [email protected]. This is how to solve it:

  1. In the configuration where you get the directory path for the user (e.g. mysql.users.cf), you have to be sure to use mailbox accounts only (not email addresses) to search:
SELECT home_dir FROM accounts WHERE CONCAT(username, '@', 'mail.example.com')='%s'

It should return the full path to the inbox if exists, or nothing if the username doesn't exists or is just an alias for another email (account without mailbox). You can test with this command:

postmap -q '[email protected]' mysql:/etc/postfix/mysql.users.cf

(please note that we use [email protected] and not [email protected])

NOTE: mail.example.com is the FQDN of your mail server and it is used to identify mailboxes from incoming emails and external accounts. You can not use your domain (e.g. example.com) here as postfix won't be able to tell the difference between incoming emails and forwarded ones (resulting in duplicate emails). You could use localhost instead, but postfix will complain as @localhost is used mainly for OS physical accounts.

  1. In the alias configuration file (e.g. mysql.aliases.cf), return the addresses you want to store or forward to:
SELECT address FROM aliases WHERE email='%s'

You can return a list of addresses (one by line) or a single line separated by comma. You can test with this command:

postmap -q '[email protected]' mysql:/etc/postfix/mysql.aliases.cf

Now, the trick here is that you must return only email addresses , do not return account names here, instead append your server FQDN as explained above. For example, the previous SQL should return something like (the order is not important):

SELECT address FROM aliases WHERE email='[email protected]'

[email protected]   <-- external
[email protected]  <-- internal/local
[email protected]  <-- internal/local
[email protected]      <-- internal/local
[email protected]      <-- external

(note that we are returning local mailboxes as *@mail.example.com as explained above)

In this way, postfix can clearly identify what should be stored and what should be forwarded and duplicates emails are no more.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .