1

The Setup

We are running exim version 4.72 on Debian Squeeze 6 (Squeeze) (LTS).

Exim is used for outgoing emails only - it does not handle emails from other servers/domains - it only handles (outgoing) message from a local running java (web) application.

For anonymization, lets assume the domain we use for our company email addresses, as well for the domain our java web application is running on, is called example.com.

The Background

We have a web application written in a java framework which allows it's users to send out emails. The envelope-from address of these emails belongs to our domain: support@example.com.
Technically these emails are created and get send from the java web application (we use the JavaMail API) locally to exim, which then eventually sends them out to their destination address.
Incoming emails to our domain are handled by Google Apps/Gmail (We also set up SPF records, etc.)
This setup is working fine without any problems so far.

For completion, here is our (anonymised) update-exim4.conf.conf:

root@server:~# cat /etc/exim4/update-exim4.conf.conf 
# ... comments are skipped ...

dc_eximconfig_configtype='internet'
dc_other_hostnames='node1.example.com'
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost=''
CFILEMODE='644'
dc_use_split_config='true'
dc_hide_mailname=''
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'

The Problem

Recently we are receiving bounce emails because users of our java web application tried to send out emails to non-existing email addresses.

Lets have a look on such a case in the (anonymised) log file:

root@server:~# cat /var/log/exim4/mainlog
# try to send an email to an unrouteable address
2014-06-01 17:34:07 1Wr7lv-0000CR-G0 <= support@example.com H=localhost (node1.example.com) [127.0.0.1] P=esmtp S=620363 id=484648301.663.1401636847496.JavaMail.webapp@node1
2014-06-01 17:34:07 1Wr7lv-0000CR-G0 ** doesnotexist@anotherdomain.com: Unrouteable address
2014-06-01 17:34:07 1Wr7lv-0000CR-G0 Completed

# Immediately a bounce message is generated and gets send to the the origin sender
2014-06-01 17:34:07 1Wr7lv-0000CU-OZ <= <> R=1Wr7lv-0000CR-G0 U=Debian-exim P=local S=108065
2014-06-01 17:34:09 1Wr7lv-0000CU-OZ => support@example.com R=dnslookup T=remote_smtp H=aspmx.l.google.com [74.125.136.26] X=TLS1.0:RSA_ARCFOUR_SHA1:16 DN="C=US,ST=California,L=Mountain View,O=Google Inc,CN=mx.google.com"
2014-06-01 17:34:09 1Wr7lv-0000CU-OZ Completed

What we want to achieve

We now want to show (or notify - whatever) each java web application user within the java web application that an email which he/she wanted to send actually failed to sent.
(Remember: The bounce email are currently sent back to support@example.com - not to the web application users email address - so these users have no clue their emails did not get send)

The basic (and simple) idea we now have:
Make exim write the id of each failed message (which in the above case is 484648301.663.1401636847496.JavaMail.webapp@node1) into a local running PostgreSQL database as soon as possible we know the message delivery failed (or: as soon as possible the corresponding bounce message was generated and/or sent).
If this is possible, we could easily read these "failing" ids from the database within our java web application, and - voilà! - we can show users that their messages have been bounced.
Simple idea, isn't it?

Just for a better understanding: The mentioned (unique) ids already get generated for each message within the java web application before the JavaMail API sends the message to exim - so we can be sure our java web application knows each id for each message (and of course also knows which user sent out a specific message).

Possible Solutions?

I did some research on how this could be solved and this is with what I came up with by now (Please correct my if I am wrong - these are just my thoughts right now):

Approach 1: Pipe the failed message and it's id to a bash script. Such a bash script could then easily store the id in the local PostgreSQL database. Is this possible and how could this be achieved? The problem I see is that as soon a message failed, no other exim router gets executed - which means I can not write a router which handles a failed message - the failed message would never reach the router. Or am I wrong?

Approach 2: Another possibility I was thinking about is to not have a look at the original message which failed, but the newly generated bounce message. As far as I understand this newly bounce message is also going through all routers until one accepts it. So maybe we could write a router which checks if a message is a bounce message and then pipe it to the bash script? But how do I know that if a message is a bounce message? Does it have a special header or something else significant? (Remember: As we only use exim for outgoing emails we can be sure this would always be a bounce message "generated" by our java web application users).

Approach 3: Similiar to approach 2, but instead of writing a router we could make use of the system filter, which looks after a bounce message and then executes a command / pipe...

Approach 4: Would it be possible to configure error_copy or even error_to to not be an email address but a pipe to shell script (which of course received the id and/or whole failed message)?

Two more things: It seems exim can also talk directly to databases (I read this somewhere) - but I have no idea how I could make use of this for my described problem. Plus: However this is solved, I do not want to modify any configuration files which gets overriden when exim gets updated via apt-get (I do not want to re-configure exim after each update).

You help is needed

How would you solve this problem (saving the failed message ids into the database)? Please correct me If I was wrong somewhere - I am not that deep into the exim configuration. Also, as you can see, the solutions I think about are more theoretical - it would be nice I someone could show me real world configurations and where to put this configs to make them work.

Thank you very very much for any help!

1

Exim version 4.82.1 has a feature called TPDA (Transport Post Delivery Action) which would allow you to do exactly what you want. You should be able to switch your repository sources and install that newer version of exim which will give you a version that has that capability.

If you insist on staying with your current distribution provided version, I don't think any of those are good options.

  1. The best solution is to create a unique sender for each message. For example, take the recipient, append the time, and then create an MD5 hash, and make that be the sender @support.example.com. Put that info in a table with information about who it was getting sent to as well as who was sending the email. When you handle email inbound to the domain support.example.com, if the recipient exactly matches one of those hashes, you can do a lookup to see who sent it, notify the sender, disable the recipient from future sending, etc.
  2. The next approach is to have the support@ email address be delivered to two places: a mailbox that your support staff can read and the second is a script that you write which figures out who sent the message, notify the sender, and disable the recipient from future sending, etc.
  3. One option you didn't suggest was writing an application to tail your log files, parse the lines and extract the delivery information and put it in the database then, notify the sender of the problem, and disable the recipient from future sending, etc.
  • Thanks! I also just found out about TPDA, which looks very promising. I found the bug and the commit - right now it seems it's still experimental (but, according to the bug, people use it in production already), which means I would have to compile exim with the EXPERIMENTAL_TPDA=yes flag by hand right now. Is compiling exim by hand the only option or are there any binaries/repositories? – mkurz Jun 1 '14 at 23:20
  • Also I could not find out how to install a newer version of exim on this Debian Wheezy server right now by just switching the repository source, it seems version 4.72 is the last one offered by the debian team right now (I couldn't find it in the backports either). I would love to use TPDA right now, because it is definitely the way to go in the future - but if compiling by hand is the only option it would be a bit more distracting right now. – mkurz Jun 1 '14 at 23:24
  • It should be available in Debian Experimental. That repo also has the latest 4.83-RC1: lists.exim.org/lurker/message/20140529.115944.2152882d.en.html – Todd Lyons Jun 2 '14 at 14:42
  • 1
    But one more thing: I got TPDA working so message get logged in PostgreSQL when for failures like Network is unreachable - this works fine. But it does not log errors like Unrouteable address. This is probably because TPDA is for transport only and if the address is not even routeable it does not even get from the router to a transport – mkurz Jun 4 '14 at 14:26
  • 1
    Correct, you'll have to do that particular check in an ACL when you do verify=recipient. – Todd Lyons Jun 5 '14 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.