I have a client who uses an external mail server to log and filter their inbound email before passing it on to their main mail system (which is Office365). The external server is running Sendmail, formerly on Debian, now on Ubuntu (Amazon's version).
The change of OS (and hosting) has also involved a bump in the version of Sendmail, from 8.14.4 on the older server (quite elderly, yes) to 8.15.2 on the newer one. Unfortunately it's also induced a slight change in behaviour and I am struggling to find whether this is controlled by a flag or other config setting.
The external server does some filtering of spam and other cruft and, up until the change of server, all of the remaining mail was successfully delivered to O365. (Some of it was marked as spam there, but it all got to their servers at least.) This is no longer true. Most of the mail which is being held, and not delivered, is bounce messages from their mailshots, but they do need to see those to clean their lists.
It seems that the mail which is not being delivered is mail which has names which are not fully qualified in the mail headers. They aren't from the hop where it arrives on the external server -- we would reject e.g. nonexistent domains at that stage -- but from earlier on in the mail's journey. So we might see headers like
Received: from EXTERNAL.com (EXTERNAL.com [NN.NN.NN.NN])
by OUR.SERVER (8.15.2/8.15.2/Debian-10) with ESMTP id xELIDED
for <[email protected]>; Sat, 11 May 2019 10:49:17 GMT
Received: from EXTERNAL.com.local (EXTERNAL.com [NN.NN.NN.NN])
by EXTERNAL.com (8.16.0.22/8.16.0.22) with ESMTPS id xELIDED
(version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT)
for <[email protected]>; Sat, 11 May 2019 18:49:15 +0800
-- it is the .local header which is causing the problems. But we aren't seeing these problems when the mail arrives at the server, only when it leaves the server to head to Office365. At that point, we see
EXTERNAL.com.local: Name server timeout
And the transaction concludes with
timeout writing message to OURDOMAIN-com.mail.protection.outlook.com.
<[email protected]>... Deferred: Name server: OURDOMAIN-com.mail.protection.outlook.com.: host name lookup failure
Closing connection to OURDOMAIN-com.mail.protection.outlook.com.
i.e. the eventual failure message suggests that it's failing to connect to O365. This isn't the case since the connection starts successfully, other connections to the O365 servers are connecting successfully (this is a moderately high-traffic server).
Sendmail logging and tcpdump shows that the initial part of the SMTP connection goes fine. In the DATA section, the headers are transmitted and then the connection terminates due to not being able to look up a hostname which is not immediately relevant to the connection (it never gets as far as transmitting any of the body of the email).
As well as situations where there was a hostname within the headers which could not be resolved, I've also seen this happen on an email where the reply-to had been set to a domain which did not exist and was thus not resolvable. (Quite likely spam but that isn't really the issue here.)
I've looked through the change logs for the relevant versions of Sendmail and haven't spotted anything obviously relevant; I've also spent a little bit of time poking through the source code. The logs from tcpdump show the connection being terminated on our side, not Microsoft's -- we have an engineer from their side attempting to help us but, because there is never a successful connection for these mails, they're struggling to see what is going on. DNS lookups for everything else seem to be working fine.
If anyone knows where to find the config which will say "don't try to look up irrelevant hostnames", I'm all ears. These aren't our misconfigurations!
Thanks in advance.
Edit Adding in a bit of the strace log:
connect(9, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
sendto(9, "<22>May 15 09:52:20 sendmail[135"..., 176, MSG_NOSIGNAL, NULL, 0) = 176
>>> EHLO our.server.name
250-VE1EUR02FT009.mail.protection.outlook.com Hello [NN.NN.NN.NN]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 SMTPUTF8
>>> MAIL From:<> SIZE=23108
250 2.1.0 Sender OK
>>> RCPT To:<[email protected]>
>>> DATA
250 2.1.5 Recipient OK
354 Start mail input; end with <CRLF>.<CRLF>
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 12
connect(12, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
sendto(12, "R>\1\0\0\1\0\0\0\0\0\1\2the\4dodgy\3com\5local\0\0"..., 46, MSG_NOSIGNAL, NULL, 0) = 46
recvfrom(12, "R>\201\202\0\1\0\0\0\0\0\1\the\4dodgy\3com\5local\0\0"..., 8192, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 46
[repeats six more times]
the.dodgy.com.local: Name server timeout
timeout writing message to OURDOMAIN-com.mail.protection.outlook.com.
sendto(9, "<18>May 15 09:52:20 sendmail[135"..., 130, MSG_NOSIGNAL, NULL, 0) = 130
<[email protected]>... Deferred: Name server: OURDOMAIN-com.mail.protection.outlook.com.: host name lookup failure
sendto(9, "<22>May 15 09:52:20 sendmail[135"..., 296, MSG_NOSIGNAL, NULL, 0) = 296
Closing connection to OURDOMAIN-com.mail.protection.outlook.com.
+++ exited with 70 +++
So the lookup that fails is the one on the.dodgy.com.local
(referred to as EXTERNAL.com.local
in the earlier sanitised log but I wanted to preserve the structure of the function calls here), but the one that is reported back by Sendmail is a failure of a lookup on the server which it was connected to at the time. It isn't doing that lookup and it wouldn't have failed if it did.
Edit (2): changing the DNS resolver does not help.
/etc/resolv.conf
. They are probably broken.OURDOMAIN-com.mail.protection.outlook.com.
doesn't actually exist then?EXTERNAL.com.local: Name server timeout
error mentioned above, and terminates complaining about a nameserver timeout onOURDOMAIN-com.mail.protection.outlook.com.
, but the timeout on the legit domain never actually happens.