I'm going to preface this with a statement that I am no expert in sendmail. I rarely use it, but in this situation I have to for my client.
A client of mine has a mail server running Debian 7.7, Sendmail 8.14.4, and OpenSSL 1.0.1e. It generates emails (stuff like account reset) and sends them to customers. I'll call it "Server A".
Messages sent from Server A to any mail server at the domain *.outlook.com (which strangely doesn't include @outlook.com messages, those are handled by hotmail) were being deferred due to a failed TLS handshake. This is only a problem for servers owned by Microsoft; everything else is working. The error in the log looked something like:
sendmail: ruleset=tls_server, arg1=SOFTWARE, relay=mail.protection.outlook.com, reject=403 4.7.0 TLS handshake sendmail: s2L9EakQ022098: to=<email@example.com>, delay=00:00:55, xdelay=00:00:31, mailer=esmtp, pri=181653, relay=mail.protection.outlook.com. [126.96.36.199], dsn=4.0.0, stat=Deferred
Note: These logs are made up because my client doesn't want things copied and pasted less they accidentally leak sensitive data. The information is there, I just typed it by hand so ignore typos, etc.
That was all I could get from the logs, even with the logs on 15 (Beyond that, the system started to become disk bound).
A tcpdump of the handshake showed that Server A was connecting to the outlook.com, EHLO went sucessfully, Server A initiated STARTTLS, outlook.com then responded with its certificate chain. All of this went normally.
Then after Server A received the microsoft certificate chain, Server A sent its own client certificate. Outlook.com then dropped the connection.
Our client certificates are self-signed certificates that we use for intraorganizational verification and shouldn't go out to outlook.com.
From my limited knowledge, my best guess is that outlook.com is requesting the client certificate for verification (or at least sendmail thinks its requesting the certificate), which sendmail on Server A is obliging. Outlook.com is then dropping the connection, either because the certificate is invalid or it is not what it was expecting.
Why is sendmail sending the client certificate to outlook.com as part of STARTTLS?
Is there a way that I can prevent sendmail from sending client certificates to servers outside our domain, even if the server requests it? From the experiments I've done, outlook.com will accept the message if I simply ignore the certificate request and send the message.
Before anyone suggests it, I have already created a access map that prevents TLS from connections with some IP addresses associated with Microsoft. This isn't a good permanent solution because I would prefer using TLS, and the list of IP address has to be manually maintained, which is a bother.