1

I am mainly asking this to check whether I should be investigating further on my end, or if I should tell the sender they should talk to their own IT department for further assistance. I am not sure if this is on my end or theirs.

From one external domain, we are not receiving any emails. The only tracking I can see is an initial SMTP conversation logged on my SMTP Virtual Server. The log shows the EHLO, STARTTLS, then MAIL, and last a QUIT. There's no RCPT sent from them that I can see anywhere. I even used Network Monitor to see if there was anything in the packets, nothing showed.

We do have GFI Mail Essentials on the same Exchange server, but nothing is logged, not even any addresses from that domain in the history. In Exchange event logs, with Transport logging all on maximum, there are zero log entries with their IP's or addresses or domain. Basically, I think the mail never goes into the Exchange transport pipeline.

Also, checked, no blacklisting on our or their servers. And, DNS lookups all resolve, reverse PTR as well from my end.

We are just a small shop, a few hundred users, and not a full time IT staff, only a single Exchange server for all email. The sender is a large federal government organization with many inbound MX records and all outbound is coming from separate SMTP servers.

Currrently I am waiting to hear from my users to see if any NDR's were received on their end. I am just confused at this point, mainly since, obviously, I can see anything from their end.

Could this be on my end still? I don't know what else to do except tell them it on their end, which I hate to do. (because I hate it when other IT people do that to me without checking everything)

Thank you for any advice or suggestions.

Below is a sample SMTP log:

2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 EHLO - +outside1.domain.com 250 0 343 19 0 SMTP - - - -
2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 STARTTLS - - 220 0 0 8 0 SMTP - - - -
2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 STARTTLS - - 220 0 29 8 0 SMTP - - - -
2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 EHLO - +outside1.domain.com 250 0 353 19 0 SMTP - - - -
2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 MAIL - +FROM: 250 0 76 41 0 SMTP - - - -
2014-06-27 19:07:36 52.0.222.46 outside1.domain.com SMTPSVC1 MAIL 192.168.0.4 0 QUIT - outside1.domain.com 240 343 76 41 31 SMTP - - - -

2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 EHLO - +outside2.domain.com 250 0 343 19 0 SMTP - - - -
2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 STARTTLS - - 220 0 0 8 0 SMTP - - - -
2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 STARTTLS - - 220 0 29 8 0 SMTP - - - -
2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 EHLO - +outside2.domain.com 250 0 353 19 0 SMTP - - - -
2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 MAIL - +FROM: 250 0 76 41 0 SMTP - - - -
2014-06-27 19:14:56 52.3.222.45 outside2.domain.com SMTPSVC1 MAIL 192.168.0.4 0 QUIT - outside2.domain.com 240 516 76 41 63 SMTP - - - -

By the way, I did notice some SSL traffic in the Network Monitor where they request the STARTTLS. In that I see ClientHello from both parties, that always end in an SSL Error about authentication invalid or similar. I have a hunch this is not my problem, because its also showing on other normal traffic. I believe the servers are simply attempting TLS connection, then falling back on normal unencrypted connection. Hence the second EHLO. (right? I am just making a guess on that, I really don't get the SSL protocol stuff at that level.) I really think this is something with the lack of RCPT in the transmission, no mail will route without that.

I just don't know what else to check, is it me, or them? Won't know more until I get report of an NDR from them. Thanks!

UPDATE 6-30-2014 Finally received an NDR from sending users:

4.4.0 - Other network problem "(336130315, 'error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number')"

After some initial research, my first thoughts are an obvious fix; upgrade the server, its old! But I am wondering if there's anyway for me to work around this? Just for the time being.

From what I am gathering so far, Exchange 2003 or rather Windows 2003 doesn't support greater than TLSv1. And also, it has SSLv2 enabled. From the Googling on that NDR message, I see posts about Postfix servers needing to disable 3DES to workaround, or they have SSLv2 disallowed, which my server is announcing as available. My server does not have any of the standard 2003 server protocols disabled though, so it should try SSLv3 right?

Except, I am also reading posts about buggy 2003 server ciphers and broken TLS, so not sure what I can do. Couldn't I just disable SSL/TLS inbound on SMTP?

I'll keep Googling, but not sure how to fix this except to move away from 2003 server.

UPDATE AGAIN

I bet this has to do with Heartbleed OpenSSL upgrades on the sending servers. It looks like Exchange 2003 has can only handle a cipher list length of 64 items (Not verified yet). The ciphers used on Exchange 03 are too low down the list by the sending SMTP, and Exchange fails. I don't know if I'll be able to verify this is indeed the issue, but sure seems likely.

Found this to be interesting: http://comments.gmane.org/gmane.mail.mimedefang/17927

  • Exchange 2003 has reached its end of extended support on 4/8/2014. I highly recommend switching to a supported e-mail system. support.microsoft.com/lifecycle/search/… – MichelZ Jun 28 '14 at 14:35
  • I agree completely. I am not the main IT guy for this network, I am just covering for him while he's in the hospital for a few weeks, but I've told him before we need to update the server "when we have time". I have spoken with the directors and at least put in motion some planing for either hosted Exchange or upgrading the server. So its in the works! But its going to take some time, I can't dedicate full time to this, and we'll have to coordinate over next few months. – gregthegeek Jun 28 '14 at 19:30
1

2014/07/29 Update

Apparently SMTP service just sends some part of SSL certificate as a crap at the end of SSL packet ; ) https://lbr.id.lv/#Windows_SMTP_bug_breaks_3DES_and_AES_CBC

Solution

KB957047 fixes SMTP, KB938857 - Exchange IMAP and POP3, KB948963 adds AES support and both 3DES and AES start to work properly with SMTP service and Exchange server. It may be necessary to install KB976323 after SMTP hotfix.

Original post

Imo it is deffo connected to SSL/TLS. I have exactly the same issue - Windows Server 2003 SMTP service fails to receive some emails over SSL/TLS. And has nothing to do with block lists/ips/network issues/etc.

Btw, registration confirmation from serverfault was delivered ok -

   Protocol: TLS (SSL 3.1)
   Cipher: RC4
   Cipher strength: 128
   MAC: SHA
   Exchange: RSA
   Exchange strength: 4096

   EHLO - +mx-out.stackexchange.com 250 0 235 29 0 SMTP - - - -
   STARTTLS - - 220 0 0 8 0 SMTP - - - -
   STARTTLS - - 220 0 29 8 0 SMTP - - - -
   EHLO - +mx-out.stackexchange.com 250 0 259 29 0 SMTP - - - -
   MAIL - +FROM:<do-not-reply@serverfault.com> 250 0 78 50 0 SMTP -
   RCPT - +TO:<> 250 0 51 23 0 SMTP - - - -
   DATA - +<> 250 0 167 3706 297 SMTP
   QUIT - mx-out.stackexchange.com 240 1312 83 4 0 SMTP - - - -

Oh, I've completely forgot to answer ; )

You can either

1) Disable 3DES on w2k3 - will result in either fallback to RC4 or insecure connection. Also disabling 3DES for me resulted in some other issues. For example IIS6 and Outlook are able to use it without any issues.

2) Disable TLS on SMTP service - will result in unsecure connection.

3) Beg remote server admin to implement patch ; )

4) Upgrade w2k3.

5) Use some other SMTP server, which is not an option for you cause Exchange only works with m$ SMTP, but is an option for me.

Also there is a chance that forcing either SSL3 or TLS1 with specific cipher would work.

Btw, I think you are right, it all started after HeartBleed bug paranoia - prly OpenSSL got upgraded and prly it has either 3DES as lowest cipher or RC4 is very far on the list.

Also, sadly in your and mine scenarios non-TLS connection is not being re-negotiated, cause both servers think that common matching cipher was found and connection was successfully established. Second EHLO prly is TLS handshake or is just required after establishing TLS connection.

  • I am marking this as the accepted answer, but joeqwerty deserves some credit for excellent troubleshooting help. This is exactly the issue for me, the SSL stuff. Our actual solution: Upgrade the server, in time, or possibly moving to hosted Exchange. For the few problem email senders, we are forcing users to just use hotmail or other free service. I know, its a stupid workaround, but its only temporary. – gregthegeek Jul 11 '14 at 19:54
  • I did some more digging using WireShark - lbr.id.lv/#Windows_3DES_CBC_Issue Turns our Windows SMTP just sends some garbage. And there is update for Exchange(IMAP, POP3), which solves the issue - support.microsoft.com/kb/938857/en-us But there is no update for SMTP service.. – lbr Jul 28 '14 at 16:02
  • And some more digging - lbr.id.lv/#Windows_SMTP_bug_breaks_3DES_and_AES_CBC KB957047 fixes SMTP, KB938857 - Exchange IMAP and POP3, KB948963 adds AES support and both 3DES and AES start to work properly with SMTP service and Exchange server. – lbr Jul 29 '14 at 16:09
0

This very well could be, and likely is, your server. Check your Connection filtering to see if you have:

  1. A block list provider configured

  2. The ip address of the sending server in your Global Deny list.

The easiest way to check if either of these is causing the problem is to disable Connection Filtering on the properties of the virtual SMTP server. If you disable Connection Filtering and these emails start coming through then you know the problem is with your Connection Filtering configuration, either with the block list provider you're using or with your Global Deny list.

Also, see here for a good rundown of message flow in Exchange Server 2003, especially the section near the bottom that details anti-spam message flow:

http://technet.microsoft.com/en-us/library/aa995825(v=exchg.65).aspx

  • Thank you for the quick response. I'll definitely check the filters and disable them. The sender seems to retry about 20 minutes after the hour, so I'll just disable them for a few and see if they come through. As for filter block list provider (GFI), its not doing any IP filtering at that point, but is of course doing standard spam checks. But the thing that gets me, its not logging anything from this domain or IP. Still, the quick test is easy. Thanks again! – gregthegeek Jun 28 '14 at 5:22
  • The Block List Provider I'm referring to is in the configuration of the Exchange Server. Look at the properties of Global Settings|Message Delivery| on the Connection Filtering` tab. – joeqwerty Jun 28 '14 at 14:26
  • Oh, of course! will check that, thanks for clearing that up for me. :) – gregthegeek Jun 28 '14 at 19:27
  • Ok, so none of the disabling filters helped. We had Connection and Recipient filtering enabled only. I disabled them both, plus disable the GFI services, and also checked any IP addresses on the vhosts if sender was in those IP ranges, still same result. No RCPT was sent to us by them. I did also browse Exchange SMTP routing guide and as I thought, SMTP does a few things first, like filters that may be enabled, then sends inward to Exchange for further routing once accepted. I still don't get why I don't see any evidence of RCPT command send from the sending SMTP server. – gregthegeek Jun 28 '14 at 21:32
  • continue... I would think my SMTP would at minimum log an RCPT, even if its blocked or invalid, like an invalid recipient would fail to receive, but still be logged by SMTP. I think at this point, unless I can come up with some other cause of our system blocking them, I can't really do much until I see an NDR by them. I have requested this from my users, but prob won't get any reply till Monday. @joequery, thanks very much for your help. I am happy to test anything else you might suggest. – gregthegeek Jun 28 '14 at 21:36

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.