In short: legitimate emails are landing in Junk folders as EOP (Exchange Online Protection) stamps email messages as junk (SCL5) and SPF-failed. This happens with all external domains (e.g. gmail.com/hp.com/microsoft.com) to client’s domain (contoso.com).
We are at the beginning of migrating mailboxes to Office 365 (Exchange Online). This is a Hybrid Deployment/Rich-Coexistence configuration, where:
- On-Premises = Exchange 2003 (Legacy) & 2010 (Installed for Hybrid Deployment)
- Off-Premises = Office 365 (Exchange Online)
- EOP is configured for SPF checking.
- MX records are pointing at the on-premises as we haven't completed migrating all mailboxes from on-premises to Exchange Online.
The problem is when external users sends emails to an Office 365 mailbox in the organization (mail flow: External -> Mail Gateway -> on-premises mail servers -> EOP -> Office 365), EOP performs an SPF lookup and hard/soft failing messages with the external facing IP address of the Mail Gateway from which it received the mail.
(On-premises mailboxes do not show this problem; only mailboxes migrated to Office 365 do.)
Example 1: from Microsoft to O365
Authentication-Results: spf=fail (sender IP is 22.214.171.124) smtp.mailfrom=microsoft.com; contoso.mail.onmicrosoft.com; dkim=none (message not signed) header.d=none; Received-SPF: Fail (protection.outlook.com: domain of microsoft.com does not designate 126.96.36.199 as permitted sender) receiver=protection.outlook.com; client-ip=188.8.131.52; helo=exchange2010.contoso.com; X-MS-Exchange-Organization-SCL: 5
Example 2: from HP to O365
Authentication-Results: spf=none (sender IP is 184.108.40.206) smtp.mailfrom=hp.com; contoso.mail.onmicrosoft.com; dkim=none (message not signed) header.d=none; Received-SPF: None (protection.outlook.com: hp.com does not designate permitted sender hosts) X-MS-Exchange-Organization-SCL: 5
Example 3: from Gmail to O365
Authentication-Results: spf=softfail (sender IP is 220.127.116.11) smtp.mailfrom=gmail.com; contoso.mail.onmicrosoft.com; dkim=fail (signature did not verify) header.d=gmail.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning gmail.com discourages use of 18.104.22.168 as permitted sender) X-MS-Exchange-Organization-SCL: 5
For message headers with X-Forefront-Antispam-Report, refer to http://pastebin.com/sgjQETzM
Note: 22.214.171.124 is the public IP address of the on-premises hybrid Exchange 2010 server connector to Exchange Online.
How do we stop external emails from being marked as junk by EOP during coexistence stage of a Hybrid Deployment?
This issue was fixed by the Office 365 support (the escalated/backend team) as it has nothing to do with our settings.
We were suggested the follows:
- Whitelist the public IP in EOP Allow List (Tried. It did not help.)
- Add SPF record for our domain: "v=spf1 ip4:XXX.XXX.XXX.XXX ip4:YYY.YYY.YYY.YYY include:spf.protection.outlook.com -all" (Don't think this suggestion is valid as EOP should not check gmail.com against our SMTP IP address as it is not specified in the SPF records of gmail.com. That does not seem the way SPF works.)
- Make sure TLS is enabled (See below)
The key part is the third point. "If the TLS is not enabled, incoming email from local Exchange will not be marked as internal/trust email, and EOP will check all records," said the support.
The support determined a TLS issue from our mail headers by the below line:
- X-MS-Exchange-Organization-AuthAs: Anonymous
This indicates TLS was not enabled when EOP received email. EOP did not treat the incoming email as trust email. The correct one should be like:
- X-MS-Exchange-Organization-AuthAs: Internal
However, this was not caused by our settings; the support person helped us make sure our settings were correct by verifying the verbose SMTP logs from our Exchange 2010 Hybrid server.
At around the same time, their backend team fixed the problem without letting us know what exactly caused it (unfortunately).
After they fixed it, we found that the message headers had some significant changes as below.
For internal-originated mail from Exchange 2003 to Office 365:
X-MS-Exchange-Organization-AuthAs: Internal (It was "Anonymous")
SCL=-1 (It was SCL=5)
- Received-SPF: SoftFail (It was the same)
And for external mails (e.g. gmail.com) to Office 365:
X-MS-Exchange-Organization-AuthAs: Anonymous (It was the same)
SCL=1 (It was SCL=5)
Received-SPF: SoftFail (It was the same)
Although SPF check still soft-fails for gmail.com (external) to Office 365, the support person said it was OK, and all mails would go to the Inbox instead of Junk folder.
As a side note, during troubleshooting, the backend team found one seemingly minor configuration issue -- we had the IP from our Inbound Connector (i.e. public IP of Exchange 2010 Hybrid server) defined in our IP Allow List (suggested by another Office 365 support person as a troubleshooting step). They let us know we should not need to do this and in fact doing so can cause routing issues. They commented that on initial pass the email were not getting marked as spam so there was also a possible issue here. We then removed the IP from the IP Allow List. (However, the spam issue existed before the IP Allow List setting was made. We didn't think Allow List was the cause.)
In conclusion, "it should be EOP mechanism," said the support person. Therefore, the whole thing should be caused by their mechanism.
For anyone interested, the troubleshooting thread with one of their support persons can be viewed here: https://community.office365.com/en-us/f/156/t/403396