Windows 2008 R2 IPsec encryption in tunnel mode, hosts in same subnet

Question

In Windows there appear to be two ways to set up IPsec:

1. The IP Security Policy Management MMC snap-in (part of secpol.msc, introduced in Windows 2000).
2. The Windows Firewall with Advanced Security MMC snap-in (wf.msc, introduced in Windows 2008/Vista).

My question concerns #2 – I already figured out what I need to know for #1. (But I want to use the ‘new’ snap-in for its improved encryption capabilities.)

---

I have two Windows Server 2008 R2 computers in the same domain (domain members), on the same subnet:

server2  172.16.11.20
server3  172.16.11.30

My goal is to encrypt all communication between these two machines using IPsec in tunnel mode, so that the protocol stack is:

• IP
• ESP
• IP
• …etc.

First, on each computer, I created a Connection Security Rule:

• Endpoint 1: (local IP address), eg 172.16.11.20 for server2
• Endpoint 2: (remote IP address), eg 172.16.11.30
• Protocol: Any
• Authentication: Require inbound and outbound, Computer (Kerberos V5)
• IPsec tunnel:
  • Exempt IPsec protected connections
  • Local tunnel endpoint: Any
  • Remote tunnel endpoint: (remote IP address), eg 172.16.11.30

At this point, I can ping each machine, and Wireshark shows me the protocol stack; however, nothing is encrypted (which is expected at this point). I know that it's unencrypted because Wireshark can decode it (using the setting Attempt to detect/decode NULL encrypted ESP payloads) and the Monitor > Security Associations > Quick Mode display shows ESP Encryption: None.

Then on each server, I created Inbound and Outbound Rules:

• Protocol: Any
• Local IP addresses: (local IP address), eg 172.16.11.20
• Remote IP addresses: (remote IP address), eg 172.16.11.30
• Action: Allow the connection if it is secure
  • Require the connections to be encrypted

The problem: Though I create the Inbound and Outbound Rules on each server to enable encryption, the data is still going over the wire (wrapped in ESP) with NULL encryption. (You can see this in Wireshark.)

When the arrives at the receiving end, it's rejected (presumably because it's unencrypted). [And, disabling the Inbound rule on the receiving end causes it to lock up and/or bluescreen – fun!] The Windows Firewall log says, eg:

2014-05-30 22:26:28 DROP ICMP 172.16.11.20 172.16.11.30 - - 60 - - - - 8 0 - RECEIVE

I've tried varying a few things:

• In the Rules, setting the local IP address to Any
• Toggling the Exempt IPsec protected connections setting
• Disabling rules (eg disabling one or both sets of Inbound or Outbound rules)
• Changing the protocol (eg to just TCP)

But realistically there aren't that many knobs to turn.

Does anyone have any ideas? Has anyone tried to set up tunnel mode between two hosts using Windows Firewall?

I've successfully got it set up in transport mode (ie no tunnel) using exactly the same set of rules, so I'm a bit surprised that it didn't Just Work™ with the tunnel added.

Answer 1

After a lot of investigation, and a support case with Microsoft, I have found an answer.

The trick is: don't create any Inbound or Outbound Rules for encryption. Only create a Connection Security Rule (for the tunnel). Then, set the IPsec defaults for the firewall to encrypt every IPsec-enabled connection.

Do the following on each end of the tunnel:

1. Create a Connection Security Rule:

   • Endpoint 1: (local IP address), eg 172.16.11.20
   • Endpoint 2: (remote IP address), eg 172.16.11.30
   • Protocol: Any
   • Authentication: Require inbound and outbound, Computer (Kerberos V5)
   • IPsec tunnel:
     • Exempt IPsec protected connections
     • Local tunnel endpoint: Any
     • Remote tunnel endpoint: (remote IP address), eg 172.16.11.30

2. Force all IPsec connections to encrypt:

   1. Open the main firewall properties (right click Windows Firewall with Advanced Security > Properties…)
   2. In the IPsec Settings tab, under IPsec defaults, click Customize…
   3. Under Data protection (Quick Mode), select Advanced, then click Customize…
   4. Check the box for Require encryption for all connection security rules that use these settings.
   5. Adjust any other settings (eg you may want to remove 3DES as a protocol), then OK your way out.

If you've previously created any Inbound/Outbound Rules for encryption, disable or delete them.

This works well. Its only disadvantage is that it forces encryption on every IPsec connection; you can no longer have a mix of encrypted and integrity-protected-only connections.

---

How do you know the traffic is really tunnelled (ie ESP is carrying IP payload instead of, eg, TCP)? You can verify this with the old IPsec MMC (IP Security Policy Management, or secpol.msc).

• On one server, create a tunnel using the above instructions for WFAS.
• On another server, create a tunnel using the "old" IPsec MMC.
• These two should communicate without any trouble.
• If you switch the "old" IPsec policy to transport mode (ie remove the tunnel), the connection will break. That is how you can tell the WFAS connection is really tunnelling.