Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Configuration

I have setup a "raw" IPSec tunnel between a Windows Server 2003 (SBS) machine and a Netgear FVG318 according to the instructions in Microsoft KB816514. The configuration is as follows (using the same conventions as the article):

NetA         | SBS2003   | FVG318   | NetB
10.0.0.0/24  | 216.x.x.x | 69.y.y.y | 10.0.254.0/24

Both the Main Mode and Quick Mode Security Associations are successfully completed and appear in the IP Security Monitor. I am also able to ping the SBS2003 server on its private address from any computer on NetB.

The Problem

Any traffic sent from a computer on NetA to NetB, or from SBS2003 to NetB (excluding ICMP Ping responses), is sent out on the public network interface outside the IPSec tunnel (no encryption or header authentication, as if the tunnel were not there).

Pings sent from a computer on NetB to a computer on NetA successfully reach computers on NetA, but the responses are silently discarded by SBS2003 (they do not go out in the clear and do not generate any encrypted traffic).

Possible Solutions

Incorrect Configuration

I could have mistyped something, somewhere, or KB816514 could be incorrect in some way. I have tried very hard to eliminate the first option. Have re-created the configuration several times, tried tweaking and adjusting all the settings I could without success (most prevent the SA from being established).

NAT/RRAS

I have seen multiple posts elsewhere suggesting that this could be due to interaction between NAT and the IPSec filters. Possibly the NetA private addresses get rewritten to 216.x.x.x before being compared with the Quick Mode IPSec filters and don't get tunneled because of the mismatch. In fact, The Cable Guy article from June 2005 "TCP/IP Packet Processing Paths" suggests that this is the case, (see step 2 and 4 of the Transit Traffic path). If this is the case, is there a way to exclude NetA->NetB traffic from NAT?

Any thoughts, ideas, suggestions, and/or comments are appreciated.

Update (2011-06-26)

After failing to solve the problem, I resorted to paid Microsoft support. They were unable to solve the problem. Since then I have implemented a solution based on Linux that is working quite well. I will attempt to evaluate any proposed answers as best I can, but current configurations and time constraints will make this slow...

share|improve this question
add comment

1 Answer

Check your binding order. Network properties>Advanced>Advanced Settings Move the one you want it to route out via to the top.

share|improve this answer
    
I have tried with each of the interfaces (including [Remote Access connections]) in the top position and the behavior is unchanged. To clarify, the traffic is being sent out on the correct interface, but outside the tunnel (unencrypted to private IP) rather than in the tunnel (encrypted to public IP). –  Kevinoid May 27 '11 at 3:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.