I've got some weird problem and I'm out off options. The situation:
When connecting from my mac to the VPN server (Windows Server 2003 R2) with L2TP PSK, everything works like it should.
However, when I connect from a Windows PC, nothing happens. it spits out error 809 and sometimes 789. Now I know that my ports are OK, since the mac can connect without any problems.
It's the same for: XP, Vista SP2 and 7. None can connect. If I connect to the VPN server directly (to the internal IP instead of WAN from the router), it connect's without a problem. Connecting using PPTP works... now if only L2TP would work thank you very much Windows!
I have checked the counters on my linux router with iptables -L -nv and they do not raise when connecting. Not on ACCEPT and not on DROP. Only when connecting from the mac.
I've found the guide from Microsoft to enable: AssumeUDPEncapsulationContextOnSendRule in the registery. I have set it to "2", on the server and client. Still no go. After that registery key it started giving me error 789 instead of 809. The IPSEC services are running on the client and server.
Is there anyone that ppleease can help me with this! I've been working on this for 2 days and I'm out of options.
//edit: see solution below. This is ONLY for windows clients trying to connect to a VPN server behind a NAT router!