Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to connect a Win Server 2k8 R2 system to a remote machine via IPSec. For this, I got the following settings from the other party:

Phase 1: PSK, AES256, SHA-1 Group 2
Phase 2: AES256, SHA-1

Encryption Domain: xxx.yyy.zzz.0/24 <-> aaa.bbb.ccc.ddd

To test the IPSec connection use xxx.yyy.zzz.104:22 (telnet to that Port).

Where xxx.yyy.zzz are the first three bytes of the remote IP address and aaa.bbb.ccc.ddd represent the IP address of our 2k8 R2 server.

I tried to set the IPSec connection up in Windows Firewall with Advanced Security, but choosing PSK for the first authentication method disables the use of the second authentication method. I also tried to create a VPN of type Layer 2 Tunneling Protocol with IPSec (L2TP/IPSec) the "conventional" way for which I can provide the PSK. But then I cannot figure out how to set up all the other settings like AES, SHA, IP addresses, second authentication method, etc. at all.

I've never set up an IPSec connection before, so I'm totally new to this field. Detailled instructions or maybe some hints whether I'm searching in the right direction would very much help me.

I know that a similar question can be found here: Can I use Win 2k8 R2 as an IPSEC client?. But I couldn't figure out how to solve the problem based on the answer provided there.

share|improve this question

It's probably worth noting that by the looks of it Win2k8 only allows the combination of IPSec over L2TP, but that is not the usually used standard in other VPN devices. More often than not IPSec is put on top of IKE. From the details provided by the other side, I would suspect that that is what they are suggesting.
Further to that I remember dimly (although this may have changed over the years) that the L2TP implementation by Microsoft did not conform 100% to the standard, and therefore would only really talk to other Windows hosts.

I would personally suggest to contact the other party to get more detail from them, and then probably look into using OpenVPN, which is a lot more configurable than the VPN client in Win2k8.

share|improve this answer
Thanks for your hints, Wolfgang. That's typical of Microsoft, I guess. Unfortunately, I'm bound to Microsoft. What's even worse: I have to do with a virtual server environment. This means that, for example, OpenVPN wouldn't work within such environment. I also tried TheGreenBow IPSec VPN Client without any success. It cannot even be installed properly. So I have to try something else. – Flinsch May 30 '11 at 11:56
Well, why don't you contact the other side and see whether they can offer a IPSec over L2TP connection? – wolfgangsz May 31 '11 at 17:54
Meanwhile, I believe that it's not just me: Not only that the other party gave us wrong IP addresses in the first place. I'm wondering whether they also had set up "wrong" configurations or something. However, we let it be for now and use SFTP... ^^ But, many thanks again, Wolfgang. – Flinsch Jun 6 '11 at 6:47

See if this article helps: Overview of IPSec Rules in Windows Firewall with Advanced Security

share|improve this answer
It has not solved the problem, but nevertheless offered some useful information. Many thanks so far. – Flinsch May 23 '11 at 13:30
With the additional info is there a more specific problem you are having? – Jim B May 23 '11 at 19:39
Yes, I figured out, that the first and second authentication method are not (necessarily?) linked to the phases 1 and 2 (main mode and quick mode). It also turned out that the remote party gave me wrong IP addresses. So, until now, I've no information whether I'm on the right path. But I've already come much further. A more specific question to a new problem will arise very soon hopefully, unless the problem solves "itself". Thanks again, Jim. – Flinsch May 25 '11 at 6:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.