Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My problem - when I create SSL certificate (using selfssl7 or IIS management console), my vpn connection (L2TP with certificate authentication) will not go up (I get error 789). I have to delete this certificate for connection to work again.
I wonder if I do something wrong, here is my selfssl command line:
selfssl7 /Q /T /I /S "site name" /N cn=localhost

I am creating certificate and having troubles on the same machine, which connects to ISP's VPN server. Sorry if my question made you think otherwise.

share|improve this question
are you using UDP or TCP and what port? – ApolloSoftware Sep 11 '11 at 3:04
Sorry for the delay, I couldn't log in for a while. @AmitApollo, I use Windows VPN client, it won't let me change anything, so port must be default one (udp 1701, i guess). – Madao Sep 14 '11 at 8:52

The certificate you install on the server side has to be trusted by the client. Since you generated a self-signed certificate you should copy the certificate to the clients, import them, and mark them as trusted. If your clients are mobile (ie laptops) then when they are at the office you have a GPO to push the certificate automatically; if they are not mobile and not local you can use remote enrollment or offline enrollment. Hope this helps but if not... you should consider reading an overview and client L2TP/IPSEC config before spending much more time playing around.

share|improve this answer
Thank you for response! I've read these articles and everything makes sense, but please explain me: being able (as vpn client) to connect without this self-signed certificate means that (1) I already have a valid certificate (2) new certificate somehow replaces old one or hinders its selection or creates some sort of conflict between 'em, is that right? – Madao Sep 16 '11 at 3:51
Does the client trust your server's certificate? You can verify this by looking in the client's trusted certificate list and looking at the properties of the server certificate if it's there. – Ram Sep 16 '11 at 17:08
Well, I am client, looks like server stops trusting me after selfssl7'ing... Local https sessions are working though, no warnings. – Madao Sep 16 '11 at 17:27
Local https sessions would work because your machine trusts the certificate it generated for itself. You need to have the server trust that certificate as well, or better yet use your client to enroll for a certificate from the server - if the server issues it, the server will trust it. Both of these scenarios are covered in the links I provided above. – Ram Sep 16 '11 at 19:04
Darn, I just don't get it. It is not like I can't connect at all, I CAN, BUT if there is no extra certificate, so server trusts me already. Something must be wrong with the way self signed certificate is created. – Madao Sep 17 '11 at 18:52

As from the error code,

Possible Causes: This is a generic error which is thrown when the IPSec negotiation fails for L2TP/IPSec connections.

Possible causes for this issue could be:

a> L2TP based VPN client (or VPN server) is behind NAT.

b> Wrong certificate or pre-shared key is set on the VPN server or client

c> Machine certificate or trusted root machine certificate is not present on the VPN server.

d> Machine Certificate on VPN Server does not have 'Server Authentication' as the EKU

Possible Solution: Make sure correct certificate is used both on client and server side – for further details refer to this blog. In case Pre Shared Key (PSK) is used, make sure the same PSK is configured on the client and the VPN server machine.


share|improve this answer
So, my self signed certificate somehow invalidates correct one that is normally used to establish a connection. What do I do then? – Madao Sep 14 '11 at 9:09

Disclaimer: I've never really used L2TP myself.

To my understanding, L2TP with Certificate authentication requires a server and client certificate pair. The client certificate must be issued by the server (CA). Usually, this can be achieved with an Active Directory Certificate Authority and Group Policies to import the CA-cert and auto-enroll for client certificates. In any case, NOT by using selfssl7 to create a self-signed (server) certificate.

Did you generate the certificate using a 2048-bit keylength? By default it may use 1024 which may not be accepted by your VPN client. To determine this or discover pointers towards other errors, check the eventlog on both the client and server after initiating a (failed) session!

selfssl7 /Q /T /I /S "site name" /N cn=localhost /K 2048

To make things simpler, is it an option to use SSTP (port 443) or PPTP with user authentication for establishing a VPN?

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.