Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've some trouble with the IPsec configuration on my debian server (6 squeeze). This server should connect via IPsec VPN to an windows server, which is protected by an firewall. I've used racoon and ipsec-tools and this tutorial

However, I am not quite sure, if this tutorial fits to my purpose, because of some differences:

  • my Host and my gateway are the same server. So I don't have two different ip addresses. I guess, that's not a problem
  • the other server is an windows system behind a firewall. Hopefully, not a problem
  • the subnet of the windows system is /32 not /24. So I change it to /32.

I worked through the tutorial step by step, but I wasn't able to route the ip. The following command didn't work for me:

ip route add to via XXX.XXX.XXX.XXX src XXX.XXX.XXX.XXX

So I tried the following instead:

ip route add to .., which obviously not solved the problem.

The next problem is the compression. The windows doesn't use a compression, but 'compression_algorithm none;' doesn't work with my racoon. So the current value is 'compression_algorithm deflate;'

So my current result looks like this:

When I am trying to ping the windows host (ping, I receive the following error message from ping:

ping: sendmsg: Operation not permitted

And racoon logs:

racoon: ERROR: failed to get sainfo.

After googling for a while I came to no conclusion, what's the solution. Does this error message mean that the first phase of IPsec works?

I am thankful for any advice.

I guess my configs might be helpful.

My racoon.conf looks like this:

path pre_shared_key "/etc/racoon/psk.txt";

remote YYY.YYY.YYY.YYY {

    exchange_mode main;
    proposal {
            lifetime time 8 hour;
            encryption_algorithm 3des;
            hash_algorithm sha1;
            authentication_method pre_shared_key;
            dh_group 2;


sainfo address XXX.XXX.XXX.XXX/32 any address any {

    pfs_group 2;
    lifetime time 8 hour;
    encryption_algorithm aes 256;
    authentication_algorithm hmac_sha1;
    compression_algorithm deflate;


And my ipsec-tools.conf looks like this:



spdadd XXX.XXX.XXX.XXX/32 any -P out ipsec esp/tunnel/XXX.XXX.XXX.XXX-YYY.YYY.YYY.YYY/require;

spdadd XXX.XXX.XXX.XXX/32 any -P in ipsec esp/tunnel/YYY.YYY.YYY.YYY-XXX.XXX.XXX.XXX/require;

If anyone has an advice, that would be awesome.

Thanks in Advance.

Greets, Michael

It was a simple copy-and-paste error in an ip address.

share|improve this question

Wouldn't OpenVPN be more appropriate? OpenVPN is super simple to configure. Here is a sample config and some links to guide your through the certificate creation process.

Just configure the intermediary to be the host, and the guests can dial in and still communicate with each other.

apt-get install openvpn
mkdir /etc/openvpn/easy-rsa
mkdir -p /etc/openvpn/ccd/client_server
touch /etc/openvpn/ipp.txt
cp /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa
source ./vars
./build-key-server server
cd /etc/openvpn/easy-rsa/keys
openssl pkcs12 -export -out server.p12 -inkey server.key -in server.crt -certfile ca.crt

Then create a new file /etc/openvpn/client_server.conf and put the following in it, changing the SERVER_IP_ADDRESS as appropriate

port 8443
proto udp
dev tun
ca /etc/openvpn/easy-rsa/keys/ca.crt
pkcs12 /etc/openvpn/easy-rsa/keys/server.p12
dh /etc/openvpn/easy-rsa/keys/dh2048.pem
ifconfig-pool-persist /etc/openvpn/ipp.txt
client-config-dir /etc/openvpn/ccd/client_server
keepalive 10 120
status /var/log/openvpn-status.log
verb 3
reneg-sec 0

Then build a key per user who is going to connect, and create the config file in the ccd dir

echo "ifconfig-push" > /etc/openvpn/ccd/client_server/

The IP address MUST be suitable for a /30 subnet (see, as there is only 2 addresses available (server and client) per connection. So your next available client IP would be and so on.

Then you now have static IPs per connecting user.

Then supply the file to the end-user and use the following config file

dev tun
proto udp
resolv-retry infinite
ns-cert-type server
verb 3
reneg-sec 0
share|improve this answer
I totally agree with you! However, I am not charge for the windows server. the service provider, which is responsible for the windows server, wants to implement IPsec cause it's their standard. And they don't have a VPN running, which works with openVPN. – Michi Qne Jul 20 '12 at 9:34
The current status is: - the windows system gets a timeout in phase 2 - our debian system is not able to open the tunnel by itself. I thought the tunnel will be opened, when the server tries to connect to the host ip of the windows system. Do I need a special setup, that the tunnel opens automatically? Oh, by the way the windows server is the ipsec server and our debian server is the ipsec client. THX in advance! – Michi Qne Jul 20 '12 at 14:48
I can't give you a direct recommendation with IPSec on Debian - as we never quite got it to behave the way we wanted to. So we established a VPN ring with OpenVPN instead and its been rock-solid since. – Ben Lessani - Sonassi Jul 20 '12 at 15:19
Problem solved! It was a copy-and-paste-error! – Michi Qne Jul 23 '12 at 13:31

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.