I successfully installed IPSec Xauth RSA using this instruction: http://jsharkey.org/blog/2012/09/22/deploying-a-pure-ipsec-pki-vpn-server-for-android-devices/

In brief:

apt-get install ipsec-tools racoon
chmod 700 /etc/racoon/certs
cd /etc/racoon/certs
openssl req -new -x509 -extensions v3_ca -out myca.crt -keyout myca.key -days 3650
openssl req -new -keyout myserver.key -out myserver.csr -days 3650
openssl x509 -req -in myserver.csr -CA myca.crt -CAkey myca.key -CAcreateserial -out myserver.crt
chmod 600 myserver.key
openssl rsa -in myserver.key -out myserver.key
openssl req -new -keyout myphone.key -out myphone.csr -days 3650
openssl x509 -req -in myphone.csr -CA myca.crt -CAkey myca.key -CAcreateserial -out myphone.crt
openssl pkcs12 -export -in myphone.crt -inkey myphone.key -certfile myca.crt -name myphone -out myphone.p12

and then in racoon.conf:

path certificate "/etc/racoon/certs";

timer {
    # NOTE: varies between carriers
    natt_keepalive 45 sec;

listen {

remote anonymous {
    exchange_mode aggressive,main;
    my_identifier asn1dn;

    certificate_type x509 "myserver.crt" "myserver.key";
    ca_type x509 "myca.crt";
    peers_certfile x509 "myphone.crt";

    passive on;
    proposal_check strict;
    generate_policy on;
    nat_traversal force;

    proposal {
        encryption_algorithm aes256;
        hash_algorithm sha1;
        authentication_method xauth_rsa_server;
        dh_group modp1024;

sainfo anonymous {
    encryption_algorithm aes256;
    authentication_algorithm hmac_sha1;
    compression_algorithm deflate;

log info;

mode_cfg {
    auth_source system;
    conf_source local;
    accounting system;

Android is connected to the VPN with the following settings:

Type: IPSec Xauth RSA
IPSec user certificate: myphone
IPSec CA certificate: myphone
IPSec server certificate: (received from server)

And it connects successfully. I can browse sites (some of them does not load and on some of them I have delays but this is ok for now), use other applications that requires connection so it works.

Unfortunately, I cannot figure out how to connect my macbook to this VPN.

I've imported certificates to the system keychain, created Cisco IPSec VPN, selected the certificate myphone as the machine certificate, set the user/pwd. After clicking Connect it displays "Could not validate the server certificate".

In the syslog on the server:

Apr 19 19:12:50 playground racoon: INFO: Adding remote and local NAT-D payloads.
Apr 19 19:12:51 playground racoon: INFO: NAT-T: ports changed to:[4501]<->[4500]
Apr 19 19:12:51 playground racoon: INFO: KA found:[4500]->[4501] (in_use=7)
Apr 19 19:12:51 playground racoon: INFO: Sending Xauth request
Apr 19 19:12:51 playground racoon: [] INFO: received INITIAL-CONTACT
Apr 19 19:12:51 playground racoon: INFO: ISAKMP-SA established[4500]-[4501] spi:72cc05a48011e3e6:9b2eef1f1823779b
Apr 19 19:12:51 playground racoon: ERROR: ignore information because the message is too short - 76 byte(s).

If I change the IPSec server certificate on the Android device, it displays the similar error message (too short) so my guess that I need to either include server certificate in the android settings or set the similar options in Mac OS VPN settings (maybe in config files?).

Alternatively, will be happy if you let me know about some working solution for setting up a Debian VPN server that support Android Always-On VPN connection and Mac OS X / iPhone On-demand VPN.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.