Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We're renting a number of hosts in a public datacenter. The datacenter does not offer private VLANs; all hosts receive one (or more) public IPv4/IPv6 addresses. The hosts come with very modern CPUs (Haswell quad-core, 3.4GHz) and have Gbit uplinks. The different areas (rooms? floors? buildings?) of the datacenter are interconnected with - from what I can tell - Gbit or 500Mbit links. Our hosts are running debian wheezy. Currently we're running just above 10 hosts, with the expectation of growth in the near future.

I am looking for a way to get all hosts to communicate with each other, securely and confidentially. Layer 3 is fine, layer 2 ok (but not necessary). Since I don't have access to VLANs, it'll have to be a VPN of some sort.

What is important to me:

  1. high throughput, ideally close to wirespeed
  2. decentralized, meshed architecture - this is to make sure that throughput is not slowed down by a central element (e.g. VPN concentrator)
  3. CPU footprint is not excessive (given AESNI and GCM-cipher suites, I'm hoping this is not a ridiculous requirement)
  4. operational ease of use; not too complicated to setup; network can grow without loosing established connections

We're currently using tinc. It ticks [2] and [4], but I only reach about 600Mbit/s (simplex) of a 960Mbit/s wirespeed, and I loose one core completely. Also, tinc 1.1 - currently in development - is not yet multithreaded, so I'm stuck with singlecore performance.

Traditional IPSec is out of the question, since it requires a central element, or a sh*tload of tunnels to be configured (to achieve [2]). IPsec with opportunistic encryption would be a solution, but I'm not sure it ever made it into stable production code.

I've stumbled across tcpcrypt today. Except for the missing authentication, it looks like what I want. Userspace implementation smells slow, but so are all the other VPNs as well. And they speak of a kernel implementation. I have not tried it yet, and am interested how it behaves re [1] and [3].

What other options are there? What are people doing, who are not on AWS?

Additional Info

I'm interested in GCM, hoping that it will reduce the CPU footprint. See Intel's paper on the topic. When talking to one of the tinc developers, he explained that even using AESNI for the encryption, the HMAC (e.g. SHA-1) is still very expensive at Gbit speed.

Final Update

IPsec in transport mode works perfectly and does exactly what I want. After much evaluation I have chosen Openswan over ipsec-tools, simply because it supports AES-GCM. On the Haswell CPUs I measure about 910-920Mbit/sec simplex throughput with about 8-9% CPU load of one kworkerd.

share|improve this question
SO, low end equiment? What performance you expect to keep after doing gigabit level encryption? No way - I suggest you look for a more professional host or kill the encryption part. – TomTom Apr 16 '14 at 18:40
@tomtom according to Intel's cryptoperformance paper the encryption performance for AES-128-CBC is at 4.52 cycles per byte, meaning that 100 MB/s would eat up ~450 MHz of a single core. Decryption is considerably less costly. So unless implementation-specific issues are dragging performance down, it should work out for loads which do not need maximum CPU performance and maximum network performance at the same time. – the-wabbit Apr 17 '14 at 9:26
why would you want GCM? IPSEC is not susceptible to the attacks of TLS, so considering using GCM to work around weaknesses of TLS in CBC-modes is a moot point. – the-wabbit Apr 19 '14 at 6:52
up vote 7 down vote accepted

What you don't want is a VPN. What you do want is indeed IPsec, but not in tunnel mode. Rather, you want IPsec in transport mode.

In this configuration, each host communicates directly to its peer, and only packet payloads are encrypted, leaving IP headers in place. This way, you don't need to do any routing gymnastics to get things working.

Yes, you'll need an IPsec connection stanza for each host (unless your hosts are grouped in a subnet, in which case you can do this via a CIDR block), but those can easily be generated programmatically by your configuration management system.

You didn't ask about configuration details, but if you need some pointers (there's not all that much solid information out there on transport mode), you can refer to this blog post I wrote up recently.

share|improve this answer
I second the idea to use IPSEC transport mode for this. Using PSK with OpenSWAN however might not be the best of all setups. Linux already comes with a native IPSEC implementation and a keying daemon (racoon), I would stick to that unless there is a specific requirement not covered by KAME/racoon. – the-wabbit Apr 17 '14 at 9:38
Thanks so much! Combining your advice, I used the native IPsec implementation with racoon. I tested on small single-core machines first, and already throughput increased by 50% and latency dropped to about 60%. I'll confirm it on the Haswell nodes next week and will accept the answer then. I need to figure out though if AES-GCM is supported in the kernel, and how to signal it to IPsec. – Hank Apr 17 '14 at 22:25
@syneticon-dj - just curious...why not openswan? It still uses the kernel's ip xfrm bits for IPsec, but uses pluto as its user-space IKE daemon instead of racoon. I'm not advocating that openswan is the best out there - it's just all I've used and if there are better options, I want to move that direction. – EEAA Apr 18 '14 at 16:48
It probably boils down to personal preference, but I always had my trouble with the inelegance of Free-/OpenSWAN config files and the utterly ugly routing implementation. I very much like the dynamic part of racoonctl which very much resembles what commercial routers allow for in their IPSEC controls. KAME feels more thoroughly engineered while OpenSWAN rather feels patched together. – the-wabbit Apr 19 '14 at 7:14
@syneticon-dj, would you care to elaborate on this? Do you mean you can route several networks through an IPSec link without the need for several SA configurations, as now stands with Openswan? – rsuarez Apr 22 '14 at 16:56

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.