We're renting a number of hosts in a public datacenter. The datacenter does not offer private VLANs; all hosts receive one (or more) public IPv4/IPv6 addresses. The hosts come with very modern CPUs (Haswell quad-core, 3.4GHz) and have Gbit uplinks. The different areas (rooms? floors? buildings?) of the datacenter are interconnected with - from what I can tell - Gbit or 500Mbit links. Our hosts are running debian wheezy. Currently we're running just above 10 hosts, with the expectation of growth in the near future.
I am looking for a way to get all hosts to communicate with each other, securely and confidentially. Layer 3 is fine, layer 2 ok (but not necessary). Since I don't have access to VLANs, it'll have to be a VPN of some sort.
What is important to me:
- high throughput, ideally close to wirespeed
- decentralized, meshed architecture - this is to make sure that throughput is not slowed down by a central element (e.g. VPN concentrator)
- CPU footprint is not excessive (given AESNI and GCM-cipher suites, I'm hoping this is not a ridiculous requirement)
- operational ease of use; not too complicated to setup; network can grow without loosing established connections
We're currently using tinc. It ticks  and , but I only reach about 600Mbit/s (simplex) of a 960Mbit/s wirespeed, and I loose one core completely. Also, tinc 1.1 - currently in development - is not yet multithreaded, so I'm stuck with singlecore performance.
Traditional IPSec is out of the question, since it requires a central element, or a sh*tload of tunnels to be configured (to achieve ). IPsec with opportunistic encryption would be a solution, but I'm not sure it ever made it into stable production code.
I've stumbled across tcpcrypt today. Except for the missing authentication, it looks like what I want. Userspace implementation smells slow, but so are all the other VPNs as well. And they speak of a kernel implementation. I have not tried it yet, and am interested how it behaves re  and .
What other options are there? What are people doing, who are not on AWS?
I'm interested in GCM, hoping that it will reduce the CPU footprint. See Intel's paper on the topic. When talking to one of the tinc developers, he explained that even using AESNI for the encryption, the HMAC (e.g. SHA-1) is still very expensive at Gbit speed.
IPsec in transport mode works perfectly and does exactly what I want. After much evaluation I have chosen Openswan over ipsec-tools, simply because it supports AES-GCM. On the Haswell CPUs I measure about 910-920Mbit/sec simplex throughput with about 8-9% CPU load of one