Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am evaluating a system for a client where many OpenVPN clients connect to a OpenVPN server. "Many" means 50000 - 1000000.

Why do I do that? The clients are distributed embedded systems, each sitting behind the system owners dsl router. The server needs to be able to send commands to the clients. My first naive approach is to make the clients connect to the server via an openvpn network. This way, the secure communication tunnel can be used in both directions.

This means that all clients are always connected to the server. There are many clients summing up over the years.

The question is: does the OpenVPN server explode when reaching a certain number of clients? I am already aware of a maximum TCP connection number limit, therefore (and for other reasons) the VPN would have to use UDP transport.

OpenVPN gurus, what is your opinion?

share|improve this question
    
Could you share your final conclusions about this with us? Have you been able to do tests with > 5'000 users? –  Philipp May 13 '13 at 16:21
    
Hello Philipp, we ditched the OpenVPN plan as it was clear that we would touch ground nobody has ever touched before. We opted to a SSL based normal TCP Socket connection to a Node.js connection management server. –  Steffen Müller May 14 '13 at 16:48

3 Answers 3

up vote 12 down vote accepted

I doubt that a setup that large has ever been attempted before, so you likely will be pushing limits when trying. I could find an article on a VPN deployment for 400 clients but judging from the text, the author just relied on rough estimates about how many clients could be run per CPU and lacked some understanding about how his setup would perform.

You would mainly need to consider these two points:

  1. The bandwidth your data transfers are going to use would need encryption / decryption at the VPN server side, consuming CPU resources

  2. OpenVPN client connections consume both, memory and CPU resources on the server even when no data is transferred

Any decent PC hardware available today should easily saturate a Gigabit link with Blowfish or AES-128, even $100 embedded devices are capable of rates near 100 Mbps, so CPU bottlenecks due to bandwidth intensity should not be of any concern.

Given the default rekeying interval of 3600 seconds, a number of 1,000,000 clients would mean that the server would need to be able to complete 278 key exchanges per second on average. While a key exchange is a rather CPU-intensive task, you could offload it to dedicated hardware if needed - cryptographic accelerator cards available easily meet and exceed this number of TLS handshakes. And memory restrictions should not bother too much as well - a 64-bit binary should take care of any virtual memory restrictions you would be likely to hit otherwise.

But the real beauty with OpenVPN is that you can scale it out quite easily - simply set up an arbitrary number of OpenVPN servers and make sure your clients are using them (e.g. through DNS round-robin), configure a dynamic routing protocol of your choice (typically this would be RIP due to its simplicity) and your infrastructure would be capable of supporting an arbitrary number of clients as long as you've got enough hardware.

share|improve this answer
    
Thanks for the concise answer. Do you see alternatives to using openvpn? The main goal is just having the bidirectional communication going through the router. –  Steffen Müller Oct 19 '12 at 9:13
2  
@SteffenMüller If you do not need a complete stack but only a control channel, why not use something similar to botnets? Implementations are available and the SANS conveniently offers a paper on how to set them up –  the-wabbit Oct 19 '12 at 10:47
    
Thanks for the interesting link. Unfortunately, the bot is using simple polling to query whether the server has informations. Although this might be the way to go, I am searching for a way to establish and keep a bidirectional connection. The constant polling either causes delays in command execution or a high data volume for useless polling requests. Maybe a permanent TCP connection is the way to go? –  Steffen Müller Oct 20 '12 at 11:52
    
@SteffenMüller Botnets are proven to handle thousands of clients well - thus my suggestion to look into it. You do not have to go with the specific implementation hinted at by SANS - there really are plenty of others. Other than that, without knowing your exact requirements it is really difficult to tell. A TCP connection sending keepalives surely would be able to make sure the state relation at the NAT gateway does not age out. But you would need to take care of everything else (authentication, encryption, error handling) all by yourself. –  the-wabbit Oct 21 '12 at 9:38
2  
BTW, there isn't any reason you can't lower the rekeying interval (there is a security tradeoff, in that a key being compromised will reveal plaintext back to the last rekeying). Also, I'd be much more worried about the routing or other connection-look-up failing first. I mean, if OpenVPN is intended to have <100 connections active, what's the chance there there is an O(n) lookup of a connection somewhere? –  derobert Oct 22 '12 at 22:30

I'm looking into a similar problem, although the number of clients would be into hundreds maybe a couple of thousands.

I figured that I can't keep all the clients connected all the time.

I'm thinking of starting OpenVPN daemon on clients at randomized time intervals so they can check if they were polled. If they were they are to send an email or something that they are online and send keep alive packets for a period of time so I can connect to them.

If there is no traffic for some time daemon would be stopped.

The problem I'm facing right now is that it seems impossible to get a list of currently connected VPN clients ...

share|improve this answer
    
You can get a current list of connected clients through openvpn status log. There you see all connected ips to the current server. –  Fa11enAngel Jul 2 at 17:26

I have actually done this, albeit with "only" a few hundred remote connections similarly behind DSL routers. I can't comment too much about the rekeying issues, but a few practical things I learned along the way:

1) When deploying clients, make sure you specify multiple VPN servers in the client conf, vpn1.example.com, vpn2.example.com, vpn3..... Even if you only provide one or two of these now, you give yourself headroom. Configured properly, the clients will keep retrying them at random till they find one that works.

2) We use a custom AWS VPN server image, and can spin up additional capacity on demand, and Amazon DNS (R53) handles the DNS side of things. It is completely detached from the rest of our infrastructure.

3) At the server(s) end, make careful use of the netmask to restrict the number of potential clients. That should force clients onto an alternative server, mitigating the CPU issues. I think we limit our servers to 300 or so clients. This choice was somewhat arbitrary on our part - "gut feel" if you like.

4) Also at the server end, you should make careful use of firewalls. In simple terms, we have ours configured such that the clients can VPN connect in, but the servers strictly disallow all ssh connections inbound except from a known IP address. We can SSH to the clients if we occasionally need to, they can't SSH to us.

5) Don't rely on OpenVPN doing the reconnect for you at the client end. 9 times out of 10 it will, but sometimes it gets stuck. Have a separate process to reset/restart openVPN at the client end regularly.

6) You need a way of generating unique keys for the clients so you can disavow them sometimes. We generate these internally with our server build (PXEboot) process. Never happened to us, but we know we can do it.

7) You will need some management tools, scripts to monitor your VPN server connections effectively.

There is not much material out there about how to do this unfortunately, but it is possible, with careful configuration.

share|improve this answer
    
Thank you very much for the insights. I am surprised that the rekeying problems already hit you with 300 clients... –  Steffen Müller Oct 24 '12 at 8:53
    
To clarify - they have not, but I have not been tracking it either.... :-/ The "300" number just seemed reasonable. If we have problems we would just bump up the AWS image to a larger instance. I've never had close to that many connections on a server before, probably only about 100 max, but we run several servers and they approximately balance in line with openvpn randomly choosing a destination from a known list. –  Aitch Oct 24 '12 at 11:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.