4

We have 100+ embedded boards that we need to talk to occasionally.

We've found the best way to do this reliably (there are some strange internet connections out there) is to have the boards open a few reverse ssh tunnels to our server.

Currently each board opens three tunnels, for a total of around 300. The tunnels are inactive most of the time .

Are there any performance implications of this? Assuming we don't run out of ports or file descriptors, what is most likely to trip us up?

Edit: I'm primarily concerned with having several hundred inactive clients connected at once. Only a few of the tunnels will be in active use at any one time.

  • I'm curious about the need to open 3 tunnels per machine. Care to enlighten me? – John Gardeniers Feb 8 '12 at 20:41
  • One tunnel is for ssh, the other two are for services running on the clients that we need to connect to. Currently we pull, rather than push data. – Johan Feb 9 '12 at 10:12
3

The limit on active SSH connections is 65,534 minus the number of TCP ports in use on your system -- If we want to be conservative let's just say 60,000 (or for most practical purposes "Unlimited"). You may bump up against other system resource limits however, depending on the machine you're connecting to.

That said you might want to economize on the number of tunnels (do you really need three connections per board?), and if you don't always need the connections up you might want to have the remote site initiate them only when needed.

  • It's the other system resource limits I'm interested in. What might they be? – Johan Feb 9 '12 at 10:14
  • It's impossible to enumerate every system resource that can be exhausted - This is like asking for an exhaustive list of why a computer won't boot (the hardware fault tree alone would be a book). A few of them in rough order of likely exhaustion: max number of logins per user (if set), max number of logged-in users system-wide, max number of open filehandles, RAM, network bandwidth, CPU – voretaq7 Feb 9 '12 at 16:19
  • I know - but first on the list are things like 'Is it plugged in?' I didn't know about any logged-in user limit, so if I check that's ok I think we'll be good for a while. Thanks. – Johan Feb 15 '12 at 10:09
1

TL;DR: memory. Rest is okay (for hundreds of tunnels).

Assuming we don't run out of ports or file descriptors, what is most likely to trip us up?

I think it is safe enough to answer: memory.

A quick experiment shows 4.6 megabytes of actual memory consumed per incoming connection. Part of it may be shared, a real experiment would tell. Fix: add swap space.

Also, 86 megabytes of addressing space which is quite big. It's very probably shared, or a 32-bit server would have addressing space exhaustion before 50 connections. Fix: use a 64-bit OS.

Assuming we don't run out of ports or file descriptors,

Does this assumption hold true?

  • At the range of hundreds, ports won't be a problem, unless your server is also a high traffic server for something else.

  • File descriptors might get exhausted. A quick experiment shows 21 file descriptors consumed per incoming ssh connection, not counting a shell. The limit on file descriptors can be tuned system-wide, so it should not be a real blocker.

(Experiments done on a 64-bit Debian server.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.