Many concurrent Cassandra database connections thru SSH tunnel

Question

I have an application that uses the PyCassa Cassandra client to open many concurrent connections to a Cassandra server. When I run the client application locally on the database server, it works. However, when i forward the Cassandra port thru an ssh tunnel to the database server and connect the client application to the server thru this tunnel, a few connections get thru but generally i get errors like this:

'2012210105:49:49'|WARNING |_append_historyStorage|75  |Exception when appending to CassandraTimeSeries
Traceback (most recent call last):
  File "atr/cassandratimeseries.py", line 140, in append
    cf = self._getColumnFamily(duration, 'main')
  File "atr/cassandratimeseries.py", line 63, in _getColumnFamily
    return ColumnFamily(ConnectionPool(self._keyspace, pool_timeout=100, timeout=15, max_overflow=5), self._columnFamilyName(duration, table_type), **self._column_family_op_options)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 356, in __init__
    self.fill()
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 415, in fill
    conn = self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 403, in _create_connection
    (exc.__class__.__name__, exc))
AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: TSocket read 0 bytes

On a terminal where i was ssh'd into the server is printed a line of messages like:

channel 1034: open failed: administratively prohibited: open failed
channel 1035: open failed: administratively prohibited: open failed
channel 1036: open failed: administratively prohibited: open failed
channel 1037: open failed: administratively prohibited: open failed
channel 1038: open failed: administratively prohibited: open failed
channel 1039: open failed: administratively prohibited: open failed
channel 1040: open failed: administratively prohibited: open failed

Interestingly, if i then attempt to ssh to the server, i get:

channel 1023: chan_read_failed for istate 1
channel 1023: chan_write_failed for ostate 3
Shared connection to xx.xxx.xxx.xxx closed.

If i delete the master socket file in /tmp, then i can ssh to it again.

Any ideas what could be going wrong? I already increased the nofile limit via /etc/security/limits.conf for both the client and the server (to ten thousand) and rebooted. The "open failed: administratively prohibited" suggests that there is some administrative limit being hit. What is it and how do i raise it (and how should i have figured this out for myself?)

Are there other ways in which ssh tunnels are more limited than connecting to a local port?

thanks

Answer 1

Interesting, i haven't come across this before. I'd try what you already did so for next steps i chased down the message "administratively prohibited" in the ssh source, it occurs only in one case and it's raised in sshd.c from there i followed the logic in channels.c and session.c

This seems to be the only code in SSH which would de-restrict the number of channels allowed, there is no hard limit i can see in sshd.c or channels.c

/* setup the channel layer */
if (!no_port_forwarding_flag && options.allow_tcp_forwarding)
    channel_permit_all_opens();

Can you check your sshd_config on the server and ensure the following option is set:

AllowTcpForwarding yes

The thing i don't like about my answer is there does appear to be a 1024 limit somewhere which i would expect to have traced in the code. 1024 is often the default limit for numer of open files in limits.conf - is there anywhere else a ulimit is being set in the shell from which sshd is started?