Some SSH tunnels reporting connection refused between Linux VM and OSX Lion

Question

I'm trying to connect to a set of services running on a Centos 6.3 VM guest (VMware Fusion 4) from a Mac OS X Lion host. I've setup .ssh/config script in /var/root:

Host testbed
Hostname testbed
User <username>
Port 15922
# web server
LocalForward localhost:80 testbed:80
# zend server admin console
LocalForward localhost:10081 testbed:10081
# zend debugger
LocalForward localhost:10000 testbed:10000
# mongodb
LocalForward localhost:28017 testbed:28017
# couchdb
LocalForward localhost:10080 testbed:5984
# scrapyd
LocalForward localhost:6800 testbed:6800
# eclipse connection for zend debugger
RemoteForward testbed:10137 localhost:10137

I then run sudo ssh testbed to connect the tunnels. Sudo is required to tunnel port 80, hence root.

Some of the tunnels work fine. I can view localhost:80, localhost:6800 and localhost:10081 in Firefox. However some of the tunnels don't work. They timeout in the browser and produce this error in the shell:

channel 19: open failed: connect failed: Connection refused

I thought initially this was an artefact of the port number, so for 5984, I tunnelled it back to localhost:10080. The behaviour was the same (timeout + error). I have verified all the services are running correctly using a text-only browser (lynx) in the shell.

I did wonder if this was a firewall issue. The Guest VM has a firewall, but it opens happily for outgoing connections. The host OS firewall is 'Off' (System Preferences -> Security -> Firewall).

netstat doesn't show any collisions on the ports

tcp6       0      0  ::1.6800               *.*                    LISTEN     
tcp4       0      0  127.0.0.1.6800         *.*                    LISTEN     
tcp6       0      0  ::1.10080              *.*                    LISTEN     
tcp4       0      0  127.0.0.1.10080        *.*                    LISTEN     
tcp6       0      0  ::1.28017              *.*                    LISTEN     
tcp4       0      0  127.0.0.1.28017        *.*                    LISTEN     
tcp6       0      0  ::1.10000              *.*                    LISTEN     
tcp4       0      0  127.0.0.1.10000        *.*                    LISTEN     
tcp6       0      0  ::1.10081              *.*                    LISTEN     
tcp4       0      0  127.0.0.1.10081        *.*                    LISTEN     
tcp6       0      0  ::1.80                 *.*                    LISTEN     
tcp4       0      0  127.0.0.1.80           *.*                    LISTEN     

I've tested VM Fusion in two networking configs (NAT and Bridged) and both exhibit the same behaviour.

Is it possible that:

1. there's an IPTABLES firewall on the host OS (no iptables command in Mac OS X)?
2. there's some kind of protection in VMware Fusion that locks out certain ports from guest to host?
3. there's protection built into the linux services, so that some localhost only services won't be tunnelled?

Any kind of guidance appreciated. For completeness, here's a log of the SSH transaction using the verbose option:

OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /var/root/.ssh/config
debug1: Applying options for testbed
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: Connecting to testbed [XX.XX.XX.XXX] port 15922.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /var/root/.ssh/id_rsa type -1
debug1: identity file /var/root/.ssh/id_rsa-cert type -1
debug1: identity file /var/root/.ssh/id_dsa type -1
debug1: identity file /var/root/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '[testbed]:15922' is known and matches the RSA host key.
debug1: Found key in /var/root/.ssh/known_hosts:7
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /path/to/my/private-key.ppk
debug1: Server accepts key: pkalg ssh-dss blen 817
debug1: Authentication succeeded (publickey).
Authenticated to testbed ([10.12.1.154]:15922).
debug1: Local connections to localhost:80 forwarded to remote address testbed:80
debug1: Local forwarding listening on 127.0.0.1 port 80.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on ::1 port 80.
debug1: channel 1: new [port listener]
debug1: Local connections to localhost:10081 forwarded to remote address testbed:10081
debug1: Local forwarding listening on 127.0.0.1 port 10081.
debug1: channel 2: new [port listener]
debug1: Local forwarding listening on ::1 port 10081.
debug1: channel 3: new [port listener]
debug1: Local connections to localhost:10000 forwarded to remote address testbed:10000
debug1: Local forwarding listening on 127.0.0.1 port 10000.
debug1: channel 4: new [port listener]
debug1: Local forwarding listening on ::1 port 10000.
debug1: channel 5: new [port listener]
debug1: Local connections to localhost:28017 forwarded to remote address testbed:28017
debug1: Local forwarding listening on 127.0.0.1 port 28017.
debug1: channel 6: new [port listener]
debug1: Local forwarding listening on ::1 port 28017.
debug1: channel 7: new [port listener]
debug1: Local connections to localhost:10080 forwarded to remote address testbed:5984
debug1: Local forwarding listening on 127.0.0.1 port 10080.
debug1: channel 8: new [port listener]
debug1: Local forwarding listening on ::1 port 10080.
debug1: channel 9: new [port listener]
debug1: Local connections to localhost:6800 forwarded to remote address testbed:6800
debug1: Local forwarding listening on 127.0.0.1 port 6800.
debug1: channel 10: new [port listener]
debug1: Local forwarding listening on ::1 port 6800.
debug1: channel 11: new [port listener]
debug1: Remote connections from testbed:10137 forwarded to local address localhost:10137
debug1: channel 12: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: remote forward success for: listen 10137, connect localhost:10137
debug1: All remote forwarding requests processed
debug1: Sending environment.
debug1: Sending env LANG = en_GB.UTF-8
Last login: Mon Sep  3 12:27:10 2012 from something.somenet
[me@testbed ~]$ debug1: Connection to port 10080 forwarding to testbed port 5984 requested.
debug1: channel 13: new [direct-tcpip]
channel 13: open failed: connect failed: Connection refused
debug1: channel 13: free: direct-tcpip: listening port 10080 for testbed port 5984, connect from 127.0.0.1 port 54639, nchannels 14
debug1: Connection to port 10080 forwarding to testbed port 5984 requested.
debug1: channel 13: new [direct-tcpip]
channel 13: open failed: connect failed: Connection refused
debug1: channel 13: free: direct-tcpip: listening port 10080 for testbed port 5984, connect from 127.0.0.1 port 54640, nchannels 14

Answer 1

Are some of the services bound to localhost on the guest? If you do an ssh forward from (host) localhost:8080 to (guest) 10.x.y.z:80 and the service is listening on (guest) 127.0.0.1:80 , it will give you connection refused.

Try making a tunnel from 127.0.0.1:port to 127.0.0.1:port , where the first is host and the second is guest.