I am having a strange issue. I have an EC2 server (Arch Linux) that I am able to access (via ssh) from my local linux server without issue, however when I try to ssh into my EC2 server from my macbook, I get a connection refused.
$ ssh -vvv -i key.pem email@example.com OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011 debug1: Reading configuration data /etc/ssh_config debug2: ssh_connect: needpriv 0 debug1: Connecting to myserver.com 184.72.xxx.xx port 22. debug1: connect to address 184.72.xxx.xx port 22: Connection refused ssh: connect to host myserver.com port 22: Connection refused
I also have work clients using EC2 for some of their servers, and I have the exact same issue. I can log into those EC2 machines from other boxes, but not my macbook. I am able to ssh to other servers from my macbook, both locally and out over the network. Which means that while there may be some issue with my macbook, I am still able to ssh into other boxes. I am also able to visit websites that I am serving from my server on my macbook, so the server isn't blacklisted on my macbook, as far as I can tell. This is not the case for all EC2 boxes. I set up a test instance with the same key on my EC2 account, and I was able to ssh into that from my macbook.
Since I am able to connect to my EC2 machine from another box on my local network it rules out that ssh is not running on the server, that the port may be blocked, and that my ip might be blacklisted on the server side. If I run a tcpdump while trying to ssh or nc in from my macbook, I get nothing happening from my local IP address. It seems like the server is not even seeing my attempts. I also see no output in /var/log/auth.log for my macbook's attempts, while other attempts are logged.
I have created a new key on the server and copied the private key back to my macbook (tested elsewhere) and that failed to get me in. I have checked iptables (shut iptables down and tried to connect), /etc/hosts.deny (empty) and the security group, where ssh (port 22) is wide open. On my local network, I have swapped out my router since this issue began, but that didn't help. The issue seems to have happened around when I upgraded my mac to Lion and installed a new hard drive, keeping the same user directory. I am not sure if the problem lives on my mac, or on EC2's end, but I am pretty stuck at this point, and since there are two separate EC2 boxes that I can't seem to get into (one CentOS and one Arch Linux).
I have also tried connecting from my macbook while on another network; same results. I recompiled openssh and installed it in /opt/openssh, tried running it from that location with a couple different keys without luck. I am using ssh-agent, and have tried dropping all keys, and explicitly identifying the key that I am going to use; same results. If this was simply a bad key issue, I should get a permission denied message, or a 'too many attempts' message if it was trying to connect using too many different keys. I have tried the ip address directly, as well as the special address that amazon assigns, and neither of those work. Also, when I attempt to ssh to my server from my macbook, it lists the correct IP in the verbose output.
Here is the output of a telnet attempt to port 22:
telnet mysite.com 22 Trying 184.72.xx.xx... telnet: connect to address 184.72.xx.xx: Connection refused telnet: Unable to connect to remote host
Basically, I am completely out of ideas, and would appreciate any help. I feel like I have tried just about everything, though there must be something that I am missing. Is it possible that my macbook is blocking certain traffic without my knowledge? I checked the firewall settings and it is disabled, and ipfw is not running either (I don't think).
Update: I have attempted a traceroute to my server from my macbook, it fails saying, 'No route to host':
$ traceroute -I 184.72.xx.xx traceroute to 184.72.xx.xx (18.104.22.168), 64 hops max, 72 byte packets traceroute: sendto: No route to host 1 traceroute: wrote 184.72.xx.xx 72 chars, ret=-1 *traceroute: sendto: No route to host traceroute: wrote 184.72.xx.xx 72 chars, ret=-1 *traceroute: sendto: No route to host traceroute: wrote 184.72.xx.xx 72 chars, ret=-1 * traceroute: sendto: No route to host 2 traceroute: wrote 184.72.xx.xx 72 chars, ret=-1 *traceroute: sendto: No route to host traceroute: wrote 184.72.xx.xx 72 chars, ret=-1 *traceroute: sendto: No route to host traceroute: wrote 184.72.xx.xx 72 chars, ret=-1
From a linux box on my local network things look good:
# traceroute -I 184.72.xx.xx traceroute to 184.72.xx.xx (184.72.xx.xx), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 0.190 ms 0.237 ms 0.282 ms 2 10.1.10.1 (10.1.10.1) 0.946 ms 1.779 ms 2.138 ms 3 22.214.171.124 (126.96.36.199) 16.581 ms 18.187 ms 32.675 ms 4 te-9-2-ur02.delrayeast.fl.pompano.comcast.net (188.8.131.52) 17.810 ms 17.976 ms 18.077 ms 5 te-8-1-ur01.bocaraton.fl.pompano.comcast.net (184.108.40.206) 18.325 ms 18.427 ms 18.521 ms 6 te-3-4-ar01.stuart.fl.pompano.comcast.net (220.127.116.11) 19.430 ms 18.559 ms 18.645 ms 7 te-0-4-0-5-ar03.northdade.fl.pompano.comcast.net (18.104.22.168) 24.839 ms 24.438 ms 24.525 ms 8 pos-0-4-0-0-cr01.miami.fl.ibone.comcast.net (22.214.171.124) 23.113 ms 16.435 ms 24.480 ms 9 xe-10-1-0.edge2.Miami1.Level3.net (126.96.36.199) 23.354 ms 23.544 ms 24.256 ms 10 ae-32-52.ebr2.Miami1.Level3.net (188.8.131.52) 30.777 ms 31.698 ms 31.878 ms 11 ae-2-2.ebr2.Atlanta2.Level3.net (184.108.40.206) 36.471 ms 37.461 ms 37.654 ms 12 ae-73-73.ebr3.Atlanta2.Level3.net (220.127.116.11) 37.825 ms 37.917 ms 38.013 ms 13 ae-2-2.ebr1.Washington1.Level3.net (18.104.22.168) 50.805 ms 42.708 ms 47.774 ms 14 ae-91-91.csw4.Washington1.Level3.net (22.214.171.124) 48.827 ms 49.018 ms 49.122 ms 15 ae-4-90.edge3.Washington1.Level3.net (126.96.36.199) 56.149 ms 113.159 ms 114.077 ms 16 AMAZON.COM.edge3.Washington1.Level3.net (188.8.131.52) 88.162 ms 47.429 ms 57.533 ms 17 184.108.40.206 (220.127.116.11) 68.472 ms 52.906 ms 57.836 ms 18 18.104.22.168 (22.214.171.124) 58.755 ms 43.988 ms 50.344 ms 19 126.96.36.199 (188.8.131.52) 51.369 ms 43.720 ms 48.007 ms 20 * * * 21 184.108.40.206 (220.127.116.11) 49.900 ms 46.469 ms 50.883 ms 22 * * * 23 * * * 24 mail.myserver.com (184.72.xx.xx) 48.432 ms 45.051 ms 49.796 ms
Coming back from the server, the results also look reasonable:
# traceroute -I 76.109.130.xx traceroute to 76.109.130.xx (18.104.22.168), 30 hops max, 40 byte packets 1 10.204.200.3 (10.204.200.3) 10.902 ms 4.576 ms 0.466 ms 2 10.1.44.25 (10.1.44.25) 0.621 ms 0.634 ms 0.366 ms 3 10.1.34.136 (10.1.34.136) 0.484 ms 0.804 ms 20.380 ms 4 22.214.171.124 (126.96.36.199) 0.401 ms 0.457 ms 0.415 ms 5 188.8.131.52 (184.108.40.206) 0.373 ms 0.458 ms 0.438 ms 6 220.127.116.11 (18.104.22.168) 1.265 ms 1.280 ms 1.214 ms 7 22.214.171.124 (126.96.36.199) 2.014 ms 2.079 ms 2.089 ms 8 xe-4-0-0.edge3.Washington1.Level3.net (188.8.131.52) 1.369 ms 1.445 ms 1.477 ms 9 vlan90.csw4.Washington1.Level3.net (184.108.40.206) 1.499 ms 1.503 ms 1.498 ms 10 ae-91-91.ebr1.Washington1.Level3.net (220.127.116.11) 2.367 ms 2.272 ms 2.453 ms 11 ae-2-2.ebr3.Atlanta2.Level3.net (18.104.22.168) 15.431 ms 15.273 ms 15.684 ms 12 ae-73-73.ebr2.Atlanta2.Level3.net (22.214.171.124) 18.637 ms 21.841 ms 26.061 ms 13 ae-2-2.ebr2.Miami1.Level3.net (126.96.36.199) 29.121 ms 32.777 ms 36.370 ms 14 ae-2-52.edge2.Miami1.Level3.net (188.8.131.52) 28.909 ms 28.445 ms 28.545 ms 15 184.108.40.206 (220.127.116.11) 29.504 ms 29.760 ms 29.013 ms 16 pos-0-13-0-0-ar03.northdade.fl.pompano.comcast.net (18.104.22.168) 30.111 ms 31.494 ms 32.045 ms 17 te-8-7-ar01.stuart.fl.pompano.comcast.net (22.214.171.124) 33.002 ms 32.879 ms 33.023 ms 18 te-9-1-ur01.bocaraton.fl.pompano.comcast.net (126.96.36.199) 35.068 ms 34.887 ms 34.901 ms 19 te-9-4-ur02.delrayeast.fl.pompano.comcast.net (188.8.131.52) 36.183 ms 35.679 ms 35.730 ms 20 te-17-10-cdn04.delrayeast.fl.pompano.comcast.net (184.108.40.206) 48.517 ms 56.562 ms 55.199 ms 21 c-76-109-130-xx.hsd1.fl.comcast.net (76.109.130.xx) 43.331 ms 49.565 ms 45.136 ms
I have also tested traceroute against the other EC2 box that I am having trouble connecting to and I see the same results. Traceroute does seem to work on my macbook for servers that I can access via ssh. Again, I have no trouble getting to those boxes from my browser.