10

If I change my SSH port from 22 to 23453, I can no longer ssh in.

In more detail, I'm using a Red Hat EC2 instance on Amazon Web Services. This is the second change I've on a fresh install (first change was to add a non-root user).

I can ssh in fine using Git Bash and a local .ssh/config file, I edit the line in /etc/ssh/sshd_config that currently says

#Port 23453

to say

Port 23453

then restart sshd with

sudo service sshd restart

I then add a line "Port 23453" my .ssh/config file

Host foo 
Hostname my-ec2-public-DNS
Port 23453
IdentityFile my ssl key

If I open another Git Bash shell (without closing my existing connection) and attempt to ssh into my instance (with ssh foo) I see the following error:

ssh: connect to host my-ec2-public-DNS port 23453: Bad file number

The security group attached to this instance has two entries, both TCP

22 (SSH) 0.0.0.0/0

23453 0.0.0.0/0

My best guess is that the port is still blocked by my firewall.

The output of sudo iptables -L is as follows

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Which looks pretty open to me.

UPDATE

After adding an iptables rule

iptables -A INPUT -p tcp --dport 23453 -j ACCEPT

and trying again, still no luck.

Output of iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:23453

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Which looks sufficiently open. I'm not entirely sure how to look for packets coming in or activity on the port. But the output of netstat -ntlp (as root)

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State PID/Program name
tcp        0      0 0.0.0.0:56137               0.0.0.0:*                   LISTEN      948/rpc.statd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      930/rpcbind
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1012/cupsd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1224/master
tcp        0      0 0.0.0.0:23453               0.0.0.0:*                   LISTEN      32638/sshd
tcp        0      0 :::36139                    :::*                        LISTEN      948/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      930/rpcbind
tcp        0      0 ::1:631                     :::*                        LISTEN      1012/cupsd
tcp        0      0 :::23453                    :::*                        LISTEN      32638/sshd

Which seems to me to show sshd on 23453.

I've checked again that the instance has the port open in the security group (Port: 23453, Protocol: tcp, Source: 0.0.0.0/0)

What else can be causing the failure to connect via SSH?

Cheers

POSTMORTEM

I can now connect. It was a missing rule in iptables. The output of iptables -L now looks like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:23453 state NEW
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Improve this question
1
  • 1
    For anyone who can't see the difference between the third iptables -L (ssh works) and the second iptables -L (ssh is blocked). Look at the order of the rules in the INPUT chain (the 6 lines under the first "target"), they're read from top to bottom, so in the second set of rules, "REJECT all" is hit before "ACCEPT tcp dpt:23453". The third set of rules has the ACCEPT entry above, and hence before, the REJECT entry.
    – vowel-house-might
    Commented Sep 16, 2014 at 11:22

4 Answers 4

Reset to default
13

Your instance firewall doesn't have this port open. Try the following command:

iptables -I INPUT 3 -s 0.0.0.0/0 -d 0.0.0.0/0 -p tcp --dport 23453 -m state --state New -j ACCEPT

Note that iptables rules need to be saved to persist after a reboot. On RHEL that's:

/sbin/service iptables save
Improve this answer
6
  • This worked. If someone could tell me important difference between @melsayed's iptables rule and Frank's iptables rule, I'd be very thankful. Also, I guess this means the answer to my ultimate question is "Yes, on AWS you do have to open ports in ec2 instance firewalls as well as their security groups". Thanks everyone.
    – vowel-house-might
    Commented Nov 21, 2012 at 15:39
  • 2
    Basically, Frank's iptables command adds (-A) a new rule that would allow connection to that port. The problem is, it adds the rule at the end of the iptables list of rules. The last rule in the iptables list rejects anything that's not explicitly allowed before it. Since iptables rules are applied in order, the connection is matched and rejected before even getting to the new rule.
    – melsayed
    Commented Nov 21, 2012 at 15:50
  • 2
    I inserted a new rule in the list, at the 3rd position (-I INPUT 3). Which is matched before the reject rule at the end, hence allowing the connection. As Frank mentioned, you could use iptables -nvL to see the number of packets matched by each rule, which helps while debugging such configuration.
    – melsayed
    Commented Nov 21, 2012 at 15:51
  • 1
    /sbin/service iptables save isn't working for me, even with sudo.
    – huertanix
    Commented Aug 18, 2016 at 2:59
  • 1
    @huertanix that would really depend on the distro you are using.
    – melsayed
    Commented Aug 27, 2016 at 8:50
2

Add an iptables rule 

iptables -I INPUT 1 -p tcp --dport 23435 -j ACCEPT

which accepts traffic from any host, by port 23435, and try to ssh, if you see any packets or activity, then it means that the packets are reaching your server.

If you see no packets, that means that the AWS Security group does not have rule to allow your port.

but if you see traffic on this rule (by iptables -nvL), then you have to run "netstat -ntlp" and verify if SSH daemon is running on port 2435. and on 0.0.0.0/0.

hopefully these steps would resolve the issue. if still no, then tell me.

Improve this answer
1

Are you sure the security group is set correctly? Did you click "Apply changes"? Many people forget to actually apply their changes :)

"Bad file number" usually signifies connection timeouts, and your iptables setup looks correct.

Improve this answer
1
  • I fell foul to the "apply changes" pitfall once before. Never again. :)
    – vowel-house-might
    Commented Nov 21, 2012 at 15:17
-1

In case anyone stumbles across this topic because they changed the default port of ssh, here's a solution that worked out for me:

  1. To bypass a corporate firewall, I changed the port to 80 on /etc/ssh/sshd_conf.
  2. Unfortunately, Apache was already installed on that instance so I could not ssh anymore.
  3. I detached the volume from the instance.
  4. attached it to another instance
  5. mounted it, changed the port in the config file
  6. detached it, re attached it on the old instance
  7. rebooted: all good :D
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .