I have two centos 7.2 server. One machine ip is 10.104.196.18, another machine is 10.240.197.21. I can successfully ssh from 10.104.196.18 to 10.240.197.21. But failed to ssh into 10.240.196.18 from 10.240.197.21.
The ssh log is like this:
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.104.196.18 [10.104.196.18] port 6990.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "/root/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
The sshd log is like this:
[root@localhost ~]# /usr/sbin/sshd -dD -p 10000
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #1 type 3 ECDSA
debug1: private host key: #2 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-dD'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='10000'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 10000 on 0.0.0.0.
Server listening on 0.0.0.0 port 10000.
debug1: Bind to port 10000 on ::.
Server listening on :: port 10000.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.96.203.72 port 49845 on 10.240.197.21 port 10000
So it's apparently both the client and server pending at exchange identities. And from tcpdump we are confirmed by packets analysis.
tcpdump from 10.240.197.21
[root@localhost ~]# tcpdump -i enp22s0f3 host 10.104.196.18 -s0 -vv -X -c 1000
tcpdump: listening on enp22s0f3, link-type EN10MB (Ethernet), capture size 65535 bytes
13:22:37.402757 IP (tos 0x0, ttl 64, id 25620, offset 0, flags [DF], proto TCP (6), length 60)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [S], cksum 0x9eae (incorrect -> 0x290b), seq 2201485227, win 29200, options [mss 1460,sackOK,TS val 300522431 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 6414 4000 4006 3828 0af0 c515 E..<d.@[email protected](....
0x0010: 0a68 c412 ddc2 0016 8337 ffab 0000 0000 .h.......7......
0x0020: a002 7210 9eae 0000 0204 05b4 0402 080a ..r.............
0x0030: 11e9 9bbf 0000 0000 0103 0307 ............
13:22:37.403162 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.104.196.18.ssh > localhost.localdomain.56770: Flags [S.], cksum 0x2b22 (correct), seq 4104511511, ack 2201485228, win 28960, options [mss 1460,sackOK,TS val 312416107 ecr 300522431,nop,wscale 7], length 0
0x0000: 4500 003c 0000 4000 3c06 a03c 0a68 c412 E..<..@.<..<.h..
0x0010: 0af0 c515 0016 ddc2 f4a5 e017 8337 ffac .............7..
0x0020: a012 7120 2b22 0000 0204 05b4 0402 080a ..q.+"..........
0x0030: 129f 176b 11e9 9bbf 0103 0307 ...k........
13:22:37.403219 IP (tos 0x0, ttl 64, id 25621, offset 0, flags [DF], proto TCP (6), length 52)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [.], cksum 0x9ea6 (incorrect -> 0xca29), seq 1, ack 1, win 229, options [nop,nop,TS val 300522431 ecr 312416107], length 0
0x0000: 4500 0034 6415 4000 4006 382f 0af0 c515 E..4d.@[email protected]/....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8010 00e5 9ea6 0000 0101 080a 11e9 9bbf ................
0x0030: 129f 176b ...k
13:22:37.403801 IP (tos 0x0, ttl 64, id 25622, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xd432), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300522432 ecr 312416107], length 23
0x0000: 4500 004b 6416 4000 4006 3817 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 9bc0 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:37.604502 IP (tos 0x0, ttl 64, id 25623, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xd369), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300522633 ecr 312416107], length 23
0x0000: 4500 004b 6417 4000 4006 3816 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 9c89 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:37.808499 IP (tos 0x0, ttl 64, id 25624, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xd29d), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300522837 ecr 312416107], length 23
0x0000: 4500 004b 6418 4000 4006 3815 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 9d55 ...............U
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:38.217526 IP (tos 0x0, ttl 64, id 25625, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xd104), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300523246 ecr 312416107], length 23
0x0000: 4500 004b 6419 4000 4006 3814 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 9eee ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:39.035515 IP (tos 0x0, ttl 64, id 25626, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xcdd2), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300524064 ecr 312416107], length 23
0x0000: 4500 004b 641a 4000 4006 3813 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 a220 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:40.671529 IP (tos 0x0, ttl 64, id 25627, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xc76e), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300525700 ecr 312416107], length 23
0x0000: 4500 004b 641b 4000 4006 3812 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 a884 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:43.947534 IP (tos 0x0, ttl 64, id 25628, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xbaa2), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300528976 ecr 312416107], length 23
0x0000: 4500 004b 641c 4000 4006 3811 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 b550 ...............P
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:22:50.491548 IP (tos 0x0, ttl 64, id 25629, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0xa112), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300535520 ecr 312416107], length 23
0x0000: 4500 004b 641d 4000 4006 3810 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11e9 cee0 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:23:03.579533 IP (tos 0x0, ttl 64, id 25630, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0x6df2), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300548608 ecr 312416107], length 23
0x0000: 4500 004b 641e 4000 4006 380f 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11ea 0200 ................
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:23:29.755543 IP (tos 0x0, ttl 64, id 25631, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0x07b2), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300574784 ecr 312416107], length 23
0x0000: 4500 004b 641f 4000 4006 380e 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11ea 6840 ..............h@
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
13:24:22.171532 IP (tos 0x0, ttl 64, id 25632, offset 0, flags [DF], proto TCP (6), length 75)
localhost.localdomain.56770 > 10.104.196.18.ssh: Flags [P.], cksum 0x9ebd (incorrect -> 0x3af1), seq 1:24, ack 1, win 229, options [nop,nop,TS val 300627200 ecr 312416107], length 23
0x0000: 4500 004b 6420 4000 4006 380d 0af0 c515 E..Kd.@[email protected].....
0x0010: 0a68 c412 ddc2 0016 8337 ffac f4a5 e018 .h.......7......
0x0020: 8018 00e5 9ebd 0000 0101 080a 11eb 3500 ..............5.
0x0030: 129f 176b 5353 482d 322e 302d 4f70 656e ...kSSH-2.0-Open
0x0040: 5353 485f 362e 362e 310d 0a SSH_6.6.1..
tcpdump from 10.104.196.18
01:22:37.400147 IP 10.240.197.21.56770 > localhost.localdomain.ssh: Flags [S], seq 2201485227, win 29200, options [mss 1460,sackOK,TS val 300522431 ecr 0,nop,wscale 7], length 0
01:22:37.400219 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [S.], seq 4104511511, ack 2201485228, win 28960, options [mss 1460,sackOK,TS val 312416107 ecr 300522431,nop,wscale 7], length 0
01:22:37.400476 IP 10.240.197.21.56770 > localhost.localdomain.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 300522431 ecr 312416107], length 0
01:22:37.426399 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312416133 ecr 300522431], length 23
01:22:37.626271 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312416333 ecr 300522431], length 23
01:22:37.831584 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312416538 ecr 300522431], length 23
01:22:38.242549 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312416949 ecr 300522431], length 23
01:22:39.065132 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312417772 ecr 300522431], length 23
01:22:40.709282 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312419416 ecr 300522431], length 23
01:22:43.997804 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312422704 ecr 300522431], length 23
01:22:50.574310 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312429281 ecr 300522431], length 23
01:23:03.725361 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312442432 ecr 300522431], length 23
01:23:30.029758 IP localhost.localdomain.ssh > 10.240.197.21.56770: Flags [P.], seq 1:24, ack 1, win 227, options [nop,nop,TS val 312468736 ecr 300522431], length 23
And I also has disabled both firewall by scripts like this.
systemctl stop firewalld
systemctl disable firewalld
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -t security -F
iptables -t security -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEP //error this line ,output iptables: Bad policy name. Run `dmesg' for more information.
although has some error, but the result sounds good:
[root@localhost examples]# ~/disable_firewall.sh
iptables: Bad policy name. Run `dmesg' for more information.
[root@localhost examples]# iptables-save
# Generated by iptables-save v1.4.21 on Sat Oct 14 13:08:28 2017
*security
:INPUT ACCEPT [220:24998]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [109:12506]
COMMIT
# Completed on Sat Oct 14 13:08:28 2017
# Generated by iptables-save v1.4.21 on Sat Oct 14 13:08:28 2017
*raw
:PREROUTING ACCEPT [692:70796]
:OUTPUT ACCEPT [109:12506]
COMMIT
# Completed on Sat Oct 14 13:08:28 2017
# Generated by iptables-save v1.4.21 on Sat Oct 14 13:08:28 2017
*mangle
:PREROUTING ACCEPT [692:70796]
:INPUT ACCEPT [220:24998]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [109:12506]
:POSTROUTING ACCEPT [109:12506]
COMMIT
# Completed on Sat Oct 14 13:08:28 2017
# Generated by iptables-save v1.4.21 on Sat Oct 14 13:08:28 2017
*nat
:PREROUTING ACCEPT [395:43515]
:INPUT ACCEPT [32:7088]
:OUTPUT ACCEPT [17:1020]
:POSTROUTING ACCEPT [17:1020]
COMMIT
# Completed on Sat Oct 14 13:08:28 2017
# Generated by iptables-save v1.4.21 on Sat Oct 14 13:08:28 2017
*filter
:INPUT ACCEPT [220:24998]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [109:12506]
COMMIT
And both side can ping each other successfully. So I am confused how the packets are losed unidirectionally only?
