I'm a novice sysadmin working on a system with a CentOS server masquerading traffic with client machines also running CentOS. Something with the tables got messed up and my clients no longer have internet connectivity.
I'm trying to get ping
requests working as a first step and
have pinpointed where the traffic flow is failing, but I don't
understand the configuration enough to know what to do next.
To the more experienced sysadmins, does a quick glance at my
tables reveal anything abnormal?
Ping failure.
1. Server.
IP=98.139.183.24 ; _ tcpdump -i any "dst host $IP or src host $IP"
2. Client.
» IP=98.139.183.24; ping $IP
PING 98.139.183.24 (98.139.183.24) 56(84) bytes of data
3. Server.
For normal operation, I should be seeing responses, which makes me suspect something is wrong with the iptable configuration on the master node.
Note: b6
is mapped to 10.0.2.6
in /etc/hosts
.
» IP=98.139.183.24 ; _ tcpdump -i any "dst host $IP or src host $IP"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
20:07:38.553547 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 1, length 64
20:07:38.553580 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 1, length 64
20:07:39.552969 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 2, length 64
20:07:39.552983 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 2, length 64
20:07:40.552963 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 3, length 64
20:07:40.552975 IP b6 > ir2.fp.vip.bf1.yahoo.com: ICMP echo request, id 3120, seq 3, length 64
^C
6 packets captured
6 packets received by filter
Configuration.
Note: I've modified this to the configuration working for me to possibly help anybody with the same issue in the future.
Server.
ip addr
~ » ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 90:e2:ba:21:b8:10 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.0/8 brd 10.255.255.255 scope global p2p1
inet6 fe80::92e2:baff:fe21:b810/64 scope link
valid_lft forever preferred_lft forever
3: p2p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 90:e2:ba:21:b8:11 brd ff:ff:ff:ff:ff:ff
inet <ext IP>/24 brd <ext ip prefix>.255 scope global p2p2
inet6 2001:468:c80:2106:92e2:baff:fe21:b811/64 scope global dynamic
valid_lft 2591809sec preferred_lft 604609sec
inet6 fe80::92e2:baff:fe21:b811/64 scope link
valid_lft forever preferred_lft forever
4: p2p3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether 90:e2:ba:21:b8:14 brd ff:ff:ff:ff:ff:ff
5: p2p4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether 90:e2:ba:21:b8:15 brd ff:ff:ff:ff:ff:ff
6: em1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether d4:ae:52:99:8c:29 brd ff:ff:ff:ff:ff:ff
7: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether d4:ae:52:99:8c:2a brd ff:ff:ff:ff:ff:ff
ip route
~ » ip route
<ext ip prefix>.0/24 dev p2p2 proto kernel scope link src <ext ip>
<prefix 1>.0.0/16 dev p2p1 scope link metric 1002
<prefix 1>.0.0/16 dev p2p2 scope link metric 1003
10.0.0.0/8 dev p2p1 proto kernel scope link src 10.0.1.0
default via <ext ip prefix>.1 dev p2p2 proto static
iptables.
~ » _ iptables -t filter -L -v -n
Chain INPUT (policy ACCEPT 3715K packets, 531M bytes)
pkts bytes target prot opt in out source destination
76M 111G fail2ban-SSH tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
151M 183G ACCEPT all -- p2p2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
38M 6612M ACCEPT all -- p2p1 * 0.0.0.0/0 0.0.0.0/0
1604K 101M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
1923 142K ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
755K 62M ACCEPT all -- p2p1 p2p2 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- p2p2 p2p1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 171M packets, 35G bytes)
pkts bytes target prot opt in out source destination
Chain fail2ban-SSH (1 references)
pkts bytes target prot opt in out source destination
76M 111G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
~ » _ iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 1733K packets, 974M bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 1722K packets, 973M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 10956 packets, 892K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1989K packets, 201M bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 2001K packets, 202M bytes)
pkts bytes target prot opt in out source destination
~ » _ iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 26992 packets, 6507K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 10234 packets, 954K bytes)
pkts bytes target prot opt in out source destination
532K 54M MASQUERADE all -- * p2p2 10.0.0.0/8 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 3229 packets, 394K bytes)
pkts bytes target prot opt in out source destination
ipforward: yes
» cat /proc/sys/net/ipv4/ip_forward 1
» cat /etc/sysctl.conf | grep ip_forward net.ipv4.ip_forward = 1
Client: 10.0.2.6
.
ip addr
~ » ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 18:03:73:0d:89:15 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.6/8 brd 10.0.0.255 scope global em1
inet6 fe80::1a03:73ff:fe0d:8915/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 18:03:73:0d:89:17 brd ff:ff:ff:ff:ff:ff
ip route.
~ » ip route
<prefix 1>.0.0/16 dev em1 scope link metric 1002
10.0.0.0/8 dev em1 proto kernel scope link src 10.0.2.6
default via 10.0.1.0 dev em1
iptables.
» _ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
~ » _ iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
~ » _ iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination